Data processing method, system and medium based on memory rule engine

Abstract

The present invention discloses a data processing method, system and medium based on a memory rule engine, which belongs to the field of data processing technology. The method includes: establishing a dynamically expandable memory erasable operator library; selecting operators required for tasks; setting data sources for each operator, converting all operators into independent operator objects, generating task formula texts based on the connection relationship of all operator objects; saving the task formula texts into a database; when processing target data, first inputting the target data into a data preprocessing module for preprocessing to obtain first data; then inputting the first data into a task execution engine, and the task execution engine loads the task formula text from the database to analyze and calculate the first data to obtain a calculation result value; finally, inputting the calculation result value into a result signal processing module to execute the target task. It is applicable to both time series and non-time series data business scenarios, and is applicable to both scenarios where historical values do not need to be memorized and scenarios with multiple historical memory values.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, system and medium based on a memory rule engine. Background Art

[0002] A rule engine is a component embedded in an application that separates business decisions from application code and uses predefined semantic modules to write business decisions. It accepts data input, interprets business rules, and makes business decisions based on the business rules. Existing rule engines are generally used in application scenarios where numerical logical operations are relatively simple but parameters can be flexible. For example, in financial risk control scenarios, rule engines are used to automatically determine whether to approve a loan based on the borrower's credit score, income level, and other financial information [if the borrower's credit score is >= 700 and the borrower's monthly income is >5000, then approve the borrower's loan]. In medical diagnosis scenarios, rule engines can assist doctors in providing possible diagnoses and treatment plans based on the patient's symptoms and examination results [if the patient has high blood pressure and is >60 years old, then recommend regular checkups]. In the insurance claims industry, rule engines are used to automate the claims process, determine whether the claim application complies with company policies, and quickly process simple claims [if the claim application amount is <5000 and the accident date of the claim application is within the policy validity period, then quickly process the claim].

[0003] Existing rule engines are all built for simple data operations, such as four simple addition, subtraction, multiplication, and division operators. When using these operators to generate formulas, there is no need to consider the impact of historical values on the latest calculation results. Instead, the same operator is treated as an independent new call each time it is called, and the need to use the previous input value or result value to calculate the latest result is not considered.

[0004] Traditional data processing technologies are often limited to specific data types and business scenarios, such as solutions specifically targeting time-series or non-time-series data. These solutions are often inadequate when dealing with complex and ever-changing business requirements. In particular, when dealing with mixed business scenarios involving both time-series and non-time-series data, existing data processing tools and methods often lack sufficient flexibility and adaptability, making it difficult to meet the urgent needs of enterprises for efficient and accurate data analysis. For time-series data business scenarios, such as financial transaction analysis and IoT device monitoring, the time-varying nature of data is crucial, requiring the ability to capture time-series relationships and trends. However, traditional time-series data processing technologies typically focus solely on the temporal dimension of data, ignoring the potential need for multiple historical memory values within business logic—the ability to make current decisions or predictions based on data from multiple past time points. On the other hand, non-time-series data business scenarios, such as customer relationship management and inventory management, while not explicitly involving time-series analysis, also require processing complex data relationships and rules. This is particularly true in decision support systems, where intelligent reasoning and judgment based on historical data and business rules are often required. Existing non-time-series data processing solutions often lack sufficient memory mechanisms, making it difficult to effectively support business logic requiring multi-step, multi-conditional judgments. Existing data processing systems often use hard-coded or fixed configurations to implement specific data processing logic, which not only limits system flexibility but also increases maintenance and upgrade costs. When conducting data analysis, business personnel often need to rely on professional technicians to perform complex programming or configuration tasks, which not only reduces work efficiency but also limits their ability to independently explore and innovate. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a data processing method, system and medium based on a memory rule engine.

[0006] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides: a data processing method based on a memory rule engine, comprising the following steps:

[0007] Establish a dynamically scalable memory-erasable operator library;

[0008] On the analysis task creation page, select the operator required for the task;

[0009] Set up data sources for each operator, convert all operators into independent operator objects, and generate task formula text based on the connection relationship between all operator objects;

[0010] Save the task formula text into the database;

[0011] When processing target data, the target data is first input into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and calculate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0012] Preferably, the internal processing logic of the operators in the dynamically expandable memory-erasable operator library includes the following steps:

[0013] The calculation result object of the operator is recorded as result, and the calculation result object result is initialized and assigned according to the operator operation parameters and operator operation logic;

[0014] If the environment parameter clear passed to the task execution engine during execution is 1, the value in the memory unit corresponding to the operator object is reset. Otherwise, the value in the memory unit corresponding to the operator object is retained for use in subsequent operations. Then, according to the objectName of the operator object, the memory value is retrieved from the memory unit corresponding to the operator object, recorded as memory[objName].

[0015] Perform calculations based on the memory value memory[objName] and the operator operation parameters passed in when the operator object is called, and update the obtained operation result value to the calculation result object result;

[0016] After completing the update of the memory cell value according to the operator operation logic, the final calculation result object result is returned;

[0017] The memory unit values are stored in a three-layer tree-like storage structure, wherein the first layer of the three-layer tree-like storage structure stores the names of each operator object, the second layer of the three-layer tree-like storage structure stores the names of memory parameters that require operator logic access, and the third layer of the three-layer tree-like storage structure stores the actual memory unit values.

[0018] Preferably, for a memoryless operator, after the calculation result object result is initialized and assigned a value, the operator operation parameters are directly used to perform calculations according to the operator operation logic, and after the obtained operation result value is updated to the calculation result object result, the final calculation result object result is directly returned.

[0019] Preferably, when setting operator operation parameters, they are set according to the principle of the minimum number of operator operation parameters required for the operator and nested calculations of the same operator.

[0020] Preferably, when the expression of the falling edge operator Down is called as Down(name1,a), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the falling edge operator Down includes the following steps:

[0021] Initialize the value field of the calculation result object result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of the calculation result object result;

[0022] According to the operator object name of the falling edge operator Down, the parameter value of the last call is retrieved from the memory unit;

[0023] If the parameter value of the last call is greater than the value of the operator operation parameter a, the value of the calculation result object result is updated to 1, otherwise the value of the calculation result object result is not updated;

[0024] Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the operator operation parameter a, and return the calculation result object result.

[0025] Preferably, when the expression of the CrossOver operator is CrossOver(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the CrossOver operator includes the following steps:

[0026] Initialize the value field of the CrossOver operator's calculation result object result to 0, and assign the timestamp fields of the operator's operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), and result.value = 0.

[0027] According to the operator object name of the CrossOver operator, the first parameter value and the second parameter value of the last call are retrieved from the memory unit;

[0028] Perform calculations using the first and second parameter values and operator parameters a and b. If the first parameter value retrieved from the memory cell is less than or equal to the second parameter value and the value of the operator parameter a passed in this time is greater than the value of b, then the value of the calculation result object result is updated to 1; otherwise, the value of the calculation result object result is not updated.

[0029] Update the memory value of the operator object in the memory unit of the CrossOver operator to the values of the two operator operation parameters a and b of this call, and return the calculation result object result.

[0030] Preferably, when the expression of the memoryless multiplication operator Mul is called as Mul(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the memoryless multiplication operator Mul includes the following steps:

[0031] Initialize the value field of the result object result of the memoryless multiplication operator Mul to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), and result.value = 0;

[0032] Use the value fields of operator parameters a and b to perform calculations, update result.value = a.value * b.value, and then directly return the value of the calculation result object result at this time.

[0033] Preferably, when the dynamically expandable memory-erasable operator library does not have the operator required for the task, the operator required for the task is added to the expandable memory-erasable operator library according to the operator creation specification.

[0034] A second aspect of the present invention provides: a data processing system based on a memory rule engine, for implementing any of the above-mentioned data processing methods based on a memory rule engine, comprising:

[0035] Operator library establishment module, used to establish a dynamically scalable and erasable operator library;

[0036] The operator selection module is used to select the operators required for the task on the analysis task creation page;

[0037] The task formula text generation module is used to set the data source for each operator, convert all operators into independent operator objects, and generate the task formula text based on the connection relationship between all operator objects;

[0038] Database module, used to save task formula text into the database;

[0039] The target task execution module inputs the target data into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0040] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned data processing methods based on a memory rule engine is implemented.

[0041] The beneficial effects of the present invention are:

[0042] 1) It is applicable to both time-series data and non-time-series data business scenarios. It can handle both simple scenarios that do not require historical value memory and scenarios with multiple historical memory values. At the same time, the technical solution of the present invention has good scalability, and can continuously enrich and improve the available operators in the operator library, supporting business personnel to complete business data analysis in the simplest drag-and-drop method; it simplifies the data analysis operations and efficiency of business personnel, better supports the separation of business and technology, and improves enterprise work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a data processing method based on a memory rule engine;

[0044] Figure 2 Create a flowchart for the task formula text;

[0045] Figure 3 Flowchart for target data processing;

[0046] Figure 4 Schematic diagram of falling edge operator and counter operator;

[0047] Figure 5 Schematic diagram of addition, subtraction, multiplication and division operators;

[0048] Figure 6 It is the logic flow chart for the internal processing of the operator;

[0049] Figure 7 Schematic diagram of the up-crossing operator;

[0050] Figure 8 Schematic diagram of the Greater operator;

[0051] Figure 9 Schematic diagram of the memory unit structure of the three-layer pointer model;

[0052] Figure 10 Schematic diagram of another three-layer pointer model memory unit structure. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0054] This invention utilizes a rules engine to develop a non-coding configuration tool. This tool enables the splitting, parsing, and storage of message data reported by IoT devices according to specified configuration rules, eliminating the need for back-end developers to hard-code these operations and improving the convenience of data collection and storage. Furthermore, this invention utilizes a general rules engine (e.g., Aviatorscript) to design a set of function parameter specifications and standard logic processing procedures to establish a series of scalable and memorizable general operator libraries. This allows business personnel to independently perform more complex business data analysis through a non-coding drag-and-drop interface, fully leveraging their business knowledge and effectively avoiding communication with back-end developers, thereby reducing costs and increasing efficiency.

[0055] See Figure 1-10 The first aspect of the present invention provides: a data processing method based on a memory rule engine, comprising the following steps:

[0056] Establish a dynamically scalable memory-erasable operator library;

[0057] On the analysis task creation page, select the operator required for the task;

[0058] Set up data sources for each operator, convert all operators into independent operator objects, and generate task formula text based on the connection relationship between all operator objects;

[0059] Save the task formula text into the database;

[0060] When processing target data, the target data is first input into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and calculate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0061] In this embodiment, if Figure 2As shown, on the page provided by the system (S01 module), through the various operators provided by the system, select the required operators and assign values to the operators and input data sources, thereby creating data analysis tasks and corresponding operation formulas, and saving the generated data analysis formulas to the database (the task table sample stored in the final S03 module is shown in Table 1);

[0062] Table 1

[0063]

[0064] exist Figure 2 After the data analysis formula exists in the S03 module shown in the figure, Figure 3 The task execution engine shown in S12 will automatically load the analysis task from S03. After the data preprocessing module shown in S11 processes the data into a standard format, it will automatically flow into the S12 module for analysis and calculation to calculate the result value to be calculated by the corresponding formula, and then Figure 3 The S13 result signal processing module shown in FIG. 1 automatically executes subsequent tasks after obtaining the result. The existing rule engine is used to build a business for simple data operations, such as Figure 5 As shown in the figure, there are four simple addition, subtraction, multiplication and division operators. When using these operators to generate formulas, there is no need to consider the impact of historical values on the latest operation results. Instead, the same operator is treated as an independent new call each time it is called, and the need to use the previous input value or result value to calculate the latest result is not considered. Figure 4 The two operators shown in the S21 and S22 modules need to use certain historical values to calculate the result of the latest call (the logic of the falling edge operator shown in S21 is: if the last input timing signal is 1 and the current input timing signal is 0, then the output result value is 1, otherwise the output result value is 0. Obviously, the calculation result of the latest call of the operator needs to be combined with the last input value to obtain an accurate result; the logic of the counter operator shown in S22 is: count how many times the signal 1 appears in the continuous timing signal. Obviously, the calculation result of the latest call of the operator needs to be calculated based on the value of the last calculation result).

[0065] In order to enable the rule engine to cater to more complex business scenarios (adapting to time series data, non-time series data, and a mixture of memoryless and memory-based data), this invention designs a set of technical specifications for a data analysis system and method with memory-erasable rule engine operators based on a general rule engine (such as Aviatorscript), so that Figure 2 On the S01 analysis task creation page shown in the figure, you can complete the formula generation of the data analysis task by dragging and dropping. Figure 3The S11 data preprocessing module processes the data into data that meets the technical specifications, and then executes the analysis results smoothly in the S12 task execution engine. Figure 2 The final analysis task shown in Table 1 is created on the S01 analysis task creation page. First, a set of extensible, memory-erasable operator libraries needs to be established for users to use directly on the interface provided by the S01 module.

[0066] The concept of turning operators into operator objects is a key point. It is necessary to ensure that the same operator is a different operator object in the same execution formula, and it is also necessary to ensure that the same operator is a different operator object in different execution formulas. Only in this way can Figure 3 The S12 task execution engine shown here precisely controls the historical memory cache value of each operator object entity, thereby ensuring the correctness of the calculation results of each execution formula. To ensure the above two points, when building an extensible memory-erasable operator library, the specifications that each independent operator needs to follow are shown in Table 2:

[0067] Table 2

[0068]

[0069] In each attribute field in Table 2, funcName is the name of the operator and paramNum is the number of external parameters received by the operator. These two parameters serve as the basic information for defining the operator. The memory attribute is the internal attribute value of the operator object, which is used to save the memory cache value of each operator object. The three attribute values of funcName, paramNum and memory are not accepted when calling. The clear attribute is an environment parameter, not a value directly involved in the operator operation. It is a control variable used to control whether to reset the operator object memory. Figure 3 The S12 module shown in the figure automatically determines and assigns values during the running environment [for example, setting the memory cache value to 1 at 12 o'clock every night, or clearing the memory cache value of the specified operator object when the operation value reaches a certain condition or threshold after multiple operations]; objName and operator operation parameters [paramObject1, paramObject2, ...] need to be clearly specified when calling, as they need to be in Figure 2 As shown in the figure, the S02 module needs external parameters to generate the task formula text. objName specifies the specific operator object involved in the operation, and the operator operation parameters [paramObject1, paramObject2, ...] specify the specific parameters involved in the operator operation. Figure 2In the process of dragging operators and setting input data sources to generate corresponding formulas on the page provided by the S01 module shown in the figure, all operator objects (even Figure 5 The simplest addition, subtraction, multiplication and division operators shown in the figure need to create each operator object according to the following specifications: operator object => operator symbol (operator object name, parameter 1, parameter 2, parameter 3, ...).

[0070] In some embodiments, when setting operator operation parameters, they are set according to the principle of the minimum number of operator operation parameters required for the operator and nested calculations of the same operator.

[0071] In this embodiment, in order to improve the versatility of operator development and avoid excessive complexity and diversity of operator operation parameter objects, the present invention sets the operator operation parameters according to the principle of the minimum number of operator operation parameters required by the operator and nested calculations of the same operator. For example, in order to meet the scenario where multiple data need to be added together (for example, the sum of 4 parameter objects a+b+c+d needs to be calculated) or multiple data need to be multiplied together (for example, the product of 6 parameter objects a*b*c*d*e*f needs to be calculated), as shown in the following example: Figure 5 The number of parameters of the addition operator Add shown in S31 and the multiplication operator Mul shown in S32 are both 2. All operations with multiple parameters are processed as nested modes of the same operator. Taking the addition operation of multiple parameter objects as an example (such as calculating the sum of 4 parameter objects: a+b+c+d), there is no need to develop a new 4-parameter addition operator to meet user needs. Figure 1 In the S01 module shown, the four parameters set are processed as the expression of the same addition operator (Add) nested three times in a row, that is, the generated Figure 2 The expression corresponding to the S02 module shown is Add(name1,Add(name2,Add(name3,a,b),c),d).

[0072] The same addition operator Add is nested three times. According to the above operator object creation specifications, the creation of several operator objects and the related specific semantics are as follows:

[0073] Figure 5The expression text of the addition operator module S31 when called once is: Add(name1,a,b). This expression creates an addition operator object named name1, which is created by the memoryless operator whose funcName is Add. During the calculation process, the parameters a and b are replaced by the two parameter objects passed in by the data preprocessing module S11 in the task execution engine module S12. Since the memoryless operator is used when creating the addition operator object named name1, each time the operator object is called, it only calculates the sum of the values of the parameters a and b passed in to the operator object at the current call time, that is, a+b is the output result value of the operator object.

[0074] Figure 7 The expression text of the crossover operator module in S51 when it is called once is CrossOver(name1,a,b). This expression creates a crossover operator object named name1. During the calculation process, the parameters a and b are replaced by the two parameter objects passed in by the data preprocessing module S11 in the task execution engine module S12. The operator object is created by the erasable operator CrossOver with memory. The two operation parameters passed in to the operator object when the operator object is called for the nth time are a and b. n with b n , when the operator object is called for the n+1th time, the two operation parameters passed into the operator object are a n+1 with b n+1 , if a n <=b n And a n+1 >b n+1 The calculation result of the n+1th call is 1, otherwise the calculation result is 0;

[0075] Figure 4 The expression text Down(name1,a) when the falling edge operator module of S21 is called once is shown. This expression creates a falling edge operator object named name1. During the calculation process, the parameter a will be replaced by a parameter object passed in by the data preprocessing module S11 in the task execution engine module S12. This operator object is created by the memory-erasable operator Down. The operation parameter passed in when calling this operator object for the nth time is a. n When the operator object is called for the n+1th time, the operation parameter passed into the operator object is a n+1 , if a n+1 n The calculation result of the n+1th call is 1, otherwise the calculation result is 0;

[0076] Figure 4 ​The expression text Counter(name1,a) of the counter operator module S22 when it is called once creates a counter operator object named name1. During the calculation process, the parameter a is replaced by a parameter object passed in by the data preprocessing module S11 in the task execution engine module S12. The operator object is created by the erasable operator Counter with memory. The calculation result obtained after passing in the operator operation parameters when calling the operator object for the nth time is R. n When the operator object is called for the n+1th time, the operation parameter passed into the operator object is a n+1 , if a n+1 If it is greater than or equal to 1, the calculation result R of the n+1th call is output. n+1 =1+R n Otherwise, the result R n+1 =R n ;

[0077] Operator objects created according to the above operator object creation specifications can be combined and nested to form more complex expressions. The following are several expressions composed of operator objects and their semantic examples:

[0078] Figure 5 The expression text of the addition operator module of S31 when nested calling is: Add(name2,Add(name1,a,b),c). This expression creates two addition operator objects, the names of the two objects are name1 and name2 respectively. The calculation of this expression is Figure 3 The S12 module shown will go through two calculation steps: the first step is to calculate the calculation result of the addition operator object named name1. The addition operator Add is a memoryless operator. During the calculation process, the parameters a and b will be replaced by two actual parameter objects for calculation, and the calculation result value of this step will be a+b; then in the second step, the calculation result value will be passed as the operator operation parameter object together with another operator operation parameter object c that actually participates in the calculation to the addition operator object named name2, and the final calculation result value of the expression Add(name2,Add(name1,a,b),c) will be a+b+c;

[0079] Will Figure 4 The expression text of the two operator modules shown in the figure is Counter(name3,Down(name2,Down(name1,a))). This expression creates three operator objects in total, namely the falling edge operator object named name1, the falling edge operator object named name2, and the counter operator object named name3. The calculation of this expression is Figure 3The S12 module shown in the figure will go through two calculation steps: the first step is to calculate the calculation result of the falling edge operator object named name1, and the operation parameter passed into the operator object when the operator object is called for the nth time is a n When the operator object is called for the n+1th time, the operation parameter passed into the operator object is a n+1 When the operator object is called for the n+2th time, the operation parameter passed into the operator object is a n+2 , with the symbol Dr representing the output result sequence of the falling edge operator object name1 (Dr1, Dr2, ..., Dr n ,Dr n+1 ,Dr n+2 ), the calculation result of the operator object after being called for the n+1th time is recorded as Dr n+1 (If a n+1 n， Dr. n+1 =1, otherwise Dr n+1 =0), the calculation result after the n+2th call is recorded as Dr n+2 (If a n+12 n+1， Dr. n+2 =1, otherwise Dr n+2 =0); In the second step, the value of the falling edge operator object name2 will be calculated. At this time, the calculation result value of the operator object named name1 will be used as the operator operation parameter of the falling edge operator name2, and the symbol DDr represents the output result sequence of the falling edge operator object name2 (DDr1, DDr2, ..., DDr n ,DDr n+1 ,DDr n+2 ), the calculation result after the operator is called for the n+1th time is recorded as DDr n+1 (If Dr. n+1 <Dr n， Then DDr n+1 =1, otherwise DDr n+1 =0), the calculation result after the n+2th call is recorded as DDr n+2 (If Dr. n+2 <Dr n+1， Then DDr n+2 =1, otherwise DDr n+2 =0); In the third step, the calculation result sequence of the falling edge operator object name2 will be used as the operator operation parameter sequence of the counter operator object Counter, and the symbol Cr will represent the calculation result sequence of the counter object (Cr1, Cr2, ..., Cr n ,Cr n+1 ,Cr n+2 ​​), the operator parameter object passed in when the operator object is called for the n+1th time is DDr n+1 , if DDr n+1 If it is greater than or equal to 1, the calculation result Cr n+1 =1+Cr n , otherwise Cr n+1 =Cr n The operator parameter object passed in when the operator is called for the n+2th time is DDr n+2 , if DDr n+2 If it is greater than or equal to 1, the calculation result Cr n+2 =1+Cr n+1 , otherwise Cr n+2 =Cr n+1 .

[0080] In some embodiments, the internal processing logic of the operators in the dynamically expandable memory-erasable operator library includes the following steps:

[0081] The calculation result object of the operator is recorded as result, and the calculation result object result is initialized and assigned according to the operator operation parameters and operator operation logic;

[0082] If the environment parameter clear passed to the task execution engine during execution is 1, the value in the memory unit corresponding to the operator object is reset. Otherwise, the value in the memory unit corresponding to the operator object is retained for use in subsequent operations. Then, according to the objectName of the operator object, the memory value is retrieved from the memory unit corresponding to the operator object, recorded as memory[objName].

[0083] Perform calculations based on the memory value memory[objName] and the operator operation parameters passed in when the operator object is called, and update the obtained operation result value to the calculation result object result;

[0084] After completing the update of the memory cell value according to the operator operation logic, the final calculation result object result is returned;

[0085] The memory unit values are stored in a three-layer tree-like storage structure, wherein the first layer of the three-layer tree-like storage structure stores the names of each operator object, the second layer of the three-layer tree-like storage structure stores the names of memory parameters that require operator logic access, and the third layer of the three-layer tree-like storage structure stores the actual memory unit values.

[0086] In this embodiment, in addition to following the above-mentioned development specifications when developing operators, in order to better be compatible with scenarios with both memory and non-memory, and time series data and non-time series data, this technical solution also designs the processing logic inside the operator accordingly. Figure 6 The specific steps are as follows:

[0087] Step 1: First, record the calculation result object as result, and perform corresponding initialization and assignment operations on the result object result based on the passed operator operation parameters [paramObject1, paramObject2, ...] (if the operator operation parameter does not have a timestamp field value, the field is filled with the default value -1) and the operator operation logic;

[0088] Step 2: According to Figure 3 The environment parameter clear is passed in when the S12 module is executed. If clear is 1, the value of the memory unit corresponding to the operator object is reset first, and then step 3 is entered after the reset;

[0089] Step 3: According to the objectName of the operator object, retrieve the memory value from the memory unit corresponding to the operator object, recorded as memory[objName];

[0090] Step 4: Calculate the value memory[objName] retrieved from step 3 together with the operator operation parameters [paramObject1, paramObject2, ...] passed in when the operator object was called according to the operator's operation logic, and update the obtained operation result value to the result object;

[0091] Step 5: After completing the update of the memory cell value according to the logic of the operator, the final calculation result is returned.

[0092] Due to the existence of nested calls, in order to ensure that the calculation result of an operator object can be passed as the operator operation parameter of other operator objects in both memory and non-memory, time series data and non-time series data scenarios, this technical solution needs to maintain the same structure as the operator operation parameters [paramObject1, paramObject2, ...] in the aforementioned operator specification table when initializing the result to be returned in step 1 above. It also needs to include two fields: timestamp and value.

[0093] For business scenarios with non-time series data, because Figure 6 In step 1, the non-time series data involved in the calculation has been filled with the timestamp field value -1 (such as Figure 8 The non-time series operator operation parameter 10 of the Greater operator is converted to time series data in step 1 and filled with timestamp -1). When assigning a value to the timestamp field of the result object in step 1, set result.timestamp to the maximum timestamp value among all operator operation parameter objects.

[0094] For the most recently called memoryless operator that does not need to use memory data for calculation, you can skip steps 2 and 3 and directly enter step 4 to use the operator operation parameters [paramObject1, paramObject2, ...] to calculate according to the operator's own operation logic, and update the result value to the result object. Then enter step 5 and skip the update of the operator object memory unit, and directly return the final calculation result result. In order to be applicable to the scenario where a single operator object needs to adapt to multiple memory parameters, the present invention adopts the following method: Figure 9 Like Figure 10 The three-layer pointer model shown completes the cache of memory parameters. The first layer stores the names of each operator object of the operator, the second layer stores the parameter names required by the operator, and the last layer stores the memory cache values corresponding to the parameter names.

[0095] In some embodiments, for a memoryless operator, after the calculation result object result is initialized and assigned, the operator operation parameters are directly used to perform calculations according to the operator operation logic, and after the obtained operation result value is updated to the calculation result object result, the final calculation result object result is directly returned.

[0096] In some embodiments, when the expression of the falling edge operator Down is called as Down(name1, a), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the falling edge operator Down includes the following steps:

[0097] Initialize the value field of the calculation result object result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of the calculation result object result;

[0098] According to the operator object name of the falling edge operator Down, the parameter value of the last call is retrieved from the memory unit;

[0099] If the parameter value of the last call is greater than the value of the operator operation parameter a, the value of the calculation result object result is updated to 1, otherwise the value of the calculation result object result is not updated;

[0100] Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the operator operation parameter a, and return the calculation result object result.

[0101] In this embodiment, it is assumed that Figure 4 The expression used by the S21 falling edge operator (Down) module with erasable memory is Down(name1,a), and the environment parameter clear is Figure 3The S12 task is assigned a value of 0 when executing the engine. The operation steps are as follows:

[0102] Step 11: Initialize the value field of the result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of result. That is, at this time, result.timestamp = a.timestamp, result.value = 0;

[0103] Step 12: Since Figure 3 The S12 execution engine module shown sets the environment parameter clear value to 0, so there is no need to reset the value of the memory unit and directly enters step 13;

[0104] Step 13: According to the operator object name1, the memory unit (the memory unit structure of the falling edge operator Down only needs to store the parameter value of the last call, so each operator object only needs one memory unit, and the memory unit structure is as follows Figure 9 (as shown in the example above) and take the last parameter value (the default last parameter value is set to 0 when called for the first time, and will be updated in step 15 after multiple subsequent calls) and record the value as memory[name1];

[0105] Step 14: If memory[name1] retrieved in step 13 is greater than the value of parameter a [i.e. memory[name1] = 1, a.value = 0], then update result.value = 1; otherwise, the value of the result object result is not updated (result.value remains 0).

[0106] Step 15: Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the parameter object a during this call (place the value of a this time into Figure 9 The calculation result object result is returned.

[0107] In some embodiments, when the expression used to call the CrossOver operator is CrossOver(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the CrossOver operator includes the following steps:

[0108] Initialize the value field of the CrossOver operator's calculation result object result to 0, and assign the timestamp fields of the operator's operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), and result.value = 0.

[0109] According to the operator object name of the CrossOver operator, the first parameter value and the second parameter value of the last call are retrieved from the memory unit;

[0110] Perform calculations using the first and second parameter values and operator parameters a and b. If the first parameter value retrieved from the memory cell is less than or equal to the second parameter value and the value of the operator parameter a passed in this time is greater than the value of b, then the value of the calculation result object result is updated to 1; otherwise, the value of the calculation result object result is not updated.

[0111] Update the memory value of the operator object in the memory unit of the CrossOver operator to the values of the two operator operation parameters a and b of this call, and return the calculation result object result.

[0112] In this embodiment, it is assumed that Figure 7 The expression for calling the CrossOver module with erasable memory S51 and S52 is CrossOver(name1,a,b), and the environment parameter clear is Figure 3 The S12 task is assigned a value of 0 when executing the engine. The operation steps are as follows:

[0113] Step 21: Initialize the value field of the CrossOver operator's calculation result to 0, and assign a value to the result timestamp field based on the timestamp fields of the operator's operation parameters a and b. At this time, result.timestamp = Max(a.timestamp, b.timestamp), and result.value = 0;

[0114] Step 22: Since Figure 3 The execution engine module S12 sets the environment parameter clear value to 0, so there is no need to reset the value of the operator object memory unit, and directly enters step 23;

[0115] Step 23: Based on the operator object name1, the memory unit structure of the CrossOver operator needs to store the two parameter values from the last call. Therefore, each operator object requires two memory units. The memory unit structure is as follows: Figure 10 (as shown in the example above) (the default last parameter value is set to 0 when called for the first time, and will be updated in step 25 after multiple subsequent calls), and record this value as memory[name1] (which can be further divided into two values: memory[name1].last1 and memory[name1].last2);

[0116] Step 24: Use memory[name1] retrieved in step 23 and the operator parameters a and b involved in this calculation to perform calculations. If the first parameter value retrieved from the memory unit (memory[name1].last1) is less than or equal to the second parameter value (memory[name1].last2) and the value of the operator parameter a passed in this time is greater than the value of b, then update result.value = 1. Otherwise, the value of the calculation result object result is not updated (result.value remains 0).

[0117] Step 25: Update the memory value of the operator object in the CrossOver memory unit to the values of the two operator operation parameters during this call (place the value of a this time into Figure 10 In the memory unit pointed to by name1->last1, the value of b is placed as follows Figure 10 The calculation result object result is returned.

[0118] In some embodiments, when the expression of the memoryless multiplication operator Mul is called as Mul(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the memoryless multiplication operator Mul includes the following steps:

[0119] Initialize the value field of the result object result of the memoryless multiplication operator Mul to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), and result.value = 0;

[0120] Use the value fields of operator parameters a and b to perform calculations, update result.value = a.value * b.value, and then directly return the value of the calculation result object result at this time.

[0121] In this embodiment, Figure 5 The expression when the memoryless multiplication operator (Mul) module is called is Mul(name1,a,b), and the environment parameter clear is Figure 3 The S12 task is assigned a value of 0 when executing the engine. The operation steps are as follows:

[0122] Step 31: Initialize the value field of the result of the operator Mul to 0, and assign a value to the timestamp field of result according to the timestamp fields of the operator operation parameters a and b. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0;

[0123] Steps 32 and 33: Since this operator is a memoryless operator, skip steps 32 and 33 and go directly to step 34.

[0124] Step 34: Since this operator is a memoryless operator, the operator operation parameters a and b are directly used to perform the calculation, and result.value = a.value * b.value is updated;

[0125] Step 35: Since this operator is a memoryless operator, skip the memory unit update and directly return the calculation result object result.

[0126] In some embodiments, when the dynamically expandable memory-erasable operator library does not have the operator required by the task, the required operator is added to the expandable memory-erasable operator library according to the operator creation specification.

[0127] A second aspect of the present invention provides: a data processing system based on a memory rule engine, for implementing any of the above-mentioned data processing methods based on a memory rule engine, comprising:

[0128] Operator library establishment module, used to establish a dynamically scalable and erasable operator library;

[0129] The operator selection module is used to select the operators required for the task on the analysis task creation page;

[0130] The task formula text generation module is used to set the data source for each operator, convert all operators into independent operator objects, and generate the task formula text based on the connection relationship between all operator objects;

[0131] Database module, used to save task formula text into the database;

[0132] The target task execution module inputs the target data into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0133] In this embodiment, the operator definition specification and processing flow described in the present invention are combined Figure 2 and Figure 3 The technical modules shown can develop a set of scalable user-defined data analysis systems suitable for mixed scenarios of real-time data and non-real-time data, with memory and without memory. For example, after the developer completes the development of the required operators, the user can directly configure on the page that if the number of jumps of a certain signal a from 0 to 1 is greater than N times (N=10), an alarm prompt will be issued [the user Figure 2On the page shown in the S01 module, complete the following steps by dragging, dropping, and dropping. Figure 8 The topology shown and the corresponding data source settings will create the corresponding Figure 2 The task formula text shown in S02 is Greater(name3,Counter(name2,Down(name1,a)),10), for Figure 3 The S12 task execution engine is executed. When the result of the expression execution is 1 at a certain moment, Figure 3 The corresponding alarm prompt is performed in the result signal processing module S13 shown in FIG. 1 ; For example, in stock trading, when a signal a crosses a signal b and a signal c crosses a signal d at the same time, a prompt signal is issued [the user can Figure 2 On the page shown in the S01 module, complete the following steps by dragging, dropping, and dropping. Figure 7 After the topology shown is set up with the corresponding data source, the corresponding Figure 2 The task formula text And(name3,CrossOver(name1,a,b),CrossOver(name2,c,d)) shown in S02 is provided for Figure 3 The S12 task execution engine is executed. When the result of the expression execution is 1 at a certain moment, Figure 3 The corresponding prompt processing is performed in the result signal processing module S13].

[0134] Figure 2 and Figure 3 There are two independent processes for the overall processing of the system: data analysis task creation process and data analysis task execution process.

[0135] Figure 2 Provide software pages and corresponding system processing procedures (data analysis task creation procedures) for business personnel: Business personnel Figure 2 In the S01 module shown, based on actual business needs, the operator to be used is selected by dragging and dropping from the available operator library established in accordance with the method described in the present invention (for example, if the sum of two data points needs to be calculated, drag and drop the operator from the optional operator library). Figure 5 ), and set the data source required for the operator to be input for the selected operator (for example, if the two data required for the business to be calculated and are A and B respectively, then double-click the Add operator to pop up the signal selection window, set the operator calculation parameters for the Add operator to A and B respectively, and the system automatically adds a globally unique name to each selected operator to turn the operator into an operator object), and then the S02 module generates the corresponding task formula text according to the specifications described in this technical solution (according to the input and output connection relationship of each operator object on the page, the final execution formula to be calculated is generated, such as: Figure 7The input and output connection relationship of each operator is shown in the figure. The final execution formula is And(name3,CrossOver(name1,a,b),CrossOver(name2,c,d)). It is finally generated together with other information related to the task set by the user on the S01 page. Figure 2 The S03 module shown is saved together in the database to form a data analysis task table (the final data analysis task table is shown in Table 1, where the task execution formula column shown in Table 1 is the execution formula created by the S02 module in accordance with the specifications of this technical document) for Figure 3 The following data analysis task execution process is shown.

[0136] Figure 3 To perform the processing flow for the actual data analysis task, users Figure 2 After the data analysis task creation process is completed, it will automatically run without the need for further intervention and settings. When the analysis task is actually executed, the data analysis task execution process will first load, correspond, and process the data sources required for each actual execution task after the data analysis task is created in the S11 module. For example, if you need to calculate the sum of signals A and B, you need to find the data A and B required for the calculation from the actual data source and perform necessary processing so that the processed A and B data can meet the requirements of the subsequent execution engine [such as: converting from hexadecimal messages to decimal data] or other business needs [such as: only calculating the sum of A and B within a certain period of time]. The S12 module is used to access the actual processed data as the calculation parameters in the task execution formula, and the corresponding operator objects in the formula will be processed according to Figure 6 The processing flow shown calculates the value of each operator object one by one until the final formula value; the above final formula value will be given to the result signal processing module S13 for subsequent result processing, such as: saving the analysis results to the database, drawing analysis charts, etc.

[0137] In order to enable the entire system to be compatible with processing time-series business and non-time-series business, mixed business scenarios with memory and without memory, this solution is based on Figure 2 and Figure 3 When creating and executing analysis tasks, the process proposes Figure 6 The overall operator object operation processing logic and the corresponding operator establishment specifications ensure the flexibility of system use and the correctness of analysis task operation results.

[0138] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned data processing methods based on a memory rule engine is implemented.

[0139] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A data processing method based on a memory rule engine, characterized in that: The following steps are involved: Establish a dynamically scalable memory-erasable operator library; On the analysis task creation page, select the operator required for the task; Set up data sources for each operator, convert all operators into independent operator objects, and generate task formula text based on the connection relationship between all operator objects; Save the task formula text into the database; When processing target data, the target data is first input into the data preprocessing module for preprocessing to obtain first data; then the first data is input into the task execution engine, which loads the task formula text from the database and performs analysis and calculation on the first data to obtain a calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task; The internal processing logic of the operators in the dynamically expandable memory-erasable operator library includes the following steps: The calculation result object of the operator is recorded as result, and the calculation result object result is initialized and assigned according to the operator operation parameters and operator operation logic; If the environment parameter clear passed to the task execution engine during execution is 1, the value in the memory unit corresponding to the operator object is reset. Otherwise, the value in the memory unit corresponding to the operator object is retained for use in subsequent operations. Then, according to the objectName of the operator object, the memory value is retrieved from the memory unit corresponding to the operator object, recorded as memory[objName]. Perform calculations based on the memory value memory[objName] and the operator operation parameters passed in when the operator object is called, and update the obtained operation result value to the calculation result object result; After completing the update of the memory cell value according to the operator operation logic, the final calculation result object result is returned; The memory unit values are stored in a three-layer tree-like storage structure, wherein the first layer of the three-layer tree-like storage structure stores the names of each operator object, the second layer of the three-layer tree-like storage structure stores the names of memory parameters that require operator logic access, and the third layer of the three-layer tree-like storage structure stores the actual memory unit values.

2. The data processing method based on a memory rule engine according to claim 1, characterized in that: For memoryless operators, after initializing and assigning the calculation result object result, the operator operation parameters are directly used to perform calculations according to the operator operation logic, and the obtained calculation result value is updated to the calculation result object result, and the final calculation result object result is directly returned.

3. The data processing method based on a memory rule engine according to claim 1, characterized in that: When setting operator operation parameters, they are set according to the principle of the minimum number of operator operation parameters required by the operator and nested calculations of the same operator.

4. The data processing method based on a memory rule engine according to claim 1, characterized in that: When the expression used to call the Down operator is Down(name1,a), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the Down operator includes the following steps: Initialize the value field of the calculation result object result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of the calculation result object result; According to the operator object name of the falling edge operator Down, the parameter value of the last call is retrieved from the memory unit; If the parameter value of the last call is greater than the value of the operator operation parameter a, the value of the calculation result object result is updated to 1, otherwise the value of the calculation result object result is not updated; Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the operator operation parameter a, and return the calculation result object result.

5. The data processing method based on a memory rule engine according to claim 1, characterized in that: When the CrossOver operator is called with the expression CrossOver(name1, a, b), and the environment parameter clear passed in by the task execution engine is assigned a value of 0 during execution, the CrossOver operator's operation logic includes the following steps: Initialize the value field of the CrossOver operator's calculation result object result to 0, and assign the timestamp fields of the operator's operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), and result.value = 0. According to the operator object name of the CrossOver operator, the first parameter value and the second parameter value of the last call are retrieved from the memory unit; Perform calculations using the first and second parameter values and operator parameters a and b. If the first parameter value retrieved from the memory cell is less than or equal to the second parameter value and the value of the operator parameter a passed in this time is greater than the value of b, then the value of the calculation result object result is updated to 1; otherwise, the value of the calculation result object result is not updated. Update the memory value of the operator object in the memory unit of the CrossOver operator to the values of the two operator operation parameters a and b of this call, and return the calculation result object result.

6. The data processing method based on a memory rule engine according to claim 1, characterized in that: When the memoryless multiplication operator Mul is called with the expression Mul(name1, a, b), and the environment parameter clear passed in by the task execution engine is assigned a value of 0 during execution, the operation logic of the memoryless multiplication operator Mul includes the following steps: Initialize the value field of the result object result of the memoryless multiplication operator Mul to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), and result.value = 0; Use the value fields of operator parameters a and b to perform calculations, update result.value = a.value * b.value, and then directly return the value of the calculation result object result at this time.

7. The data processing method based on a memory rule engine according to claim 1, characterized in that: When the dynamically scalable memory-erasable operator library does not have the operator required for the task, the required operator is added to the scalable memory-erasable operator library according to the operator creation specification.

8. A data processing system based on a memory rule engine, characterized in that: A method for implementing a data processing method based on a memory rule engine according to any one of claims 1 to 7, comprising: Operator library establishment module, used to establish a dynamically scalable and erasable operator library; The operator selection module is used to select the operators required for the task on the analysis task creation page; The task formula text generation module is used to set the data source for each operator, convert all operators into independent operator objects, and generate the task formula text based on the connection relationship between all operator objects; Database module, used to save task formula text into the database; The target task execution module inputs the target data into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the data processing method based on the memory rule engine as described in any one of claims 1 to 7 is implemented.

Images