A data processing method and apparatus

Abstract

The application discloses a data processing method and device, and relates to the technical field of computers. A specific implementation manner of the method comprises the following steps: receiving a definition operation of a syntax rule and a syntax label of an index calculation expression; using the syntax rule to perform lexical analysis and syntax analysis on the index calculation expression, and obtaining a syntax analysis tree; creating an accessor to traverse the syntax analysis tree, and replacing the syntax label in a node in the syntax analysis tree with implementation logic code of the corresponding syntax rule by analyzing the node; determining an index identifier in the index calculation expression, obtaining index data corresponding to the index identifier, processing the index data in sequence according to the position of the node where the implementation logic code is located in the syntax analysis tree, obtaining processing result data, and returning the processing result data. In the implementation manner, for an index definer who can give a data expression, the index definer can efficiently define and efficiently execute a new index without writing redundant codes, and the learning cost and operation cost are relatively low.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology

[0002] In data statistical processing, there are scenarios where it is necessary to calculate derived indicators based on basic indicators. For example, calculating the GDP per capita of a region based on its GDP (Gross Domestic Product) and population data. Currently, this is mainly achieved through offline job pre-calculation and configuration scripts. However, these two methods have problems such as high deployment costs or high learning costs, and redundant and time-consuming code writing. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data processing method and apparatus that can at least solve the problems of high deployment costs, high learning costs, and redundant and time-consuming code writing in existing technologies for calculating derived indicators based on basic indicators.

[0004] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising:

[0005] The system receives the definition operations of syntax rules and syntax tags for the index calculation expression, and uses the syntax rules to perform lexical analysis and syntax analysis on the index calculation expression to obtain a syntax parse tree.

[0006] An accessor is created to traverse the syntax parse tree, and by parsing the nodes in the syntax parse tree, the syntax tags in the nodes are replaced with the implementation logic code of the corresponding syntax rules;

[0007] Determine the indicator identifier in the indicator calculation expression, obtain the indicator data corresponding to the indicator identifier, process the indicator data in sequence according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data and return it.

[0008] Optionally, the step of using the grammar rules to perform lexical and syntactic analysis on the index calculation expression to obtain a syntax parse tree includes:

[0009] The characters in the index calculation expression are obtained, and a lexical analyzer is used to perform lexical analysis on the sequence of characters to convert the character sequence into token symbols;

[0010] A parser is used to perform syntactic analysis on the sequence of token symbols to obtain the parse tree of the index calculation expression.

[0011] Optionally, the indicator identifier includes a basic indicator identifier and / or a derived indicator identifier, and determining the indicator identifier in the indicator calculation expression includes:

[0012] Based on the basic indicator identifier table, determine whether each indicator identifier is a basic indicator identifier;

[0013] For each derived indicator identifier in the indicator calculation expression, query the derived indicator calculation expression corresponding to each derived indicator identifier;

[0014] Replace each derived indicator identifier with the derived indicator calculation expression to obtain the target indicator calculation expression.

[0015] Optionally, obtaining the indicator data corresponding to the indicator identifier includes:

[0016] Check if there is indicator data in memory corresponding to the indicator identifier;

[0017] If it exists, retrieve the indicator data from memory; or

[0018] If it does not exist, the indicator data is retrieved from the database using the remote procedure call protocol.

[0019] Optionally, retrieving the indicator data from the database using a remote procedure call protocol includes:

[0020] Using the remote procedure call protocol, query the target data table in the database that corresponds to the indicator identifier;

[0021] Send a data query request to the target data table, the data query request including the indicator identifier, to receive the indicator data returned by the target data table.

[0022] Optionally, the indicator data is in key-value pair format, where the key is date and the key is a numerical value;

[0023] The sequential processing of the indicator data includes:

[0024] Determine the left operand, right operand, and operator; wherein, the operator corresponds to the implementation logic code, the left operand is located before and adjacent to the operator, and the right operand is located after and adjacent to the operator;

[0025] When both the left and right operands are mapped object types, perform operator operations on the values ​​with the same key name in the left and right operands; or

[0026] When the left operand is a mapped object type and the right operand is a numeric type, perform operator operations between the key value corresponding to each key name in the left operand and the right operand; or

[0027] When the left operand is a numeric type and the right operand is a mapped object type, perform operator operations on the key values ​​corresponding to each key name in the left operand and the right operand; or

[0028] When both the left and right operands are numeric types, perform operator operations on the left and right operands.

[0029] To achieve the above objectives, according to another aspect of the present invention, a data processing apparatus is provided, comprising:

[0030] The analysis module is used to receive the definition operations of the syntax rules and syntax tags of the index calculation expression, and use the syntax rules to perform lexical analysis and syntax analysis on the index calculation expression to obtain a syntax parse tree;

[0031] The parsing module is used to create accessors to traverse the syntax parse tree and, by parsing the nodes in the syntax parse tree, replace the syntax tags in the nodes with the implementation logic code of the corresponding syntax rules.

[0032] The calculation module is used to determine the indicator identifier in the indicator calculation expression, obtain the indicator data corresponding to the indicator identifier, process the indicator data in sequence according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data and return it.

[0033] Optionally, the analysis module is used for:

[0034] The characters in the index calculation expression are obtained, and a lexical analyzer is used to perform lexical analysis on the sequence of characters to convert the character sequence into token symbols;

[0035] A parser is used to perform syntactic analysis on the sequence of token symbols to obtain the parse tree of the index calculation expression.

[0036] Optionally, the indicator identifier includes a basic indicator identifier and / or a derived indicator identifier, and the calculation module is used for:

[0037] Based on the basic indicator identifier table, determine whether each indicator identifier is a basic indicator identifier;

[0038] For each derived indicator identifier in the indicator calculation expression, query the derived indicator calculation expression corresponding to each derived indicator identifier;

[0039] Replace each derived indicator identifier with the derived indicator calculation expression to obtain the target indicator calculation expression.

[0040] Optionally, the computing module is used for:

[0041] Check if there is indicator data in memory corresponding to the indicator identifier;

[0042] If it exists, retrieve the indicator data from memory; or

[0043] If it does not exist, the indicator data is retrieved from the database using the remote procedure call protocol.

[0044] Optionally, the computing module is used for:

[0045] Using the remote procedure call protocol, query the target data table in the database that corresponds to the indicator identifier;

[0046] Send a data query request to the target data table, the data query request including the indicator identifier, to receive the indicator data returned by the target data table.

[0047] Optionally, the indicator data is in key-value pair format, where the key is date and the key is a numerical value;

[0048] The computing module is used for:

[0049] Determine the left operand, right operand, and operator; wherein, the operator corresponds to the implementation logic code, the left operand is located before and adjacent to the operator, and the right operand is located after and adjacent to the operator;

[0050] When both the left and right operands are mapped object types, perform operator operations on the values ​​with the same key name in the left and right operands; or

[0051] When the left operand is a mapped object type and the right operand is a numeric type, perform operator operations between the key value corresponding to each key name in the left operand and the right operand; or

[0052] When the left operand is a numeric type and the right operand is a mapped object type, perform operator operations on the key values ​​corresponding to each key name in the left operand and the right operand; or

[0053] When both the left and right operands are numeric types, perform operator operations on the left and right operands.

[0054] To achieve the above objectives, according to another aspect of the present invention, a data processing electronic device is provided.

[0055] An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the data processing methods described above.

[0056] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the data processing methods described above.

[0057] According to the solution provided by the present invention, one embodiment of the invention has the following advantages or beneficial effects: 1. Users can independently and flexibly configure the syntax rules of the indicator calculation expression, and set a syntax tag after each syntax rule. IDEA can replace the syntax tag with the implementation logic code used to operate the syntax rule. Therefore, there is no need to search for the implementation logic again, and it can be used directly, which improves the efficiency of using the syntax rules; 2. It responds in a timely manner to whether the indicator in the indicator calculation expression is a basic indicator. In the case that the data of the derived indicator does not exist in memory, the derived indicator is replaced with the corresponding derived indicator calculation formula, presenting the formula superposition effect, thereby improving the final calculation accuracy. In some cases, all derived indicators can be replaced with the corresponding calculation formula, so as not to be affected by the existence of the derived indicator data; 3. It realizes the efficient definition and efficient execution of the calculation of derived indicators. For those who can provide the indicator definition with data expression, there is no need to write redundant code to achieve the purpose of independently developing new indicators, with low learning and operation costs.

[0058] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0059] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0060] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0061] Figure 2 This is a schematic flowchart of a specific data processing method according to an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the main modules of a data processing device according to an embodiment of the present invention;

[0063] Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0064] Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation

[0065] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0066] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0067] This section provides a detailed description of existing technologies. Currently, there are two common practices for calculating derived indicators based on basic indicators:

[0068] Method 1: Offline job pre-calculation, then pushing the calculation results to the business system's database. However, this method requires waiting a certain amount of time to see the results every time the calculation formula is modified, resulting in high deployment costs. Although the calculation time for each derived indicator job is relatively short, the waste of machine resources is still significant when the number of derived indicators to be calculated is too large.

[0069] Method 2: Configure script code for real-time calculations within the business system. Script code is also a computer programming language, involving concepts such as variables, functions, classes, and APIs (Application Programming Interfaces). The metric definer needs to be proficient in these concepts, resulting in a high learning curve and lengthy, time-consuming code for new metrics. Furthermore, the mathematical expressions for some new metrics provided by the metric definer may not all be coded and implemented.

[0070] The terms used in this plan are explained as follows:

[0071] Indicator: A set of statistical values ​​over time, such as the annual GDP of a region.

[0072] ANTLR4: ANTLR (Another Tool for Language Recognition) is a powerful cross-language parser that can be used to read, process, execute, or translate structured text or binary files. It is widely used to build languages, tools, and frameworks. ANTLR can generate a parser from a grammatical perspective that can build and traverse parse trees.

[0073] A parser, typically appearing as a compiler or interpreter, performs syntax checking and constructs a data structure composed of input words / tokens. A parser usually uses a lexer to separate individual tokens from the input character stream and uses the token stream as its input. In practice, parsers can be written manually or automatically generated using tools.

[0074] CommonTokenStream: The core class in ANTLR4, used to process tokens identified after lexical analysis.

[0075] Abstract Syntax Tree (AST): An abstract representation of source code structure. It represents the syntactic structure of a language in the shape of a tree and can be used for code syntax checking, code style checking, code formatting, code highlighting, code error reporting, and code auto-completion.

[0076] Remote Procedure Call (RPC) is a computer communication protocol. This protocol allows a program running on one computer to call a subroutine in another address space (typically a computer on an open network), as if calling a local program, without requiring additional programming for this interaction.

[0077] A Map is an object that maps keys to values. Maps cannot contain duplicate keys, and each key can map to at most one value. The three common Map implementations in Java are HashMap, TreeMap, and LinkedHashMap. To ensure the order of the keys, this solution preferably uses LinkedHashMap.

[0078] IDEA: A commercially available Java Integrated Development Environment (IDE) software tool.

[0079] See Figure 1 The diagram shows the main flowchart of a data processing method provided by an embodiment of the present invention, which includes the following steps:

[0080] S101: Receive the definition operation of the syntax rules and syntax tags of the index calculation expression, and use the syntax rules to perform lexical analysis and syntax analysis on the index calculation expression to obtain a syntax parse tree;

[0081] S102: Create an accessor to traverse the syntax parse tree, and replace the syntax tags in the nodes with the implementation logic code of the corresponding syntax rules by parsing the nodes in the syntax parse tree;

[0082] S103: Determine the indicator identifier in the indicator calculation expression, obtain the indicator data corresponding to the indicator identifier, process the indicator data in sequence according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data and return it.

[0083] In the above implementation, for step S101, IDEA provides an input interface for the user to input the indicator calculation expression, such as ($PAYEMS-lag($PAYEMS,1)), where PAYEMS is the indicator ID of [Monthly Total Employment in Region M]. It should be noted that the required symbols differ in different scenarios. In actual economic indicator ID naming, some consist of pure numbers, while others consist of numbers, underscores, and letters. To accommodate more situations as much as possible, a $ symbol is set before the economic indicator ID in the syntax.

[0084] The configuration interface allows users to customize the syntax rules and syntax tags for indicator calculation expressions, as shown in the example below:

[0085]

[0086] The `expr` command is a manual command-line counter used in UNIX / LINUX to evaluate expression variables, typically for integer values, but also for strings. `expr` expressions: separate each item with a space; place a backslash `\` before shell-specific characters (the software that provides the user interface); strings containing spaces and other special characters must be enclosed in quotation marks.

[0087] The above syntax definition supports calculations between indicators and other indicators, between an indicator and itself, and between an indicator and a constant. You can input indicator calculation expressions and use ANTLR4 to parse them to check for any errors in the syntax definition. It should be noted that ANTLR4 allows you to add a label starting with a hash symbol "#" after each syntax rule. This will be used during subsequent automatic code generation to achieve the final syntax logic by replacing each label with the implementation logic code.

[0088] For syntax rules, a syntax rule file can be provided for ANTLR4. This file uses the lexical definition rules from the ANTLR tool grammar definition to define all keywords, operators, and recognizable data types appearing in the system. Simultaneously, the syntax definition rules from the ANTLR tool grammar definition and the nesting of syntax rules define the structure and precedence of operators in the expression syntax. In computer science, grammar is the foundation of compiler principles; it is the method for describing a programming language and implementing its compiler.

[0089] While the steps for generating a parse tree using ANTLR4 are existing, the specific implementation logic differs for different syntax rules. Therefore, this solution uses user-defined syntax rules to perform lexical and syntactic analysis on the index calculation expression to correctly generate the parse tree for the index calculation expression. Specifically:

[0090] 1) Obtain the characters in the input indicator calculation expression;

[0091] 2) Use a lexer, combined with the ANTLR4 common token stream class CommonTokenStream, to perform lexical analysis on the index calculation expression; where lexical analysis refers to the process of converting a character sequence into a token in computer science, and the lexer is generally used by the parser.

[0092] 3) Place the tokens generated by the lexical analyzer into the buffer;

[0093] 4) Extract tokens from the buffer and feed them into the parser;

[0094] 5) Use the Expression Parser to perform syntax analysis on the sequence of token symbols, and finally obtain the syntax parse tree of the index calculation expression.

[0095] Use ANTLR4 to automatically generate code. For example, right-click IndexEngine.g4 in IDEA and select ConfigureANTLR Recognizer to generate code for the lexical analyzer, parser, syntax tree traverser, and syntax tree accessor.

[0096] For step S102, as described above, ANTLR4 can generate a syntax parse tree corresponding to the indicator calculation expression. Then, this syntax parse tree is traversed, and this solution uses Java code to correctly implement the expected behavior of the indicator calculation expression.

[0097] There are generally two approaches: listeners and visitors. As described above, a custom syntax label has been defined, so this solution prefers visitors to directly access the label. By creating an visitor, IndexEngineVisitor, the traversal of the syntax parse tree begins, and the result of the visitor's traversal (the result of the formula calculation) is returned.

[0098] This section describes the implementation details of the accessor IndexEngineVisitor. This class inherits from IndexEngineBaseVisitor (which is empty, so an accessor needs to be created) in the ANTLR4 automatically generated code. It parses the nodes in the syntax parse tree to replace the labels in the nodes with the implementation logic code of the corresponding syntax rules, thereby implementing the syntax rules of the index calculation expression. In practice, one node may correspond to one label, or multiple nodes may correspond to the same label.

[0099] For the aforementioned code, the labels that the accessor needs to parse mainly include:

[0100] visitId: Parsing indicator ID

[0101] visitAddSub: Parsing Addition and Subtraction

[0102] visitMulDiv: Analytic Multiplication and Division

[0103] visitLiteral: Deciphering Numbers

[0104] `visitLagFuncs`: Parses the `lag` function; the `Lag` function can retrieve the first N rows (Lag) of the same field as a separate column in a single query.

[0105] visitParen: parsing parentheses

[0106] For step S103, traditional expressions do not support calculations using arrays as variables, such as [1,2,3]. The indicator calculation expression in this solution supports array operations. The key to the indicator calculation expression lies in parsing the indicator ID, which is accomplished using the aforementioned `visitId(IndexEngineParser.IdContext ctx)` method. First, the indicator data needs to be obtained, specifically:

[0107] 1) Check if data for the metric ID exists in memory;

[0108] 2) If the data exists, return the data in memory directly; otherwise, call the `getByIndexId` method to retrieve the data for that metric ID from the database. The `getByIndexId` method calls a remote RPC method based on the metric ID to retrieve the metric data, allowing the remote method to be called as if it were a local method.

[0109] ① Based on the transmitted indicator ID, query the registry to find out which target data table the data for that indicator ID exists in;

[0110] ② Initiate a data query request to the target data table. The data query request includes the indicator ID. Finally, return the queried data to ANTLR4 in sorted by date for calculation.

[0111] 3) The above steps enable the indicator calculation expression to support array operations. The return type is Map.<String,Double> The type is Double, where Double represents the corresponding numerical value and String represents the date (day, month, quarter, year, etc.) of the numerical value, such as: 2020-04-01, 2020-10, 2020-Q1, 2020, etc.

[0112] It should be noted that the indicator data stored in memory in this solution is typically derived indicator data, while the basic indicator data is usually stored in a database and retrieved using getByIndexId. The derived indicator data in memory is usually automatically deleted after the indicator calculation expression has completed its calculation. However, if it is called by multiple indicator calculation expressions, it will be automatically deleted after all these indicator calculation expressions have completed their calculations.

[0113] Taking the aforementioned indicator calculation expression ($PAYEMS-lag($PAYEMS,1)) as an example, the specific operation process of obtaining the indicator data corresponding to the indicator ID and processing the indicator data in sequence according to the position of the node where the implementation logic code is located in the syntax parse tree is as follows:

[0114] 1. Query the data of $PAYEMS and store it in map1;

[0115] 2. The result of calculating lag($PAYEMS,1) is stored in map2. getLagIndexId contains the actual logic for calculating lag. To support the lag function, the visitLagFuncs function processes the following:

[0116] 1) Obtain the indicator ID and the backward offset value from lag($PAYEMS,1);

[0117] 2) Check if the data for the metric ID exists in memory; if not, query the database to fill the memory.

[0118] 3) Query the misaligned data based on the indicator ID and the value that is shifted backward.

[0119] For example, if a monthly indicator X starts collecting data from July 2021 to December 2021, then lag(X,1) will only have values ​​for 5 months, as shown in Table 1.

[0120] Table 1

[0121] Before misalignment After misalignment 2021-12 100 2021-12 99 2021-11 99 2021-11 101 2021-10 101 2021-10 102 2021-09 102 2021-09 97 2021-08 97 2021-08 96 2021-07 96

[0122] 3. Next, process the subtraction operands (similar to multiplication and division).

[0123] 1) Determine the operator, the left operand located before and adjacent to the operator, and the right operand located after and adjacent to the operator; wherein, one operator corresponds to one implementation logic code, which is the minus sign "-" in this embodiment;

[0124] 2) If the left operand is of type map, then check if the right operand is of type map.

[0125] A. If the right operand is a map type, then the values ​​with the same key in the two maps will be subtracted, that is, the values ​​with the same date will be subtracted.

[0126] B. If the right operand is a numeric type, then the value corresponding to each key of the left operand is subtracted from the numeric value.

[0127] The calculation results for both scenarios A and B above are still map.

[0128] 3) If the left operand is not a map type, check if the right operand is a map type.

[0129] A. If the right operand is of type map, then the value corresponding to each key of the left operand is subtracted from the value of the right operand. In this case, the result type is Map.

[0130] B. If the right operand is not of type map, then subtract the left and right operands. The result will be of type Double.

[0131] The method provided in this embodiment of the invention has at least the following advantages over the prior art:

[0132] 1. Users can independently and flexibly configure the syntax rules of indicator calculation expressions, and set syntax tags after each syntax rule. IDEA can replace the syntax tag with the implementation logic code used to operate the syntax rule. Therefore, there is no need to search for the implementation logic later, and it can be used directly, which improves the efficiency of syntax rule usage.

[0133] 2. Timely response to whether the indicator in the indicator calculation expression is a basic indicator. In the case that the data of the derived indicator does not exist in memory, the derived indicator is replaced with the corresponding derived indicator calculation formula, presenting a formula superposition effect, thereby improving the final calculation accuracy. In some cases, all derived indicators can be replaced with the corresponding calculation formula, so as not to be affected by the existence of derived indicator data.

[0134] 3. It enables efficient definition and execution of derived indicators, and for those who can provide the data expression for the indicator definition, there is no need to write redundant code to achieve the goal of independently developing new indicators, with low learning and operation costs.

[0135] See Figure 2 The diagram illustrates a specific data processing method according to an embodiment of the present invention, including the following steps:

[0136] S201: Receive the definition operation of the syntax rules and syntax tags of the index calculation expression, and use the syntax rules to perform lexical analysis and syntax analysis on the index calculation expression to obtain a syntax parse tree;

[0137] S202: Create an accessor to traverse the syntax parse tree, and replace the syntax tags in the nodes with the implementation logic code of the corresponding syntax rules by parsing the nodes in the syntax parse tree;

[0138] S203: Determine the indicator identifier in the indicator calculation expression, and based on the basic indicator identifier table, determine whether each indicator identifier is a basic indicator identifier;

[0139] S204: For each derived indicator identifier in the indicator calculation expression, query the derived indicator calculation expression corresponding to each derived indicator identifier;

[0140] S205: Replace each derived indicator identifier with the derived indicator calculation expression to obtain the target indicator calculation expression;

[0141] S206: Obtain the indicator data corresponding to each indicator identifier in the target indicator calculation expression, process the indicator data in sequence according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data and return it.

[0142] In the above embodiments, for steps S201, S202, and S206, please refer to... Figure 1 The descriptions of steps S101 to S103 shown will not be repeated here.

[0143] In the above embodiments, for steps S203 to S205, in actual operation, if the indicator calculation expression input by the user contains derived indicators, the derived indicators can be replaced with an expression composed of basic indicators to facilitate subsequent data acquisition. For example:

[0144] Indicator A + Indicator B → Indicator C;

[0145] Indicator C + Indicator D → Indicator E. Here, "+" simply means that the target indicator's data is generated from the data of multiple indicators, which is different from the addition operation.

[0146] Table 2 Basic Indicator Identification Table

[0147] Basic Indicator Labeling Table Indicator A Indicator B Indicator D

[0148] Table 3. Calculation Expressions for Derivative Indicators

[0149] Calculation formula for derived indicators Indicator A + Indicator B = Indicator C Indicator C - Indicator D = Indicator E Indicator B * Indicator D = Indicator F

[0150] If the user-input indicator calculation expression is (Indicator A + Indicator D * Indicator E), including Indicator A, Indicator D, and Indicator E, where, according to Table 2, Indicator A and Indicator D are basic indicators, while Indicator E is a derived indicator, according to Table 3, Indicator E needs to be replaced with the expression (Indicator C - Indicator D). Since Indicator C is also a derived indicator, it needs to be further replaced with the expression (Indicator A + Indicator B). Finally, the user-input indicator calculation expression is updated to a target indicator calculation expression consisting only of basic indicators.

[0151] (Indicator A + Indicator D * ((Indicator A + Indicator B) - Indicator D))

[0152] It should be noted that the above situation applies to cases where derived indicator data does not exist in memory. For example, if indicator C data does not exist, indicator C needs to be replaced with the corresponding expression according to Table 3. However, if indicator C data exists in memory, it is preferable not to replace it to avoid problems such as excessively long target indicator calculation expressions and calculation times. However, replacement can also be chosen, depending on the settings of the staff.

[0153] Here's a specific example: Region M typically only publishes the "Total Monthly Employment in Region M." The raw data is stored in a database, with the structure shown in Table 4 below, where PAYEMS is the indicator ID for the "Total Monthly Employment in Region M."

[0154] Table 4

[0155] PAYEMS,2021-08,147190,Thousands of Persons PAYEMS,2021-07,146821,Thousands of Persons PAYEMS,2021-06,145878,Thousands of Persons PAYEMS,2021-05,144940,Thousands of Persons PAYEMS,2021-04,144326,Thousands of Persons PAYEMS,2021-03,144057,Thousands of Persons PAYEMS,2021-02,143272,Thousands of Persons PAYEMS,2021-01,142736,Thousands of Persons

[0156] The user-defined calculation formula is: Newly added employment (m) = Total employment in region M (m) - Total employment in region M (m-1), where m is the month and (m-1) is the month preceding month m. However, this formula configuration is inconvenient. Therefore, the indicator data is referenced using $Indicator ID to participate in the formula calculation. The modified indicator calculation expression is: ($PAYEMS-lag($PAYEMS,1)).

[0157] The ANTLR4 parser is used to perform lexical and syntactic analysis on the index calculation expression ($PAYEMS-lag($PAYEMS,1)), resulting in a parse tree. Subsequent calculations are as follows:

[0158] 1. Query the data of $PAYEMS and store it in map1, and calculate the result of lag($PAYEMS,1) and store it in map2;

[0159] 2. Iterate through the keys of map1, query the value of the key in map1, subtract its value in map2, and calculate the result as a map.

[0160] Here is a more colloquial explanation:

[0161] Newly added employment (August 2021) = Total employment in region M (August 2021) - Total employment in region M (July 2021)

[0162] Newly added employment (July 2021) = Total employment in region M (July 2021) - Total employment in region M (June 2021)

[0163] ……

[0164] The calculation results are as follows:

[0165] "category":"2021-02",

[0166] "value":"536.0"

[0167] "category":"2021-03",

[0168] "value":"785.0"

[0169] "category":"2021-04",

[0170] "value":"269.0"

[0171] "category":"2021-05",

[0172] "value":"614.0"

[0173] "category":"2021-06",

[0174] "value":"938.0"

[0175] "category":"2021-06",

[0176] "value":"943.0"

[0177] "category":"2021-08",

[0178] "value":"369.0"

[0179] See Figure 3 The diagram shows a schematic representation of the main modules of a data processing apparatus 300 provided in an embodiment of the present invention, including:

[0180] Analysis module 301 is used to receive the definition operations of syntax rules and syntax tags for index calculation expressions, and use the syntax rules to perform lexical analysis and syntax analysis on the index calculation expressions to obtain a syntax parse tree;

[0181] The parsing module 302 is used to create an accessor to traverse the syntax parse tree and, by parsing the nodes in the syntax parse tree, replace the syntax tags in the nodes with the implementation logic code of the corresponding syntax rules.

[0182] The calculation module 303 is used to determine the indicator identifier in the indicator calculation expression, obtain the indicator data corresponding to the indicator identifier, process the indicator data in sequence according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data and return it.

[0183] In the apparatus of this invention, the analysis module 301 is used for:

[0184] The characters in the index calculation expression are obtained, and a lexical analyzer is used to perform lexical analysis on the sequence of characters to convert the character sequence into token symbols;

[0185] A parser is used to perform syntactic analysis on the sequence of token symbols to obtain the parse tree of the index calculation expression.

[0186] In the apparatus of this invention, the indicator identifier includes a basic indicator identifier and / or a derived indicator identifier. The calculation module 303 is used for:

[0187] Based on the basic indicator identifier table, determine whether each indicator identifier is a basic indicator identifier;

[0188] For each derived indicator identifier in the indicator calculation expression, query the derived indicator calculation expression corresponding to each derived indicator identifier;

[0189] Replace each derived indicator identifier with the derived indicator calculation expression to obtain the target indicator calculation expression.

[0190] In the apparatus of this invention, the computing module 303 is used for:

[0191] Check if there is indicator data in memory corresponding to the indicator identifier;

[0192] If it exists, retrieve the indicator data from memory; or

[0193] If it does not exist, the indicator data is retrieved from the database using the remote procedure call protocol.

[0194] In the apparatus of this invention, the computing module 303 is used for:

[0195] Using the remote procedure call protocol, query the target data table in the database that corresponds to the indicator identifier;

[0196] Send a data query request to the target data table, the data query request including the indicator identifier, to receive the indicator data returned by the target data table.

[0197] In the device implemented by the present invention, the index data is in key-value pair form, where the key name is date and the key value is numerical value;

[0198] The computing module 303 is used for:

[0199] Determine the left operand, right operand, and operator; wherein, the operator corresponds to the implementation logic code, the left operand is located before and adjacent to the operator, and the right operand is located after and adjacent to the operator;

[0200] When both the left and right operands are mapped object types, perform operator operations on the values ​​with the same key name in the left and right operands; or

[0201] When the left operand is a mapped object type and the right operand is a numeric type, perform operator operations between the key value corresponding to each key name in the left operand and the right operand; or

[0202] When the left operand is a numeric type and the right operand is a mapped object type, perform operator operations on the key values ​​corresponding to each key name in the left operand and the right operand; or

[0203] When both the left and right operands are numeric types, perform operator operations on the left and right operands.

[0204] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0205] Figure 4 An exemplary system architecture 400 to which embodiments of the present invention can be applied is shown, including terminal devices 401, 402, 403, network 404, and server 405 (this is merely an example).

[0206] Terminal devices 401, 402, and 403 can be various electronic devices with displays and support for web browsing, and have various communication client applications installed. Users can use terminal devices 401, 402, and 403 to interact with server 405 through network 404 to receive or send messages, etc.

[0207] Network 404 is a medium used to provide a communication link between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0208] Server 405 can be a server that provides various services. It should be noted that the methods provided in the embodiments of the present invention are generally executed by server 405, and correspondingly, the devices are generally set in server 405.

[0209] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0210] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing a terminal device of the present invention. Figure 5 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0211] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0212] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0213] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0214] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0215] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0216] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an analysis module, a parsing module, and a calculation module. The names of these modules do not necessarily limit the module itself; for example, a calculation module can also be described as a "data calculation module."

[0217] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0218] The system receives the definition operations of syntax rules and syntax tags for the index calculation expression, and uses the syntax rules to perform lexical analysis and syntax analysis on the index calculation expression to obtain a syntax parse tree.

[0219] An accessor is created to traverse the syntax parse tree, and by parsing the nodes in the syntax parse tree, the syntax tags in the nodes are replaced with the implementation logic code of the corresponding syntax rules;

[0220] Determine the indicator identifier in the indicator calculation expression, obtain the indicator data corresponding to the indicator identifier, process the indicator data in sequence according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data and return it.

[0221] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: The system receives the definition operations of syntax rules and syntax tags for the index calculation expression, and uses the syntax rules to perform lexical analysis and syntax analysis on the index calculation expression to obtain a syntax parse tree; wherein, the lexical analysis of the sequence of characters in the index calculation expression is performed using a lexical analyzer, the process of which includes: performing lexical analysis on the sequence of characters using a lexical analyzer and the public token flow class of the ANTLR4 tool; An accessor is created to traverse the parse tree and, by parsing the nodes in the parse tree, replaces the syntax tags in the nodes with the implementation logic code of the corresponding syntax rules; wherein, the accessor inherits from the base class accessor in the ANTLR4 automatically generated code, and the base class accessor is empty; Determine the indicator identifier in the indicator calculation expression, obtain the indicator data corresponding to the indicator identifier, process the indicator data sequentially according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data, and return it; wherein, the indicator data includes data before and after misalignment, and the sequential processing of the indicator data includes: Determine the left operand, right operand, and operator; wherein, the operator corresponds to the implementation logic code, the left operand is located before and adjacent to the operator, and the right operand is located after and adjacent to the operator; When both the left and right operands are mapped object types, perform operator operations on the values ​​with the same key name in the left and right operands; or When the left operand is a mapped object type and the right operand is a numeric type, perform operator operations between the key value corresponding to each key name in the left operand and the right operand; or When the left operand is a numeric type and the right operand is a mapped object type, perform operator operations on the key values ​​corresponding to each key name in the left operand and the right operand; or When both the left and right operands are numeric types, perform operator operations on the left and right operands.

2. The method according to claim 1, characterized in that, The process of using the grammar rules to perform lexical and syntactic analysis on the index calculation expression to obtain a syntax parse tree includes: The characters in the index calculation expression are obtained, and a lexical analyzer is used to perform lexical analysis on the sequence of characters to convert the character sequence into token symbols; A parser is used to perform syntactic analysis on the sequence of token symbols to obtain the parse tree of the index calculation expression.

3. The method according to claim 1, characterized in that, Indicator identifiers include basic indicator identifiers and / or derived indicator identifiers. Determining the indicator identifier in the indicator calculation expression includes: Based on the basic indicator identifier table, determine whether each indicator identifier is a basic indicator identifier; For each derived indicator identifier in the indicator calculation expression, query the derived indicator calculation expression corresponding to each derived indicator identifier; Replace each derived indicator identifier with the derived indicator calculation expression to obtain the target indicator calculation expression.

4. The method according to claim 1 or 3, characterized in that, The step of obtaining the indicator data corresponding to the indicator identifier includes: Check if there is indicator data in memory corresponding to the indicator identifier; If it exists, retrieve the indicator data from memory; or If it does not exist, the indicator data is retrieved from the database using the remote procedure call protocol.

5. The method according to claim 4, characterized in that, The step of retrieving the indicator data from the database using a remote procedure call protocol includes: Using the remote procedure call protocol, query the target data table in the database that corresponds to the indicator identifier; Send a data query request to the target data table, the data query request including the indicator identifier, to receive the indicator data returned by the target data table.

6. The method according to claim 1, characterized in that, The indicator data is in key-value pair format, with the key being the date and the key being a numerical value.

7. A data processing apparatus, characterized in that, include: The analysis module is used to receive the definition operations of syntax rules and syntax tags for the index calculation expression, and to perform lexical analysis and syntax analysis on the index calculation expression using the syntax rules to obtain a syntax parse tree; wherein, the lexical analyzer is used to perform lexical analysis on the sequence of characters in the index calculation expression, the process of which includes: using the lexical analyzer and the public token flow class of the ANTLR4 tool to perform lexical analysis on the sequence of characters; The parsing module is used to create accessors to traverse the syntax parse tree, and to replace the syntax tags in the nodes with the implementation logic code of the corresponding syntax rules by parsing the nodes in the syntax parse tree; wherein, the accessor inherits from the base class accessor in the ANTLR4 automatically generated code, and the base class accessor is empty; The calculation module is used to determine the indicator identifier in the indicator calculation expression, obtain the indicator data corresponding to the indicator identifier, process the indicator data sequentially according to the position of the node where the implementation logic code is located in the syntax parsing tree, obtain the processing result data, and return it; wherein, the indicator data includes data before and after misalignment, and the sequential processing of the indicator data includes: Determine the left operand, right operand, and operator; wherein, the operator corresponds to the implementation logic code, the left operand is located before and adjacent to the operator, and the right operand is located after and adjacent to the operator; When both the left and right operands are mapped object types, perform operator operations on the values ​​with the same key name in the left and right operands; or When the left operand is a mapped object type and the right operand is a numeric type, perform operator operations between the key value corresponding to each key name in the left operand and the right operand; or When the left operand is a numeric type and the right operand is a mapped object type, perform operator operations on the key values ​​corresponding to each key name in the left operand and the right operand; or When both the left and right operands are numeric types, perform operator operations on the left and right operands.

8. The apparatus according to claim 7, characterized in that, The indicator identifiers include basic indicator identifiers and / or derived indicator identifiers. The calculation module is used for: Based on the basic indicator identifier table, determine whether each indicator identifier is a basic indicator identifier; For each derived indicator identifier in the indicator calculation expression, query the derived indicator calculation expression corresponding to each derived indicator identifier; Replace each derived indicator identifier with the derived indicator calculation expression to obtain the target indicator calculation expression.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images