A method, device, and storage medium for processing statistical analysis scripts.

Abstract

This application provides a method, device, and storage medium for processing statistical analysis scripts, relating to the field of data processing technology. The method includes: acquiring a statistical analysis script to be processed, the script containing at least one statistical operation instruction; analyzing the script to construct an abstract syntax tree (API); the API includes at least: statistical nodes corresponding to each statistical operation instruction; the statistical nodes recording how the statistical operation instructions reference statistical data; when traversing to a statistical node corresponding to a statistical operation instruction in the API, obtaining the corresponding statistical data according to the reference method recorded in the statistical node; obtaining at least one functional interface of a statistical computing engine through a bridging layer; sending the statistical data to the functional interface to invoke the interface to execute the statistical operation instruction; and receiving the script execution result returned by the bridging layer. This application can reduce memory spikes during large-scale statistical data execution, reduce cross-process / cross-service data transfer, reduce repeated interface initialization, and improve script execution efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, and storage medium for processing statistical analysis scripts. Background Technology

[0002] Scripting languages ​​are widely used in various computer application scenarios such as automated processing, batch task execution, and software function expansion. Users can control the computer to execute preset operation processes by writing script instructions.

[0003] In statistical analysis applications, data analysts commonly use scripts to batch process data tables, call statistical functions, perform hypothesis testing and numerical optimization, etc., to complete data analysis tasks.

[0004] However, the statistical scripts in these technologies can only run in specific software and cannot be reused across platforms or applications. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a method, device, and storage medium for processing statistical analysis scripts, thereby decoupling the script analysis module from the statistical computing engine through a bridging layer, enabling the statistical computing engine to be deployed and reused independently.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a method for processing statistical analysis scripts, the method comprising: Obtain a statistical analysis script to be processed, wherein the statistical analysis script contains at least one statistical operation instruction; The statistical analysis script is analyzed to construct an abstract syntax tree; the abstract syntax tree includes at least: statistical nodes corresponding to each statistical operation instruction; the statistical nodes record how the statistical operation instructions reference the statistical data. When traversing to the statistical node corresponding to the statistical operation instruction in the abstract syntax tree, the corresponding statistical data is obtained according to the reference method recorded in the statistical node. At least one functional interface of the statistical computing engine is obtained through the bridging layer, and the statistical data is sent to the functional interface to call the functional interface to execute the statistical operation instruction. Receive the script execution result returned by the bridging layer.

[0007] In some embodiments, analyzing the statistical analysis script to construct an abstract syntax tree includes: A lexical analyzer is used to perform lexical analysis on the statistical analysis script, converting the character sequence contained in the statistical analysis script into a sequence of syntax elements; the sequence of syntax elements includes a data object reference descriptor used to locate the object to be analyzed; A parser is used to perform syntactic analysis on the sequence of syntactic elements to construct the abstract syntax tree; the statistical nodes in the abstract syntax tree contain the data object reference descriptors, which serve as a way to reference statistical data objects.

[0008] In some embodiments, the syntax elements include: lexical units and syntax units, the lexical analysis includes multi-word identifier merging processing, and the step of using a lexical analyzer to perform lexical analysis on the statistical analysis script, converting the character sequence contained in the statistical analysis script into a sequence of syntax elements, includes: The statistical analysis script is traversed using a lexical analyzer with lexical units as the traversal unit. When the first word of the current lexical unit is the same as that of at least one candidate multi-word identifier in the preset multi-word identifier mapping table, the subsequent word sequence of the candidate multi-word identifier is matched with the lexical units to be traversed after the current lexical unit in turn. If a target multi-word identifier among the candidate multi-word identifiers successfully matches the lexical unit to be traversed, then the current lexical unit and the lexical unit to be traversed are merged into a syntax unit corresponding to the target multi-word identifier and output. If a match fails, the current lexical unit will be output.

[0009] In some embodiments, the step of using a parser to perform syntactic analysis on the sequence of syntactic elements and constructing the abstract syntax tree includes: The statement structure in the sequence of grammatical elements is parsed using a recursive descent algorithm; When the recursive descent algorithm parses an expression, it calls the operator precedence parsing algorithm to parse the expression. Based on the parsing results, the abstract syntax tree is constructed; wherein, when the data object reference descriptor is parsed during the construction of the abstract syntax tree, a data object reference node containing data object identification information is generated in the abstract syntax tree.

[0010] In some embodiments, each node in the abstract syntax tree has a corresponding node type, which includes: basic type and statistical semantic type.

[0011] In some embodiments, before analyzing the statistical analysis script to construct an abstract syntax tree, the method further includes: Lexical, syntactic, and semantic checks are performed on the statistical analysis script. If an error is found during the check, stop the subsequent checks and output the error message.

[0012] In some embodiments, the semantic check includes: checking if a function name exists; and when an error is found during the check, stopping subsequent checks and outputting error information includes: When it is detected that the target function name does not exist in the preset function library, a fuzzy matching step is performed on the target function name according to the preset normalized index; Based on the normalized index, at least one function name with the highest similarity to the target function name is determined from the preset function library as a candidate function name; The output includes an error message containing the name of the candidate function.

[0013] In some embodiments, when the script execution result indicates an error, the script execution result includes: the error location and the error code; the method further includes: Obtain the user-preset target natural language type; Based on the target natural language type, read the error message template associated with the error code from the attribute file of the target natural language type; The placeholder in the error message template is replaced with the actual position based on the error location to generate the target error message.

[0014] Secondly, one embodiment of this application provides a schematic diagram of processing a statistical analysis script, wherein the apparatus includes: an acquisition module, an analysis module, and a receiving module; wherein: The acquisition module is used to acquire the statistical analysis script to be processed, which contains at least one statistical operation instruction. The analysis module is used to analyze statistical analysis scripts to construct an abstract syntax tree; the abstract syntax tree includes at least: statistical nodes corresponding to each statistical operation instruction; and the statistical nodes record how the statistical operation instructions reference the statistical data. The acquisition module is specifically used to obtain the corresponding statistical data according to the reference method recorded in the statistical node when traversing to the statistical operation instruction in the abstract syntax tree. It obtains at least one functional interface of the statistical computing engine through the bridging layer and sends the statistical data to the functional interface to call the functional interface to execute the statistical operation instruction. The receiving module is used to receive the script execution results returned by the bridging layer.

[0015] In some embodiments, the analysis module is specifically used to perform lexical analysis on the statistical analysis script using a lexical analyzer, converting the character sequence contained in the statistical analysis script into a sequence of grammatical elements; the sequence of grammatical elements includes a data object reference descriptor used to locate the object to be statistically analyzed; the grammatical analyzer performs grammatical analysis on the sequence of grammatical elements to construct an abstract syntax tree; the statistical nodes in the abstract syntax tree contain the data object reference descriptor as a way to reference statistical data objects.

[0016] In some embodiments, the grammatical elements include: lexical units and grammatical units. Lexical analysis includes multi-word identifier merging processing and an analysis module, specifically used to traverse the statistical analysis script using a lexical analyzer with lexical units as traversal units; when the first word of the current lexical unit is the same as that of at least one candidate multi-word identifier in the preset multi-word identifier mapping table, the subsequent word sequences of the candidate multi-word identifiers are sequentially matched with the lexical units to be traversed after the current lexical unit; if a target multi-word identifier among the candidate multi-word identifiers is successfully matched with the lexical unit to be traversed, the current lexical unit and the lexical unit to be traversed are merged into the grammatical unit corresponding to the target multi-word identifier and output; if the matching fails, the current lexical unit is output.

[0017] In some embodiments, the analysis module is specifically used to parse the statement structure in the sequence of syntactic elements according to the recursive descent algorithm; when the recursive descent algorithm parses an expression, it calls the operator precedence parsing algorithm to parse the expression; based on the parsing result, it constructs an abstract syntax tree; wherein, when constructing the abstract syntax tree, when the data object reference descriptor is parsed, a data object reference node containing data object identification information is generated in the abstract syntax tree.

[0018] In some embodiments, each node in the abstract syntax tree has a corresponding node type, which includes: basic type and statistical semantic type.

[0019] In some embodiments, the analysis module is specifically used to perform lexical checks, syntax checks, and semantic checks on the statistical analysis script; when an error is found during the checks, subsequent checks are stopped and error information is output.

[0020] In some embodiments, semantic checking includes: checking whether a function name exists; an analysis module, specifically configured to perform a fuzzy matching step on the target function name according to a preset normalized index when it is detected that the target function name does not exist in a preset function library; based on the normalized index, determining at least one function name with the highest similarity to the target function name from the preset function library as a candidate function name; and outputting an error message containing the candidate function name.

[0021] In some embodiments, when the script execution result indicates an error, the script execution result includes: error location and error code. The acquisition module is specifically used to acquire the user-preset target natural language type; read the error information template associated with the error code from the attribute file of the target natural language type according to the target natural language type; and replace the placeholder in the error information template with the actual location according to the error location to generate the target error message.

[0022] Thirdly, another embodiment of this application provides a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0023] Fourthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0024] The beneficial effects of this application are as follows: The method for processing statistical analysis scripts provided in this application involves obtaining a statistical analysis script to be processed, wherein the statistical analysis script contains at least one statistical operation instruction; analyzing the statistical analysis script to construct an abstract syntax tree; the abstract syntax tree includes at least: statistical nodes corresponding to each statistical operation instruction; the statistical nodes record the reference method of the statistical operation instruction to the statistical data; when traversing to the statistical node corresponding to the statistical operation instruction in the abstract syntax tree, obtaining the corresponding statistical data according to the reference method recorded in the statistical node, obtaining at least one functional interface of the statistical computing engine through a bridging layer, sending the statistical data to the functional interface to call the functional interface to execute the statistical operation instruction; and receiving the script execution result returned by the bridging layer. The method provided in this application parses the statistical analysis script into an abstract syntax tree that records the reference method of the statistical data, and dynamically obtains the statistical data according to the reference method during execution, avoiding preloading all data into memory and reducing the memory peak during large-scale data execution. Furthermore, this application uses a bridging layer to call an independently deployed statistical computing engine to perform statistical operations. This isolates the script analysis module from the statistical computing engine through the bridging layer, allowing only statistical data to be transmitted rather than the complete script context, thus reducing the amount of data transferred across processes and / or services. Additionally, this application uses the bridging layer to manage the caching of functional interface instances. Multiple calls to the same statistical operation can reuse the same functional interface, thereby reducing the overhead of repeated interface initialization and improving the overall execution efficiency of the statistical analysis script. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a method for processing a statistical analysis script according to an embodiment of this application is shown. Figure 2 This specification illustrates a method for constructing an abstract syntax tree according to an embodiment. Figure 3 A flowchart illustrating the processing of a statistical analysis script according to another embodiment of this application is shown; Figure 4 This illustration shows a schematic diagram of a statistical analysis script processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0028] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0029] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0030] The technical solution provided in this application is applicable to scenarios requiring statistical analysis script processing, such as grade statistics, business intelligence, financial analysis, and medical statistics. Users need to write statistical analysis scripts based on their data analysis needs. These scripts contain at least one statistical operation instruction, and the computer needs to execute the scripts to complete data processing, statistical calculations, and result output.

[0031] To facilitate understanding of the embodiments of this application, some terms used in this application are explained below: Double-Checked Locking (DCL) lazy loading: a thread-safe singleton object creation pattern for high-concurrency scenarios. When initially retrieving an object, it first checks if the instance exists without locking. If not, it enters a synchronized block and checks the instance state again. Only when the second check confirms that the instance has not been created is the object initialized in a specified way and assigned to a volatile variable. Subsequent retrieval operations directly return the initialized instance. This ensures thread safety while avoiding the performance overhead of traditional synchronized patterns, achieving a balance between lazy initialization and efficient access.

[0032] Recursive descent algorithm: A top-down parsing method that writes a parsing function for each non-terminal symbol in the grammar rules. The functions recursively call each other to parse nested grammar structures. It is suitable for processing statement structures (such as If...Then...End, For...Next, assignment statements, etc.).

[0033] Operator precedence resolution algorithm: A resolution algorithm specifically designed to handle expression precedence (Pratt resolution). By assigning a precedence value to each operator, the order of operations is determined according to the precedence during the resolution process. It is suitable for handling expressions involving arithmetic operations, logical operations, comparison operations, and other operations that involve operator precedence.

[0034] Figure 1 A flowchart illustrating a method for processing a statistical analysis script according to an embodiment of this application is shown. Figure 1 As shown, the statistical analysis script processing method provided in this application may include steps 110-140, wherein: Step 110: Obtain the statistical analysis script to be processed. The statistical analysis script contains at least one statistical operation instruction.

[0035] A statistical analysis script is a text file containing statistical operation instructions, used to describe analytical tasks such as data processing, statistical calculation, and result output.

[0036] Statistical analysis scripts are scripts written by users for statistical analysis objects. The statistical objects are the actual data content that the statistical analysis script needs to process when it is executed. The forms of the statistical objects include, but are not limited to, data tables, matrices, associative arrays, data sequences, etc.

[0037] Statistical operation commands are statements in statistical analysis scripts used to perform statistical calculations, including but not limited to: data table operations (open, filter, merge), statistical function calls (mean, standard deviation, test), matrix operations, etc.

[0038] The statistical script processing method provided in this application can be deployed in various application scenarios. In a web service scenario, users can access a web front-end interface, write or input the statistical analysis script to be processed and its corresponding statistical objects in the text edit box of the web front-end interface, and click the execute button to trigger the retrieval of the statistical analysis script and statistical objects for subsequent analysis. In an integrated development scenario, the script analysis module is deployed as a plugin. Users select the statistical objects to be processed from multiple candidate data tables in the editor, write the script in the editor, and click the run button. The script analysis module retrieves the statistical analysis script to be processed and its associated statistical objects from the editor buffer. In a microservice scenario, the script analysis module is deployed as an independent microservice. Other services send script execution requests to the script analysis module via remote procedure calls or message queues. The script analysis module retrieves the sent statistical analysis script and statistical objects and then performs subsequent analysis operations.

[0039] Step 120: Analyze the statistical analysis script to construct an abstract syntax tree.

[0040] The abstract syntax tree (AST) includes at least the following: statistical nodes corresponding to each statistical operation instruction. Each statistical node records how the statistical operation instruction references the statistical data. The reference method describes how the statistical data is located and retrieved. For example, if the reference method is "column name 'blood pressure'", it means the statistical data is retrieved by searching for the "blood pressure" column in the current statistical object; if the reference method is "file path 'data.csv'", it means the statistical data is retrieved by reading the statistical object from the specified path.

[0041] The statistical object must contain at least one statistical data item. For example, when the statistical object is a student statistics table, it can include multiple statistical data items: grade statistics, height statistics, weight statistics, etc.

[0042] An Abstract Syntax Tree (ABS) is a tree-like data structure used to represent the syntactic structure of statistical analysis scripts. Each node in an ABS corresponds to a syntactic element in the statistical analysis script, such as a variable, operator, or function call. Each node contains a node type and a node value. The node type can include ordinary types and statistical semantic types.

[0043] As an example, basic types may include: numeric type [XSL Number]: representing numeric values ​​as IEEE 754 double-precision floating-point numbers, and uniformly representing missing values ​​as NaN (Not a Number); string type [XSL String]; and empty value type [XSL Empty].

[0044] Statistical semantic types can include: Data Table type [XSL Data Table]: representing a data table in columnar storage and supporting column reference syntax; Column type [XSL Column]; Row State type [XSL Row State]: representing row state with 16-bit integer bit fields; Function type [XSL Function]; Matrix type [XSL Matrix]: representing a matrix as a two-dimensional double-precision floating-point array, using 1-based indexing; Class type [XSL Class]; Object type [XSL Object].

[0045] In some embodiments, the node type may also include extended types, such as namespace type [XSLNamespace], pattern matching type [XSL Pattern], large object type [XSL LOB], missing value type [XSLMissing], and UI component description type [XSL UID escription].

[0046] Each statistical node records not the specific numerical value of the statistical data, but rather the way the statistical data is referenced: for example, a column reference node records "need to find the column named 'blood pressure' in the current data table." A function call node records "need to call the function named Col Mean, whose parameter is a reference to the 'blood pressure' column." A variable node records "need to find the variable named 'average score'." This approach eliminates the need to load and process actual data during the analysis phase of the statistical analysis script; it only records the location information of the statistical data in the abstract node tree, thereby saving memory resources and supporting on-demand loading.

[0047] Step 130: When traversing to the statistical node corresponding to the statistical operation instruction in the abstract syntax tree, obtain the corresponding statistical data according to the reference method recorded in the statistical node, obtain at least one functional interface of the statistical computing engine through the bridging layer, and send the statistical data to the functional interface to call the functional interface to execute the statistical operation instruction.

[0048] The bridge layer is used to connect the script analysis module and the statistical computing engine, and is responsible for functions such as obtaining functional interfaces, data transmission, data type conversion, and call forwarding.

[0049] The functional interfaces serve as the entry points for the statistical computing engine, with each interface corresponding to a type of statistical computing capability. Examples include: basic statistics interfaces (mean calculation, variance calculation, etc.), probability distribution interfaces (normal distribution calculation, cumulative distribution function, quantile function, t-distribution calculation, etc.), hypothesis testing interfaces, and numerical optimization interfaces (maximizing and minimizing the objective function based on the Nelder-Mead algorithm, returning an associative array containing the optimal values ​​of each variable, the convergence state, and the optimal function value).

[0050] The evaluator module traverses the abstract syntax tree. When it accesses a statistics node, it first identifies the node type. For example, if the statistics node is a column reference node, it reads the column name of the record (such as "blood pressure") from the statistics node. Subsequently, the evaluator module, based on the data table associated with the current context, searches for and extracts all values ​​of that column by column name as the corresponding statistics.

[0051] As an example, when the statistics node is a variable node, the value corresponding to the variable name is found in the current scope to obtain the corresponding statistics. When the statistics node is a file reference node, the data file is read according to the recorded file path and parsed into a data table to obtain the corresponding statistics.

[0052] In the embodiments of this application, the evaluator module can traverse the abstract syntax tree through a double dispatch mechanism. Specifically, when a node is reached, the node itself dispatches its node type to the evaluator, which then dispatches the corresponding specific evaluation method based on the node type. Compared to the chained judgment method of traversing node types one by one in related technologies, the above-mentioned double dispatch mechanism improves the execution efficiency of the evaluator.

[0053] The method provided in this application supports multiple variable access modes in its evaluator module. For example, when the user does not declare an access mode, the default access mode of "first searching the local scope, then searching the global scope" is used for variable resolution. When the user specifies "::name", the specified variable is forced to be searched directly in the global scope, skipping the local scope. When the user specifies "here:name", the specified variable is forced to be searched in the current local scope. This design allows users who write statistical analysis scripts to precisely control the access scope of variables, avoiding naming conflicts and accidental overwriting.

[0054] After obtaining the corresponding statistical data, the evaluator module sends a call request to the bridging layer. The bridging layer selects the corresponding function interface based on the type of statistical operation instruction. Subsequently, the bridging layer sends the statistical operation instruction and the obtained statistical data to the function interface to execute the corresponding statistical operation.

[0055] In some embodiments, the bridge layer converts the statistical operation instructions and the acquired statistical data into a data type that the statistical computing engine can recognize, and sends the converted data to the functional interface to call the functional interface to perform statistical calculations based on the converted data.

[0056] Because the script analysis module and the statistical computing engine are isolated through a bridging layer, the script analysis module cannot directly access the statistical computing engine. It can only obtain and call at least one functional interface in the statistical computing engine through the bridging layer. In other words, the script analysis module and the statistical computing engine can be deployed independently in different processes or on different servers, thereby achieving elastic isolation of computing resources and enabling the statistical computing engine to be upgraded and replaced independently.

[0057] In one embodiment of this application, the functional interface adopts a lazy loading mechanism to load the functional interface. The functional interface is only created and loaded on demand when it needs to be called, thereby reducing memory peaks.

[0058] In some embodiments, to further ensure thread safety, a double-checked locking (DCL) lazy loading mechanism can be used to load the functional interface, thereby ensuring that only one interface instance is created in a multi-threaded environment.

[0059] Step 140: Receive the script execution result returned by the bridging layer.

[0060] After the bridge layer obtains the raw calculation results returned by the statistical calculation engine, it converts the raw calculation results into script execution results that can be recognized by the scripting language, and returns the script execution results to the script analysis module.

[0061] In the embodiments of this application, the script analysis module is used to acquire statistical analysis scripts, perform lexical analysis and syntax analysis on the statistical analysis scripts, construct abstract syntax trees, and traverse execution scheduling, etc. The bridge layer is used for acquiring and managing functional interfaces, bidirectional conversion between data types between the script analysis side and the statistical analysis engine side, and forwarding calls, etc.

[0062] The method for processing statistical analysis scripts provided in this application involves: acquiring a statistical analysis script to be processed, which contains at least one statistical operation instruction; analyzing the statistical analysis script to construct an abstract syntax tree (API); the API includes at least: statistical nodes corresponding to each statistical operation instruction; the statistical nodes recording how the statistical operation instruction references statistical data; when traversing to a statistical node corresponding to a statistical operation instruction in the API, obtaining the corresponding statistical data according to the reference method recorded in the statistical node; obtaining at least one functional interface of the statistical computing engine through a bridging layer; sending the statistical data to the functional interface to call the functional interface to execute the statistical operation instruction; and receiving the script execution result returned by the bridging layer. The method provided in this application parses the statistical analysis script into an API that records the reference method of statistical data, and dynamically obtains statistical data according to the reference method during execution, avoiding preloading all data into memory and reducing memory spikes during large-scale data execution. Furthermore, this application uses a bridging layer to call an independently deployed statistical computing engine to execute statistical operations, isolating the script analysis module and the statistical computing engine through the bridging layer, transmitting only statistical data rather than the complete script context, thus reducing the amount of data transmitted across processes and / or services. Furthermore, this application uses a bridging layer to cache and manage functional interface instances, allowing multiple calls to the same statistical operation to reuse the same functional interface, thereby reducing the overhead caused by repeated interface initialization and improving the overall execution efficiency of the statistical analysis script.

[0063] In some embodiments, after obtaining the initial statistical analysis script uploaded by the user and before constructing the abstract syntax tree, lexical checks, syntax checks, and semantic checks can be performed on the statistical analysis script; if an error is found during the checks, subsequent checks are stopped and error information is output.

[0064] After obtaining the initial statistical analysis script, the script analysis module performs static analysis to determine if any errors are found. If errors are detected, subsequent checks are paused, and error information is returned to the user. The user then modifies the initial statistical analysis script based on the error information, prompting the script analysis module to continue checking the modified script until no errors are found. At this point, the error-free initial statistical analysis script is obtained as the final statistical analysis script.

[0065] The static analysis performed on the initial statistical analysis script includes, but is not limited to, lexical checks, syntactic checks, and semantic checks. These checks can be performed in stages or in parallel.

[0066] The initial statistical analysis script is divided into several parts: Lexical checking, Syntax checking, and Semantic checking. Lexical checking includes checking for illegal characters, unterminated strings, encoding problems, mismatched parentheses, missing operands, and grammatical errors. Semantic checking includes verifying the existence of function names, the validity of the number of parameters, the definition of variable scope, and the depth of nested loops.

[0067] In some embodiments, semantic checking includes: checking if a function name exists. When it is detected that the target function name does not exist in a preset function library, a fuzzy matching step is performed on the target function name according to a preset normalized index; based on the normalized index, at least one function name with the highest similarity to the target function name is determined from the preset function library as a candidate function name; and an error message containing the candidate function name is output.

[0068] In the embodiments of this application, the preset function library can be maintained by a built-in function registration module. The built-in function registration module contains multiple registration modules, each covering different fields, such as: mathematics, probability distribution, statistics, matrix linear algebra, date and time, file input and output, finance, pattern matching, numerical optimization, data table operation function domains, etc.

[0069] The built-in function registration module maintains three independent indexes: an exact name index, an alias index, and a normalized index. The exact name index is used for efficient function lookup in O(1) constant time at runtime. The alias index is used to support multiple name calls for the same function, such as "average" and "Average" both pointing to "Col Mean". The above method of maintaining multiple indexes ensures that the required function can be quickly located and called when executing statistical analysis scripts, thereby reducing function lookup time and avoiding invalid calls.

[0070] When checking whether the target function name exists in the preset function library, you can first search the preset function library using the exact name index. If the exact name index does not find the corresponding function name, then search the preset function library using the alias index. If the alias index also does not find the corresponding function name, then determine whether there is a candidate function name corresponding to the target function name in the preset function library based on the normalized index.

[0071] As an example, suppose the initial statistical analysis script contains "average (score table)". Static analysis of the script determines that the target function is named "average". First, an exact name index is used to search for "average" in the pre-defined function library, but it is not found. Then, an alias index is used to search for "average" in the pre-defined function library. The corresponding mapping function "Col Mean" is found, indicating that the target function exists in the pre-defined function library, and subsequent checks continue.

[0072] If the initial statistical analysis script contains "Col Meen (grade sheet)," a static analysis is performed on the script. The target function name is determined to be "Col Meen," but no corresponding function is found using either the exact name index or the alias index. Therefore, a normalized index is used to calculate the similarity. The script then searches the preset function library for the candidate function name closest to the target function name, which is "Col Mean," and outputs the error message: "Function 'Col Meen' does not exist. Did you mean to use 'Col Mean'?"

[0073] Figure 2 This specification illustrates a method for constructing an abstract syntax tree according to an embodiment, such as... Figure 2 As shown, step 120 may include: steps 121-122, wherein: Step 121: Use a lexical analyzer to perform lexical analysis on the statistical analysis script, converting the character sequence contained in the statistical analysis script into a sequence of syntax elements.

[0074] The sequence of syntactic elements includes data object reference descriptors used to locate the objects to be counted. A lexical analyzer can be used to perform lexical analysis on the statistical analysis script to identify consecutive combinations of characters as syntactic units.

[0075] In the embodiments of this specification, a greedy merging method of multi-word identifiers can be used to identify and merge multi-word function names in the statistical analysis script. This not only reduces memory transfer overhead but also avoids ambiguity in the subsequent syntax analysis stage.

[0076] In some embodiments, grammar elements include lexical units and syntax units. When determining the sequence of grammar elements, a lexical analyzer iterates through the statistical analysis script, using lexical units as the traversal unit. If the first word of the current lexical unit is the same as that of at least one candidate multi-word identifier in a preset multi-word identifier mapping table, the subsequent word sequences of the candidate multi-word identifiers are sequentially matched with the lexical units to be traversed after the current lexical unit. During the matching process, only horizontal whitespace characters, such as spaces and tabs, are allowed to be skipped at any given time; cross-line matching is not allowed.

[0077] If a candidate multi-word identifier matches the target multi-word identifier and the lexical unit to be traversed, the current lexical unit and the lexical unit to be traversed are merged into the syntax unit corresponding to the target multi-word identifier and output; if the match fails, the current lexical unit is output as a lexical unit of ordinary identifier type.

[0078] As an example, taking the statistical analysis script "For Each Row dt = 1" as an example, the candidate multi-word identifiers with "For" as the first word in the preset multi-word identifier mapping table include: ["For","Each","Row"] (corresponding to the syntax unit FOR_EACH_ROW) and ["For","Each"] (corresponding to the syntax unit FOR_EACH).

[0079] The lexical analyzer scans and analyzes the script unit by unit. When the current lexical unit is "For", the lexical analyzer checks the preset multi-word identifier mapping table and finds that "For" is the first word of the candidate multi-word identifiers ["For","Each","Row"] and ["For","Each"].

[0080] The lexical analyzer saves the current traversal position and then attempts to match words in descending order of word count: first, it attempts to match ["For","Each","Row"], which succeeds. The three lexical units For, Each, and Row are merged into a single syntax unit FOR_EACH_ROW and output. It continues traversing the subsequent lexical unit "dt = 1". If this lexical unit does not exist in the mapping table, it is output normally.

[0081] Finally, we obtain the syntax unit sequence [FOR_EACH_ROW] [dt =1] corresponding to "For Each Row dt = 1".

[0082] In the embodiments of this specification, when matching multi-word identifiers, the lexical analyzer first saves the current scan position (i.e., the state triple: character index, row number, column number), and then attempts to match candidate multi-word identifiers sequentially from most to least word. If a match with one of the candidate multi-word identifiers fails, it returns to the previously saved position and attempts to match the next shorter candidate multi-word identifier. If a match with all candidate multi-word identifiers fails, it returns to the saved current scan position and treats the first word as a regular identifier. This processing method ensures that regardless of whether the match is successful or unsuccessful, the lexical analyzer can return to the current scan position before the start of the matching process, without losing characters or misaligning them, thus guaranteeing the correctness and determinism of the lexical analysis.

[0083] After analysis by the lexical analyzer, an ordered sequence of syntax elements is output. Each unit in the sequence corresponds to a minimal syntactic component in the statistical analysis script, such as an identifier, keyword, function name, operator, etc.

[0084] Step 122: Use a parser to perform syntactic analysis on the sequence of syntactic elements and construct an abstract syntax tree.

[0085] The statistical nodes in the abstract syntax tree contain reference descriptors for the data objects, serving as a way to reference statistical data objects. A parser can be used to perform syntactic analysis on the sequence of syntactic elements to organize the sequence into a tree structure, labeling each node with type information to generate the abstract syntax tree. The abstract syntax tree specifically includes statistical nodes to represent statistical operation instructions, including function call nodes (e.g., [Col Mean]), column reference nodes (e.g., [:blood pressure]), variable nodes (e.g., [average score]), and assignment nodes (e.g., [=]).

[0086] In the embodiments of this application, the statement structure in the sequence of grammatical elements is parsed using a recursive descent algorithm; when the recursive descent algorithm parses an expression, an operator precedence parsing algorithm is called to parse the expression; based on the parsing result, an abstract syntax tree is constructed. Specifically, when constructing the abstract syntax tree, if a data object reference descriptor is parsed, a data object reference node containing data object identification information is generated in the abstract syntax tree.

[0087] The parser first determines the parsing function corresponding to each statement structure in the sequence of grammatical elements according to the recursive descent algorithm. For example, it calls the `parseAssignment()` function when it encounters an assignment statement, the `parseIfStatement()` function when it encounters an `If` keyword, and the `parseForStatement()` function when it encounters a `For` keyword. Each parsing function then identifies and processes each grammatical component in the sequence of grammatical elements according to the grammatical rules.

[0088] When the recursive descent parsing process encounters a position where an expression needs to be parsed, such as the expression on the right side of the equals sign in an assignment statement, the expression in the condition part of an If statement, or the boundary expression of a loop statement, the operator precedence parsing algorithm is called to parse the expression.

[0089] The operator precedence resolution algorithm maintains a precedence table, supporting 10 precedence levels for assignment, logical operations, comparison, string concatenation, addition / subtraction, multiplication / division, matrix element-level operations, exponentiation, message sending, and postfix operations. For example, multiplication / division has higher precedence than addition / subtraction, and comparison operators have lower precedence than addition / subtraction. Furthermore, the expression is organized into a tree structure conforming to the precedence rules through recursive processing. After expression parsing, a recursive descent algorithm is used to attach the parsed result as a child node to the current statement node. The recursive descent algorithm and the operator precedence resolution algorithm work alternately to complete the parsing of the syntax element sequence and generate a complete abstract syntax tree.

[0090] Figure 3This illustration shows a flowchart of a statistical analysis script processing method according to another embodiment of this application. When the script execution result indicates an error, the script execution result includes: the error location and the error code, such as... Figure 3 As shown, the processing method for the statistical analysis script may further include steps 150-170, wherein: Step 150: Obtain the user-preset target natural language type.

[0091] The target natural language type is the user's preset, which is the type of natural language expected to be used when receiving error message information, such as English, Chinese, French, Russian, Vietnamese, etc.

[0092] The target natural language type can be a language preference preset by the user in the system configuration, or it can be determined through operating system language settings, browser language settings, configuration files, or by the user's active selection.

[0093] Step 160: Based on the target natural language type, read the error message template associated with the error code from the attribute file of the target natural language type.

[0094] The property file can be a text file storing key-value pairs, used to store error message templates in different languages. The error message template includes a string template associated with the error code, containing placeholders (such as {line}, {column}, {name}), which are used to replace the actual parameter values ​​at runtime to generate the final error message.

[0095] In the embodiments of this application, each error type is assigned an 8-bit numeric code. The first 4 bits identify the functional module to which the error belongs (such as the lexical module, syntax module, semantic module, etc.), and the last 4 bits identify the error sequence number within that module. For example, 2005 represents the 5th error in the syntax module: mismatched parentheses.

[0096] Step 170: Replace the placeholders in the error message template with the actual locations based on the error locations to generate the target error message.

[0097] Error location refers to the specific location where an error occurs in the statistical analysis script. It typically includes the line number and column number where the error occurred, provided by the state triple recorded during the lexical or syntax analysis phase.

[0098] The above method of generating and outputting target error message information in the target natural language type based on the user's preset target natural language type is convenient for understanding the cause of the error and reduces the user's understanding cost.

[0099] For example, when executing a statistical script that only involves two columns of data in a data table containing a large number of rows and columns, the relevant technology requires loading the entire data table. The method provided in this application only loads the referenced two columns of data based on the column reference descriptors in the statistical nodes. The bridging layer only transmits the column arrays corresponding to the referenced two columns of data, thereby reducing memory usage and cross-process transmission volume.

[0100] Based on the same inventive concept, this application also provides a statistical analysis script processing device corresponding to the statistical analysis script processing method. Since the principle of the device in this application to solve the problem is similar to the statistical analysis script processing method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0101] Figure 4 This illustration shows a schematic diagram of a statistical analysis script processing method provided in an embodiment of this application, such as... Figure 4 As shown, the device includes: an acquisition module 210, an analysis module 220, and a receiving module 230; wherein: The acquisition module 210 is used to acquire the statistical analysis script to be processed, which contains at least one statistical operation instruction. Analysis module 220 is used to analyze statistical analysis scripts to construct an abstract syntax tree; the abstract syntax tree includes at least: statistical nodes corresponding to each statistical operation instruction; the statistical nodes record how the statistical operation instructions reference the statistical data. The acquisition module 210 is specifically used to obtain the corresponding statistical data according to the reference method recorded in the statistical node when traversing to the statistical node corresponding to the statistical operation instruction in the abstract syntax tree, obtain at least one functional interface of the statistical computing engine through the bridge layer, and send the statistical data to the functional interface to call the functional interface to execute the statistical operation instruction. The receiving module 230 is used to receive the script execution results returned by the bridging layer.

[0102] In some embodiments, the analysis module 220 is specifically used to perform lexical analysis on the statistical analysis script using a lexical analyzer, converting the character sequence contained in the statistical analysis script into a sequence of grammatical elements; the sequence of grammatical elements includes a data object reference descriptor used to locate the object to be statistically analyzed; the grammatical analyzer performs grammatical analysis on the sequence of grammatical elements to construct an abstract syntax tree; the statistical nodes in the abstract syntax tree contain the data object reference descriptor as a way to reference statistical data objects.

[0103] In some embodiments, the grammatical elements include: lexical units and grammatical units. Lexical analysis includes multi-word identifier merging processing. The analysis module 220 is specifically used to use a lexical analyzer to traverse the statistical analysis script with lexical units as the traversal unit. When the first word of the current lexical unit is the same as that of at least one candidate multi-word identifier in the preset multi-word identifier mapping table, the subsequent word sequences of the candidate multi-word identifiers are matched with the lexical units to be traversed after the current lexical unit in turn. If there is a target multi-word identifier among the candidate multi-word identifiers that matches the lexical unit to be traversed, the current lexical unit and the lexical unit to be traversed are merged into the grammatical unit corresponding to the target multi-word identifier and output. If the matching fails, the current lexical unit is output.

[0104] In some embodiments, the analysis module 220 is specifically used to parse the statement structure in the sequence of syntactic elements according to the recursive descent algorithm; when the recursive descent algorithm parses an expression, it calls the operator precedence parsing algorithm to parse the expression; and constructs an abstract syntax tree based on the parsing result; wherein, when constructing the abstract syntax tree, when the data object reference descriptor is parsed, a data object reference node containing data object identification information is generated in the abstract syntax tree.

[0105] In some embodiments, each node in the abstract syntax tree has a corresponding node type, which includes: basic type and statistical semantic type.

[0106] In some embodiments, the analysis module 220 is specifically used to perform lexical checks, syntax checks, and semantic checks on the statistical analysis script; when an error is found during the checks, subsequent checks are stopped and error information is output.

[0107] In some embodiments, semantic checking includes: checking whether a function name exists; the analysis module 220 is specifically used to perform a fuzzy matching step on the target function name according to a preset normalized index when it is detected that the target function name does not exist in the preset function library; based on the normalized index, determine at least one function name with the highest similarity to the target function name from the preset function library as a candidate function name; and output an error message containing the candidate function name.

[0108] In some embodiments, when the script execution result indicates an error, the script execution result includes: error location and error code. The acquisition module 210 is specifically used to acquire the user-preset target natural language type; read the error information template associated with the error code from the attribute file of the target natural language type according to the target natural language type; and replace the placeholder in the error information template with the actual location according to the error location to generate the target error message.

[0109] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0110] This application also provides a computer device. Figure 5 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 5 As shown, the system includes: a processor 61, a memory 62, and optionally, a bus 63. The memory 62 stores machine-readable instructions executable by the processor 61. When the computer device is running, the processor 61 communicates with the memory 62 via the bus 63. When the machine-readable instructions are executed by the processor 61, the aforementioned... Figures 1-3 The steps of processing the statistical analysis script for any item.

[0111] This application also provides a computer-readable storage medium storing a computer program, which is executed by a processor. Figures 1-3 The steps of processing the statistical analysis script for any item.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for processing statistical analysis scripts, characterized in that, The method includes: Obtain a statistical analysis script to be processed, wherein the statistical analysis script contains at least one statistical operation instruction; The statistical analysis script is analyzed to construct an abstract syntax tree; the abstract syntax tree includes at least: statistical nodes corresponding to each statistical operation instruction; the statistical nodes record how the statistical operation instructions reference the statistical data. When traversing to the statistical node corresponding to the statistical operation instruction in the abstract syntax tree, the corresponding statistical data is obtained according to the reference method recorded in the statistical node. At least one functional interface of the statistical computing engine is obtained through the bridging layer, and the statistical data is sent to the functional interface to call the functional interface to execute the statistical operation instruction. Receive the script execution result returned by the bridging layer.

2. The method according to claim 1, characterized in that, The step of analyzing the statistical analysis script to construct an abstract syntax tree includes: A lexical analyzer is used to perform lexical analysis on the statistical analysis script, converting the character sequence contained in the statistical analysis script into a sequence of syntax elements; the sequence of syntax elements includes a data object reference descriptor used to locate the object to be analyzed; A parser is used to perform syntactic analysis on the sequence of syntactic elements to construct the abstract syntax tree; the statistical nodes in the abstract syntax tree contain the data object reference descriptors, which serve as a way to reference statistical data objects.

3. The method according to claim 2, characterized in that, The grammatical elements include: lexical units and grammatical units. The lexical analysis includes multi-word identifier merging processing. The step of using a lexical analyzer to perform lexical analysis on the statistical analysis script, converting the character sequence contained in the statistical analysis script into a sequence of grammatical elements, includes: The statistical analysis script is traversed using a lexical analyzer with lexical units as the traversal unit. When the first word of the current lexical unit is the same as that of at least one candidate multi-word identifier in the preset multi-word identifier mapping table, the subsequent word sequence of the candidate multi-word identifier is matched with the lexical units to be traversed after the current lexical unit in turn. If a target multi-word identifier among the candidate multi-word identifiers successfully matches the lexical unit to be traversed, then the current lexical unit and the lexical unit to be traversed are merged into a syntax unit corresponding to the target multi-word identifier and output. If a match fails, the current lexical unit will be output.

4. The method according to claim 2, characterized in that, The step of using a parser to perform syntactic analysis on the sequence of syntactic elements and constructing the abstract syntax tree includes: The statement structure in the sequence of grammatical elements is parsed using a recursive descent algorithm; When the recursive descent algorithm parses an expression, it calls the operator precedence parsing algorithm to parse the expression. Based on the parsing results, the abstract syntax tree is constructed; wherein, when the data object reference descriptor is parsed during the construction of the abstract syntax tree, a data object reference node containing data object identification information is generated in the abstract syntax tree.

5. The method according to claim 1, characterized in that, Each node in the abstract syntax tree has a corresponding node type, which includes: basic type and statistical semantic type.

6. The method according to claim 1, characterized in that, Before analyzing the statistical analysis script to construct the abstract syntax tree, the method further includes: Lexical, syntactic, and semantic checks are performed on the statistical analysis script. If an error is found during the check, stop the subsequent checks and output the error message.

7. The method according to claim 6, characterized in that, The semantic check includes: checking if the function name exists; and when an error is found, stopping subsequent checks and outputting error information, including: When it is detected that the target function name does not exist in the preset function library, a fuzzy matching step is performed on the target function name according to the preset normalized index; Based on the normalized index, at least one function name with the highest similarity to the target function name is determined from the preset function library as a candidate function name; The output includes an error message containing the name of the candidate function.

8. The method according to claim 1, characterized in that, When the script execution result indicates an error, the script execution result includes: the error location and the error code. The method further includes: Obtain the user-preset target natural language type; Based on the target natural language type, read the error message template associated with the error code from the attribute file of the target natural language type; The placeholder in the error message template is replaced with the actual position based on the error location to generate the target error message.

9. A computer device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the computer device is running, are executed by the processor to perform the steps of the processing method of the statistical analysis script as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the processing method of the statistical analysis script as described in any one of claims 1-8.

Images