A big data analysis method, device and application supporting SAS language parsing and execution

Abstract

The present application proposes a big data analysis method, device and application supporting SAS language parsing and execution, which identifies key words in the SAS code to be run and performs word segmentation processing according to the key words to obtain the word segmentation result, wherein for the macro code field in the word segmentation result, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library, wherein for the macro variable in the word segmentation result, the macro variable is stored in the system macro variable library, and the custom local variable is stored in the custom macro variable library, and the word segmentation result is gradually placed in the pre-running code pool according to the code sequence of the SAS code to be run; the code in the code pool to be run is segmented sentence by sentence with semicolons to obtain single-sentence code for execution, so as to realize the parsing and execution of SAS without relying on SAS software.

Description

Technical Field

[0001] The present application relates to the field of code analysis, and in particular to a big data analysis method, device and application that supports SAS language analysis and execution. Background Art

[0002] SAS is a programming language for statistical analysis and data processing. It was developed by SAS Software Institute and is widely used in many fields such as scientific research, finance, healthcare, government, etc. SAS can efficiently process large-scale data sets and perform operations such as data cleaning, conversion, and merging. For example, when processing massive amounts of financial transaction data, it can quickly filter out data of a specific time period and a specific transaction type, and perform format conversion and integration to provide an accurate data basis for subsequent analysis; and it covers various statistical analysis methods, such as descriptive statistics, regression analysis, cluster analysis, factor analysis, etc. Taking medical research as an example, it can be used to analyze patients' clinical data, explore the relationship between diseases and various factors (such as age, gender, living habits, etc.) through regression analysis, and provide a basis for the diagnosis and treatment of diseases. Therefore, it is often used in risk assessment, portfolio analysis, and market trend forecasting in the financial field, and in disease monitoring, clinical research, and medical quality assessment in the healthcare field.

[0003] The syntax structure of SAS language includes data step (DATA step) for data input, processing and output, procedure step (PROC step) that provides various statistical analysis and data processing procedures, and macro language that allows users to define and use macros. However, currently only SAS software on the market is compatible with SAS language, which limits the use of SAS language.

[0004] Therefore, there is an urgent need for a solution that can convert and execute SAS language and realize free compilation of SAS language to meet the needs of massive data analysis. Summary of the invention

[0005] The present application provides a big data analysis method, device and application that supports SAS language parsing and execution, so as to solve the problem in the prior art that SAS language cannot be compatible and freely compiled by software other than SAS software, and realizes massive data analysis by parsing SAS language and combining SAS language.

[0006] To achieve the above objectives, in the first aspect, this solution provides a big data analysis method that supports SAS language parsing and execution, including the following steps:

[0007] Obtaining a SAS code to be executed, wherein the SAS code to be executed is represented by a SAS language;

[0008] Identify the key words in the SAS code to be run and perform word segmentation according to the key words to obtain the word segmentation results. For the macro code field in the word segmentation results, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library. For the macro variables in the word segmentation results, the macro variables are stored in the system macro variable library, and the custom local variables are stored in the custom macro variable library.

[0009] The word segmentation results are gradually put into the pre-running code pool according to the code sequence of the SAS code to be run. For the macro code field in the word segmentation result, the corresponding code is obtained from the system mac library or the custom mac library. For the macro variables in the word segmentation result, the variable values ​​are obtained from the system macro variable library or the custom macro variable library.

[0010] The codes in the code pool to be run are split sentence by sentence with semicolons to obtain single-sentence codes, and the meaning of each code is analyzed to obtain the running mode. When the same function word is recognized to appear again in the code pool to be run, the code segment consisting of the single-sentence code between the two same function words is executed, and the executed code segment is cleared after execution; when the end instructions corresponding to the same function word are recognized to appear in pairs, all code segments are executed, and the executed code segments are cleared after execution.

[0011] Secondly, this solution provides a big data analysis platform that supports SAS language parsing and execution, and compiles the code in C++ language, including the following:

[0012] A SAS code acquisition unit, used for acquiring a SAS code to be executed, wherein the SAS code to be executed is represented by a SAS language;

[0013] The word segmentation unit is used to identify the key words in the SAS code to be run and perform word segmentation processing according to the key words to obtain the word segmentation results. For the macro code field in the word segmentation results, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library. For the macro variables in the word segmentation results, the macro variables are stored in the system macro variable library, and the custom local variables are stored in the custom macro variable library;

[0014] The macro code processing unit is used to gradually put the word segmentation results into the pre-running code pool according to the code sequence of the SAS code to be run, wherein for the macro code field in the word segmentation result, the corresponding code is obtained from the system mac library or the custom mac library, wherein for the macro variable in the word segmentation result, the variable value is obtained from the system macro variable library or the custom macro variable library;

[0015] The execution unit is used to divide the codes in the code pool to be run sentence by sentence with semicolons to obtain single-sentence codes, analyze the meaning of each code sentence to obtain the running mode, and when the same function word is recognized to appear again in the code pool to be run, execute the code segment composed of the single-sentence codes between two identical function words, and clear the executed code segment after execution; when the end instructions corresponding to the same function word are recognized to appear in pairs, execute all code segments, and clear the executed code segments after execution.

[0016] In a third aspect, the present solution provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process. The process includes the big data analysis method supporting SAS language parsing and execution mentioned in the first aspect.

[0017] Different from the prior art, the main contributions and innovations of the present invention are as follows:

[0018] This solution implements the parsing and execution of the SAS language through the C++ language, filling the gap in SAS code compilation in the domestic mainstream data statistical analysis software. It can realize the functions of SAS without relying on SAS software, providing possibilities for the application and migration of SAS code, breaking the dependence on specific SAS software, and expanding the usage scenarios of SAS code.

[0019] Compared with existing open source data statistical analysis software (such as Python / R), the solution of the present application can effectively reduce memory usage, overcome some limitations of hardware configuration and code efficiency, ensure that the program can still run smoothly when processing massive data, significantly shorten the data processing time, improve operating efficiency, and solve the technical difficulties in processing large data sets on a single computer. It breaks through the limitations of ordinary open source statistical analysis software in processing large data sets (such as a single data set cannot be opened if it exceeds 10G), and can open, compile and calculate large data sets exceeding the conventional size, greatly expanding the data processing capabilities of a single machine.

[0020] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 This is a flowchart of the big data analysis method that supports SAS language parsing and execution provided by this solution.

[0023] Figure 2 This is a logical diagram of the big data analysis method that supports SAS language parsing and execution provided by this solution.

[0024] Figure 3 This is a schematic diagram of the operation logic of the big data analysis platform that supports SAS language parsing and execution provided by this solution.

[0025] Figure 4 This is the structural framework diagram of the big data analysis platform provided by this solution that supports SAS language parsing and execution.

[0026] Figure 5 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] Embodiment 1

[0030] like Figure 1 As shown, the embodiment of the present application provides a big data analysis method supporting SAS language parsing and execution, including the following steps:

[0031] Obtaining a SAS code to be executed, wherein the SAS code to be executed is represented by a SAS language;

[0032] Identify the key words in the SAS code to be run and perform word segmentation according to the key words to obtain the word segmentation results. For the macro code field in the word segmentation results, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library. For the macro variables in the word segmentation results, the macro variables are stored in the system macro variable library, and the custom local variables are stored in the custom macro variable library.

[0033] The word segmentation results are gradually put into the pre-running code pool according to the code sequence of the SAS code to be run. For the macro code field in the word segmentation result, the corresponding code is obtained from the system mac library or the custom mac library. For the macro variables in the word segmentation result, the variable values ​​are obtained from the system macro variable library or the custom macro variable library.

[0034] The codes in the code pool to be run are split sentence by sentence with semicolons to obtain single-sentence codes, and the meaning of each code is analyzed to obtain the running mode. When the same function word is recognized to appear again in the code pool to be run, the code segment consisting of the single-sentence code between the two same function words is executed, and the executed code segment is cleared after execution; when the end instructions corresponding to the same function word are recognized to appear in pairs, all code segments are executed, and the executed code segments are cleared after execution.

[0035] The big data analysis method provided by this solution that supports SAS language parsing and execution can implement SAS language parsing without relying on SAS software, and then realize the statistics and analysis of massive data sets, solving the problem that open source data statistical analysis software (Python / R) takes a long time to process massive data, occupies a large amount of memory, and has low operating efficiency.

[0036] Specifically, in the step of "obtaining SAS code to be run", SAS code refers to the SAS code generated during the use of SAS software, and the SAS code is expressed in SAS language. In traditional solutions, such SAS code can only be edited and executed in SAS, and cannot be run in other statistical analysis software, which greatly limits the application of SAS code. This solution aims to solve the technical difficulties of SAS code compiled in SAS language, so as to achieve compatibility of SAS code expressed in SAS language on other statistical analysis platforms.

[0037] In addition, after obtaining the SAS code to be run, this solution creates a temporary file path, calls the system mac library and system macro variable library, and creates a custom mac library and a custom macro variable library to provide the necessary basis and support for the operation of the software. The temporary file path is expressed as a temporary file storage directory, which is used to store temporary data or intermediate results generated during the program running process. It is cleared after the parsing process is completed to release system resources and prevent temporary data accumulation from occupying too much storage space.

[0038] It should be explained that the system mac library includes predefined macros and functions to implement common and complex operations or functions. By calling the system mac library, the program can directly use these built-in functions without rewriting code, which improves programming efficiency. For example, it may contain macros for data processing, statistical calculations, file operations, etc.; the system macro variable library stores system predefined macro variables, which contain information related to the system environment, operating status, default settings, etc. Calling the system macro variable library allows the program to obtain this information so that it can be configured or adjusted accordingly according to the system environment. For example, obtain the system's current date, time, operating system type and other information for operations such as logging and generating file names in a specific format.

[0039] The custom mac library allows users to define their own macros and macro variables according to specific needs to achieve personalized function expansion. Users can encapsulate some frequently used code snippets into macros and store them in the custom mac library, which is convenient for calling in different program parts or different projects to improve the reusability of the code. For example, complex calculations or data processing steps for specific business logic can be defined as macros. The custom macro variable library provides users with a space to store and manage custom macro variables. Users can define and use their own macro variables in the program, which can be used to store intermediate results, configuration parameters, user-specific settings, etc. during the program operation. For example, according to user input or dynamic conditions during program operation, set the value of the custom macro variable, and then perform different operations according to the value of these macro variables in subsequent code.

[0040] In the step of "identifying the key words in the SAS code to be executed and performing word segmentation processing according to the key words to obtain the word segmentation results", the key words include function words and macro code fields, wherein the function words include data definition words, operation logic words, keywords, variable names, character strings and numbers, and the macro code field is indicated by starting with %.

[0041] Specifically, data definition words include data and proc, where data is used to define a data set and is the key identifier for starting a data set creation or operation in the SAS language. It indicates that the subsequent code will involve operations such as data set definition, data reading, writing or processing. For example, "data mydata;" means creating a data set named "mydata", in which data can be input, edited and analyzed later; proc is used to call various data processing procedures and statistical analysis procedures. It is usually followed by a specific procedure name, such as "proc means" for calculating descriptive statistics of a data set, and "proc reg" for regression analysis. "Proc" plays a role in guiding different data processing and analysis functions, and is an important functional word for implementing complex data analysis tasks.

[0042] Operational logic words include algorithmic operators and logical function words, where algorithmic operators include but are not limited to symbols such as "+", "-", "×", "÷" and so on that operate on numerical data, and logical function words include but are not limited to ";", which is used to indicate the end of a statement in SAS language. It serves to separate different code statements, so that the compiler can correctly identify and parse each independent instruction.

[0043] Keywords include program logic function words, including but not limited to "if", "else", "then", etc., which are reserved words with specific meanings and uses in the SAS language and are used to control the structure and flow of the program or perform specific operations. For example, "if" is used for conditional judgment, "then" is used to perform a specific operation when the condition is met, and "else" is used to perform another operation when the condition is not met.

[0044] A variable name is a variable used to identify and reference a variable in a data set. It represents a specific data column in a data set or an identifier used to store data during program execution. For example, "age", "income", "name", etc. can all be used as variable names. Data can be read, assigned, calculated, and analyzed through variable names. Variable names are identifiers of data objects that are directly operated during data processing and analysis.

[0045] A string is represented as "abx 12 aa". In SAS, a string is a sequence of characters enclosed in quotation marks. Strings can be used to represent text data, such as character variable values, comments, file paths, etc. in a data set. For example, when processing data containing text information, such as a customer's name, address, etc., strings are used to store and manipulate these text data. In the code, strings can be passed as parameters to functions, used for conditional judgments (such as judging whether strings are equal), or displayed in output results.

[0046] The number is represented as "123.45", which represents numeric data and is used as a specific value in calculations, assignments, comparisons, and other operations. For example, the value of a numeric variable is stored in a data set, such as age, income, temperature, etc.; it is used as a constant in the calculation process, such as "new_variable = 123.45 * old_variable;"; it is used as a comparison value in conditional judgments, such as "if variable>123.45 then...". Numbers are one of the indispensable basic elements in data processing and analysis, and are used to represent and process various quantitative relationships.

[0047] The macro code field starts with %, which is represented as "%xxx". Macro instructions starting with "%" are special elements in the SAS language used to achieve code reuse, automation, and extended functions. Macro instructions can define complex code logic, accept parameters, and be called multiple times in the program. For example, "% macro mymacro;... % mend;" defines a macro named "mymacro". In the program, you can call this macro through "% mymacro;" to execute the code block defined in the macro. Macro instructions greatly improve programming efficiency, reduce the writing of repeated codes, and can achieve different functions according to different parameter values.

[0048] In the step of "Storing mac fields in the system mac library", the mac fields include %macro, %if / %else, %substr, and %do.

[0049] Specifically, in the step of "Storing the mac field in the system mac library", for "%if / %else", when entering the macro %macro, if the keyword %if is located for the first time, it is marked as a=0 and stored word by word in the temporary pool of the vocabulary library. If %if is encountered again, it is marked as a+1; if %else %if is located, a remains unchanged; if %else is located, it is determined whether a is 0. When a>0, a-1 is added and it continues to be stored word by word in the temporary pool of the vocabulary library. When a=0, the content of the temporary pool of the vocabulary library is stored in the sub-macro corresponding to the macro in the corresponding custom mac library and the temporary pool of the vocabulary library is cleared.

[0050] Suppose a macro contains the following code: "% if&var1 = 1 % then % do; % if&var2 = 2 %then % do;... % end; % else % if&var3 = 3 % then % do;... % end; % else;... %end;". During the processing, when the first % if is encountered, a = 0, and words such as "&var1 = 1" are stored in the temporary pool of the vocabulary library; when the second % if is encountered, a = 1, and subsequent words are stored; when % else % if is encountered, a remains 1; when the first % else is encountered, since a = 1, a is reduced by 1 and becomes 0, and words are stored; when the second %else is encountered, since a is already 0 at this time, the content in the temporary pool of the vocabulary library is stored in the corresponding position, and the temporary pool of the vocabulary library is cleared.

[0051] In addition, in the step of "Storing the mac field in the system mac library", for "%do", when entering the macro %macro, if the keyword %do is reached for the first time, it is marked as b=0 and stored word by word in the temporary pool of the vocabulary library, and if %do is followed by %while, the sas statement is executed when the condition is true, and the condition is checked before the loop; if %do is followed by %until, it stops when the condition is true, checks the condition after the loop, and stores the conditional statement in the temporary pool of the vocabulary library; if the keyword %do is located again, it is marked as b+1, and if the keyword %end is located, it is determined whether b is 0. If b>0, b-1 is stored word by word in the temporary pool of the vocabulary library at the same time. If b=0, the content of the temporary pool of the vocabulary library is stored in the sub-macro corresponding to the macro in the corresponding custom mac library and the temporary pool of the vocabulary library is cleared.

[0052] For example, the code is "% do i = 1 % to 10; % if&var>i % then % do;... % end; % doj = 1 % to 5;... % end; % end;". When the first % do is encountered, b = 0, and words such as "i = 1 % to10" are stored; when the inner % do is encountered, b = 1, and related words continue to be stored; when the first % end is encountered, since b = 1, b is reduced by 1 and becomes 0, and words continue to be stored; when the second % end is encountered, because b is already 0 at this time, the content in the temporary word pool is stored in the corresponding position and cleared.

[0053] In the step "Store the mac field in the system mac library", when the mac field is %substr, extract the substring in the string () in "ABC%substr()". For example, when intercepting text data, extracting key information, etc., you can use %substr to obtain a substring at a specific position and length, and then use the extracted substring for subsequent calculations, comparisons, or assignments.

[0054] In the step of "Storing custom fields in the custom mac library", the custom fields are represented as "%macro SELF_MAC;...%mend". Specifically, when the custom field is located, SELF_MAC is stored in the custom mac library. If another %macro appears after macro and before %mend, the second macro needs to be stored in the corresponding sub-macro of the custom mac library, and the macro variables in the sub-macro need to be stored in the sub-macro variables of the custom macro variable library. This nested macro processing method allows users to build more complex macro structures and implement more advanced functions. For example, a main macro may contain multiple sub-macros, each of which can independently complete specific tasks, and the execution order and parameter passing of these sub-macros can be coordinated and controlled through the main macro.

[0055] In the step of "Store macro variables in the system macro variable library and store custom local variables in the custom macro variable library", "%let x = ()" is recognized. If there is a participle between %let and =, the participle before = and the macro variable value after = are stored in the custom macro variable library. If there are more than two participles, an error is returned. For example, "% let var1 = value1;", "var1" and "value1" are stored in the custom macro variable library, so that in subsequent codes, "var1" can be referenced to obtain its corresponding value "value1". This provides a way to pass and use dynamically changing values ​​in the program, and different macro variable values ​​can be set according to different situations, thereby realizing more flexible code logic.

[0056] In the step of "when the same function word is recognized to appear again in the code pool to be run", the function words include data, proc, run, quit, ods, libname, filename, option, options. In the step of "when the end instructions corresponding to the same function word are recognized to appear in pairs", the function words and the corresponding end instructions are: data / run;, proc / run;, run;, quit;, ods / ;, libname / ;, filename / ;, option / ;, options / .

[0057] When the same function word appears again in the code pool to be run, the code segment consisting of the single sentence code between the two same function words is executed, so that each code block defined by the same function word can be executed as an independent logical unit. For example, in SAS code, "data" is a function word used to define a data set. When the second "data" appears, it means that the data processing logic defined by the previous "data" is complete. At this time, the code segment between the two "data"s is executed to complete the operations related to the previous data set, such as data reading, cleaning, calculation, etc., and then the definition and operation of the next data set are processed, realizing the independence and sequence of different data set processing. After executing the code segment between the two same function words, the executed code segment is immediately cleared. This helps to free up memory resources and avoid the executed code from occupying too much memory space. Especially when processing a large amount of data or complex logic, timely cleaning of unused code segments can prevent memory leaks and improve the overall performance of the program. For example, if different data sets are processed multiple times in a loop, if the executed data set processing code segment is not cleared in time, the memory usage will continue to increase with the increase in the number of loops, which may eventually cause the program to run slowly or even crash. This timely clearing mechanism can effectively avoid such problems.

[0058] When the end instructions corresponding to the same function word are identified to appear in pairs, all code segments are executed, which ensures that the entire code logic related to the function word is executed completely. For example, for the "proc" (process step) function word, its end instruction is usually "run;" or "quit;". When this pair of instructions appears at the same time, it means that all operations in the process step have been defined. At this time, the code of the entire process step is executed, including data processing, analysis, output and other operations, ensuring that the function of the process step is fully realized. This mechanism helps to control the flow of the program so that each functional unit (starting with a function word and ending with the corresponding end instruction) is executed in the correct order, avoiding confusion and errors in code execution. For example, in a program containing multiple "proc" process steps and other code logic, only when the end instructions of each process step appear in pairs can the next code logic part be accurately entered, ensuring the overall structure of the program is clear and the logic is correct.

[0059] Regarding the execution of the code segments of this solution, the code segments with different operation modes of different data definition words have different execution modes, as shown below:

[0060] For the DATA step:

[0061] If the running mode is SET statement, the corresponding execution mode is: read data from the existing data set; merge / by statement can merge the data on the left and right through the common variables.

[0062] If the running mode is a calculation statement, the corresponding execution mode is to recognize arithmetic operators, logical operators, comparison operators, minimum / maximum operators, connection operators, and built-in functions, perform corresponding calculations and write them into the data set.

[0063] If the running mode is a loop statement, the corresponding execution mode is to perform corresponding calculations and write them into the data set after recognizing the loop keywords do, do while, do until, and do over.

[0064] If the operation mode is a conditional statement, the corresponding execution mode is to identify the decision statements if, if-then-else, if-then-else-if. If the condition is determined to be true, one or more statements will be executed. On the contrary, if the condition is false, they will not be executed. The where statement selects a subset from an existing SAS data set and stipulates that the data must meet a condition before the observation is read in.

[0065] If keep or drop is recognized, the execution method is to retain or delete the variable name in the dataset through keep or drop to generate a subset dataset.

[0066] If the running mode is INPUT statement, the corresponding execution mode is to describe the input data, which generally has two types: numeric type and character type.

[0067] If the execution mode is INFORMAT / FORMAT statement, the execution mode corresponding to INFORMAT is to specify the format for INPUT, and the execution mode corresponding to FORMAT is to specify the format for PRINT.

[0068] If the running mode is the MISSING statement, the corresponding execution method is to point out the reason why some data is missing, and use a letter to register missing. When missing values ​​are encountered, they can be recorded in the data set instead of being treated as invalid data.

[0069] If the execution mode is INFILE statement, the corresponding execution mode is to read data from an external text data file.

[0070] If the execution mode is RENAME statement, the corresponding execution mode is to rename the variables in the data set.

[0071] If the execution mode is ARRAY statement, the corresponding execution mode is to define a set of variables as elements in a data group.

[0072] If the execution mode is the RETAIN statement, the corresponding execution mode is to assign initial values ​​to the variables and retain them until the next observation is read.

[0073] For the PROC step:

[0074] Identify the second word following PROC and execute the corresponding steps based on the content of the second word.

[0075] The specific contents of executing the corresponding steps according to the content of the second word are as follows:

[0076] If the second word is a statistical analysis method, the corresponding statistical analysis method is called to perform statistics and analysis on the data set and output the results. The statistical analysis methods include numerical statistics acquisition methods (proc means, procsummary, proc univariate); character statistics acquisition methods (proc freq); cluster analysis (proccluster); correlation analysis (proc corr); regression analysis (linear regression proc reg, logistic regression proclogistic); principal component analysis (proc princomp); and a type of t-test (proc ttest).

[0077] If the second word is SORT, the sorting process is performed.

[0078] If the second word is EXPORT, the dataset is exported to csv or txt; if the second word is IMPORT, the dataset is imported from csv or txt.

[0079] If the second word is TRANSPOSE, the matrix is ​​transposed.

[0080] If the second word is RANK, the data is grouped according to the numeric variable and written into the data set.

[0081] If the second word is REPORT or TABULATE, the data will be summarized and analyzed in multiple dimensions and a data report will be generated.

[0082] If the second word is PRINT, the data of the results of filtering, grouping statistics, summation, etc. will be output; or PRINTTO will be used to save the output results to the specified file.

[0083] If the second word is APPEND, the data in one dataset is added to another dataset.

[0084] If the second word is CHART or GCHART, vertical and horizontal bar charts, block charts, pie charts, etc. are drawn; if the second word is G3D, 3D graphics are drawn; if the second word is GBARLINE, a mixed column-line chart of two variables is drawn; if the second word is PLOT or GPLOT, scatter plots, line charts, and layer charts are drawn; if the second word is SGPLOT, a stacked chart or a bar chart is drawn.

[0085] If the second word is COMPARE, the attributes, variables, observations, etc. of the two data sets are compared.

[0086] If the second word is CONTENTS, view the data set information, file path name, variable information, etc.

[0087] If the second word is COPY, it copies all data sets from one logical library to another logical library, or from one file to another.

[0088] If the second word is DATASETS, the functions of renaming, copying, deleting, and repairing data sets are implemented.

[0089] If the second word is DELETE, the function of deleting the data set is realized.

[0090] If the second word is FORMAT, a custom format is created to display the data.

[0091] If the second word is OPTIONS, all SAS system options and their settings and descriptions are listed.

[0092] If the second word is STANDARD, the numerical variables of the original data are normalized, for example, the mean value of the variable is set to 0 and the standard deviation is set to 1, which can eliminate the impact of data of different magnitudes on the analysis process.

[0093] If the second word is ANOVA, it will handle one-way ANOVA and multi-way ANOVA models with crossed effects.

[0094] If the second word is FACTOR, a factor model is established to obtain the values ​​of each common factor. The eigenvalue situation, factor loading, common factor explanation ratio, etc. can be output, and PROC PLOT can be used to draw a scatter plot of the factor scores.

[0095] If the second word is NPAR1WAY, a one-way nonparametric hypothesis test is performed.

[0096] If the second word is TREE or XTREE, a decision tree is built.

[0097] In addition, it is worth mentioning that the big data analysis method of this solution that supports SAS language parsing and execution can import the parsed SAS data set into other databases for execution. Correspondingly, this solution additionally includes the following steps:

[0098] Through LIBNAME, a link is established between the database storing the execution results and the external data logic library, or the database storing the execution results is virtualized into an external data logic library. At this time, the data in the database storing the execution results can be directly operated through the external data logic library, and the results of the operation are returned to the database.

[0099] In some embodiments, the database storing the execution results includes mysql, SQL server, pgsql, db2, Oracle, hive, Gbase, DAMO, and Gauss.

[0100] In a specific embodiment, the logic diagram of the big data analysis method supporting SAS language parsing and execution is as follows: Figure 2As shown in the figure, create a temporary file storage directory, call the system mac library and system macro variable library, create a custom mac library and a custom macro variable library in turn; then obtain the SAS code to be run and perform word segmentation according to the key field, determine whether there is a macro code field starting with %, if so, process the macro code field and store it in the system mac library and the custom mac library respectively; further determine whether there is a macro variable, if so, store the macro variable in the system macro variable library, and store the %let custom local variable in the custom macro variable library; use semicolons to segment the code in the code pool to be run sentence by sentence To the single-sentence code, analyze the meaning of each code to obtain the running mode. When the same function word is recognized to appear again in the code pool to be run, the code segment consisting of the single-sentence code between the two same function words is executed, and the executed code segment is cleared after the execution is completed; when the end instructions corresponding to the same function word are recognized to appear in pairs, all code segments are executed, and the executed code segments are cleared after the execution is completed; finally, an external data logic library is created through libname, or the database is virtualized into an external data logic library, and the data in the database can be operated in the external data logic library and the results of the operation are returned to the database.

[0101] Embodiment 2

[0102] like Figure 4 As shown, this solution provides a big data analysis platform that supports SAS language parsing and execution, and executes the code by compiling in C++ language, including the following structure:

[0103] A SAS code acquisition unit, used for acquiring a SAS code to be executed, wherein the SAS code to be executed is represented by a SAS language;

[0104] The word segmentation unit is used to identify the key words in the SAS code to be run and perform word segmentation processing according to the key words to obtain the word segmentation results. For the macro code field in the word segmentation results, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library. For the macro variables in the word segmentation results, the macro variables are stored in the system macro variable library, and the custom local variables are stored in the custom macro variable library;

[0105] The macro code processing unit is used to gradually put the word segmentation results into the pre-running code pool according to the code sequence of the SAS code to be run, wherein for the macro code field in the word segmentation result, the corresponding code is obtained from the system mac library or the custom mac library, wherein for the macro variable in the word segmentation result, the variable value is obtained from the system macro variable library or the custom macro variable library;

[0106] The execution unit is used to divide the codes in the code pool to be run sentence by sentence with semicolons to obtain single-sentence codes, analyze the meaning of each code sentence to obtain the running mode, and when the same function word is recognized to appear again in the code pool to be run, execute the code segment composed of the single-sentence codes between two identical function words, and clear the executed code segment after execution; when the end instructions corresponding to the same function word are recognized to appear in pairs, execute all code segments, and clear the executed code segments after execution.

[0107] The big data analysis platform that supports SAS language parsing and execution is a data statistics and modeling software written in C++. It can recognize and execute SAS language to have the ability to process big data, and integrates common functions to make statistical analysis more flexible and fast. The execution code of the big data analysis platform that supports SAS language parsing and execution is as follows: Figure 3 shown.

[0108] Specifically, this solution establishes an external data logic library through LIBNAME, and associates the database storing the execution results with an external data logic library of a big data analysis platform that supports SAS language parsing and execution; or, virtualizes the database address storing the execution results to the big data analysis platform that supports SAS language parsing and execution.

[0109] In addition, the big data analysis platform of this solution that supports SAS language parsing and execution has additional OPTION options, which are used to set system settings for output appearance during data processing, processing of observation values ​​of data sets, initialization functions, etc. At the same time, it is also set to use FILENAME to represent a reference to a file, so that an external file can be written into a data set, or a data set can be written into an external file.

[0110] Embodiment 3

[0111] This embodiment also provides an electronic device, referring to Figure 5 , including a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the big data analysis method supporting SAS language parsing and execution.

[0112] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0113] The memory 404 may include a large-capacity memory 404 for data or instructions. The memory 404 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 402.

[0114] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the big data analysis methods supporting SAS language parsing and execution in the above embodiments.

[0115] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0116] The transmission device 406 may be used to receive or send data via a network. Specific examples of the above network may include a wired or wireless network provided by a communication provider of the electronic device.

[0117] The input / output device 408 is used to input or output information. In this embodiment, the input information may be SAS code to be executed, and the output information may be execution results.

[0118] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:

[0119] Obtaining a SAS code to be executed, wherein the SAS code to be executed is represented by a SAS language;

[0120] Identify the key words in the SAS code to be run and perform word segmentation according to the key words to obtain the word segmentation results. For the macro code field in the word segmentation results, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library. For the macro variables in the word segmentation results, the macro variables are stored in the system macro variable library, and the custom local variables are stored in the custom macro variable library.

[0121] The word segmentation results are gradually put into the pre-running code pool according to the code sequence of the SAS code to be run. For the macro code field in the word segmentation result, the corresponding code is obtained from the system mac library or the custom mac library. For the macro variables in the word segmentation result, the variable values ​​are obtained from the system macro variable library or the custom macro variable library.

[0122] The codes in the code pool to be run are split sentence by sentence with semicolons to obtain single-sentence codes, and the meaning of each code is analyzed to obtain the running mode. When the same function word is recognized to appear again in the code pool to be run, the code segment consisting of the single-sentence code between the two same function words is executed, and the executed code segment is cleared after execution; when the end instructions corresponding to the same function word are recognized to appear in pairs, all code segments are executed, and the executed code segments are cleared after execution.

[0123] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0124] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0125] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 5 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0126] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A big data analysis method supporting SAS language parsing and execution, characterized in that: The following steps are involved: Obtaining a SAS code to be executed, wherein the SAS code to be executed is represented by a SAS language; After obtaining the SAS code to be run, a temporary file path is created, the system mac library and the system macro variable library are called, and a custom mac library and a custom macro variable library are created. The system mac library includes predefined macro instructions and functions, and the system macro variable library stores system predefined macro variables. Identify the key words in the SAS code to be run and perform word segmentation according to the key words to obtain the word segmentation results. For the macro code field in the word segmentation results, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library. For the macro variables in the word segmentation results, the macro variables are stored in the system macro variable library, and the custom local variables are stored in the custom macro variable library. The mac field includes %macro, %if / %else, %substr and %do. For "%if / %else", when entering the macro %macro, if the keyword %if is located for the first time, it is marked as a=0 and stored word by word in the temporary pool of the vocabulary. If %if is encountered again, it is marked as a+1; if %else %if is located, a remains unchanged; if %else is located, it is determined whether a is 0. When a>0, a-1 is added and stored word by word in the temporary pool of the vocabulary. When a=0, the content of the temporary pool of the vocabulary is stored in the sub-macro corresponding to the macro in the corresponding custom mac library and the temporary pool of the vocabulary is cleared; For "%do", when entering the macro %macro, if the keyword %do is encountered for the first time, it is marked as b=0 and stored word by word in the temporary word pool. If %do is followed by %while, the sas statement is executed when the condition is true, and the condition is checked before the loop. If %do is followed by %until, it stops when the condition is true, checks the condition after the loop, and stores the conditional statement in the temporary pool of the vocabulary; If the keyword %do is located again, it is marked as b+1. If the keyword %end is located, it is determined whether b is 0. If b>0, b-1 is stored word by word in the temporary pool of the vocabulary. If b=0, the content of the temporary pool of the vocabulary is stored in the sub-macro corresponding to the macro in the corresponding custom mac library and the temporary pool of the vocabulary is cleared. The word segmentation results are gradually put into the pre-running code pool according to the code sequence of the SAS code to be run. For the macro code field in the word segmentation result, the corresponding code is obtained from the system mac library or the custom mac library. For the macro variables in the word segmentation result, the variable values ​​are obtained from the system macro variable library or the custom macro variable library. The codes in the code pool to be run are split sentence by sentence with semicolons to obtain single-sentence codes, and the meaning of each code is analyzed to obtain the running mode. When the same function word is recognized to appear again in the code pool to be run, the code segment consisting of the single-sentence code between the two same function words is executed, and the executed code segment is cleared after execution; when the end instructions corresponding to the same function word are recognized to appear in pairs, all code segments are executed, and the executed code segments are cleared after execution.

2. The big data analysis method supporting SAS language parsing and execution according to claim 1, characterized in that: The keyword segment includes function words and macro code fields, where function words include data definition words, operation logic words, keywords, variable names, character strings and numbers, and the macro code field starts with %.

3. The big data analysis method supporting SAS language parsing and execution according to claim 1, characterized in that: Custom fields are represented as "%macro SELF_MAC;...%mend". When locating a custom field, SELF_MAC is stored in the custom mac library. If another %macro appears after macro and before %mend, the second macro needs to be stored in the corresponding sub-macro of the custom mac library, and the macro variables in the sub-macro need to be stored in the sub-macro variables of the custom macro variable library.

4. The big data analysis method supporting SAS language parsing and execution according to claim 1, characterized in that: When "%let x =()" is recognized, if there is a participle between %let and =, the participle before = and the macro variable value after = are stored in the user-defined macro variable library. If more than two participles appear, an error is returned.

5. The big data analysis method supporting SAS language parsing and execution according to claim 1, characterized in that: Through LIBNAME, a link is established between the database storing the execution results and the external data logic library, or the database storing the execution results is virtualized into an external data logic library.

6. A big data analysis platform that supports SAS language parsing and execution, and executes code by compiling in C++ language, characterized in that: These include: A SAS code acquisition unit is used to acquire the SAS code to be run, wherein the SAS code to be run is represented by the SAS language; after acquiring the SAS code to be run, a temporary file path is created, a system mac library and a system macro variable library are called, and a custom mac library and a custom macro variable library are created, wherein the system mac library includes predefined macro instructions and functions, and the system macro variable library stores system predefined macro variables; The word segmentation unit is used to identify the key words in the SAS code to be run and perform word segmentation processing according to the key words to obtain the word segmentation results. For the macro code field in the word segmentation results, the mac field is stored in the system mac library, and the custom field is stored in the custom mac library. For the macro variables in the word segmentation results, the macro variables are stored in the system macro variable library, and the custom local variables are stored in the custom macro variable library; The mac field includes %macro, %if / %else, %substr and %do. For "%if / %else", when entering the macro %macro, if the keyword %if is located for the first time, it is marked as a=0 and stored word by word in the temporary pool of the vocabulary. If %if is encountered again, it is marked as a+1; if %else %if is located, a remains unchanged; if %else is located, it is determined whether a is 0. When a>0, a-1 is added and stored word by word in the temporary pool of the vocabulary. When a=0, the content of the temporary pool of the vocabulary is stored in the sub-macro corresponding to the macro in the corresponding custom mac library and the temporary pool of the vocabulary is cleared; For "%do", when entering the macro %macro, if the keyword %do is encountered for the first time, it is marked as b=0 and stored word by word in the temporary word pool. If %do is followed by %while, the sas statement is executed when the condition is true, and the condition is checked before the loop. If %do is followed by %until, it stops when the condition is true, checks the condition after the loop, and stores the conditional statement in the temporary pool of the vocabulary; If the keyword %do is located again, it is marked as b+1. If the keyword %end is located, it is determined whether b is 0. If b>0, b-1 is stored word by word in the temporary pool of the vocabulary. If b=0, the content of the temporary pool of the vocabulary is stored in the sub-macro corresponding to the macro in the corresponding custom mac library and the temporary pool of the vocabulary is cleared. The macro code processing unit is used to gradually put the word segmentation results into the pre-running code pool according to the code sequence of the SAS code to be run, wherein for the macro code field in the word segmentation result, the corresponding code is obtained from the system mac library or the custom mac library, wherein for the macro variable in the word segmentation result, the variable value is obtained from the system macro variable library or the custom macro variable library; The execution unit is used to divide the codes in the code pool to be run sentence by sentence with semicolons to obtain single-sentence codes, analyze the meaning of each code sentence to obtain the running mode, and when the same function word is recognized to appear again in the code pool to be run, execute the code segment composed of the single-sentence codes between two identical function words, and clear the executed code segment after execution; when the end instructions corresponding to the same function word are recognized to appear in pairs, execute all code segments, and clear the executed code segments after execution.

7. The big data analysis platform supporting SAS language parsing and execution according to claim 6, characterized in that: An external data logic library is established through LIBNAME to associate the database storing the execution results with an external data logic library of a big data analysis platform that supports SAS language parsing and execution; or, the database address storing the execution results is virtualized to the big data analysis platform that supports SAS language parsing and execution.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes a big data analysis method supporting SAS language parsing and execution according to any one of claims 1 to 5.

Images