A system and method for realizing SAS data analysis capability in Python environment

Abstract

The present invention relates to a system and method for realizing SAS data analysis capability in a python environment, wherein the system comprises the following modules: a SAS log analysis module, configured to periodically extract SAS execution logs from SAS, and perform statistics, preprocessing and analysis on the SAS program execution statement information therein; a newly added operator module, configured to use the result of periodic analysis of the SAS execution log to screen the operator list that needs to be stored; a python conversion module, configured to convert and encapsulate statements and parameters with a frequency of occurrence that meet the standard into python operators according to the screened operator list, and test and encapsulate the python operators into a sastopy library; a sastopy library, configured to convert statements in SAS into operator forms of python language when encapsulating the python operators into the sastopy library, and store them; an operator operation and maintenance module, configured to monitor and maintain the sastopy library. The present invention enables business personnel to complete data analysis tasks more intuitively and conveniently, thereby improving work efficiency and flexibility.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and more particularly to a system and method for realizing SAS data analysis capabilities in a Python environment. Background Art

[0002] With the rapid development of financial technology and data analysis technology, the efficiency and simplification of data processing methods have become the goal that all industries continue to pursue.

[0003] The existing technology has the following deficiencies: First, there is a lack of tools on the market that can directly convert SAS (Statistical Analysis System) code into Python. Second, there is no function library from SAS to other languages ​​that can achieve the migration and conversion of most commonly used SAS statements, so that other languages ​​have the ability to perform SAS data analysis. Third, there is a lack of a regular update mechanism and related operation and maintenance steps, which limits its reliability in long-term use.

[0004] Therefore, it is necessary to provide a system and method for realizing SAS data analysis capabilities in a Python environment, so that business personnel can complete data analysis tasks more intuitively and conveniently, thereby improving work efficiency and flexibility. Summary of the invention

[0005] The purpose of the present invention is to provide a system and method for realizing SAS data analysis capabilities in a python environment, so that business personnel can complete data analysis tasks more intuitively and conveniently, thereby improving work efficiency and flexibility.

[0006] In order to solve the problems existing in the prior art, the present invention provides a system for realizing SAS data analysis capabilities in a Python environment, including the following modules:

[0007] A SAS log analysis module is configured to periodically extract SAS execution logs from SAS and perform statistics, preprocessing and analysis on SAS program execution statement information therein;

[0008] A new operator module is added, which is configured to use the results of regular log analysis performed by SAS to filter the list of operators that need to be stored;

[0009] The python conversion module is configured to convert and encapsulate the statements and parameters that meet the frequency requirements into python operators according to the list of filtered operators, and then test and encapsulate the python operators into the sastopy library;

[0010] The sastopy library is configured to convert the statements in SAS into the operator form of the Python language and store them when encapsulating the Python operator into the sastopy library;

[0011] Operator operation and maintenance module, configured to monitor and maintain the sastopy library.

[0012] Optionally, in the system for implementing SAS data analysis capabilities in a python environment, the SAS log analysis module includes a log collection unit, a data preprocessing unit, a script statement analysis unit, and an analysis result generation unit;

[0013] The log collection unit is configured to periodically extract all execution logs in SAS within a period of time;

[0014] The data preprocessing unit is configured to filter and retain log information of level info; filter and retain information of SAS program lines; remove the comments in SAS program lines; and perform statistical rollback when encountering log errors;

[0015] The script statement analysis unit is configured to count and analyze SAS statements appearing in the execution log;

[0016] The analysis result generating unit is configured to generate a statistical analysis report by analyzing the results through SAS statements.

[0017] Optionally, in the system for implementing SAS data analysis capabilities in the Python environment, the method for screening the list of operators to be stored is as follows:

[0018] Scan SAS execution logs regularly to count statements and parameters that meet the frequency requirements;

[0019] Generate a pre-selected pool of update operators based on the statements and parameters that meet the frequency requirements obtained by scanning;

[0020] Determine whether the operator in the pre-selected pool already exists in the sastopy library;

[0021] If the corresponding operator does not exist in the sastopy library, determine through manual judgment and analysis whether the corresponding operator in the pre-selected pool needs to be written as a new operator;

[0022] After manual judgment, a list of operators that need to be stored is generated.

[0023] Optionally, the system for implementing SAS data analysis capabilities in the Python environment also includes integrating user-customized SAS transcription requirements to meet user personalized needs; generating an update operator pre-selection pool based on scanned statements and parameters that meet the frequency of occurrence standards and integrated user personalized needs.

[0024] Optionally, in the system for implementing SAS data analysis capabilities in a python environment,

[0025] The python conversion module is configured to perform detailed transcription of statements and parameters that meet the frequency requirements. The detailed transcription includes the transcription of parameter types, default values, and specific functions. During the operator testing phase, multiple identical data sets are tested in SAS and python environments to verify that the output of the python operator is consistent with the output of the original SAS statement in terms of accuracy and completeness.

[0026] Optionally, in the system for implementing SAS data analysis capabilities in a python environment, the sastopy library includes data step operators, process step operators, and function operators.

[0027] Optionally, in the system for implementing SAS data analysis capabilities in the Python environment, the operator operation and maintenance module is configured to regularly analyze the calling status of the sastopy library, calculate the number of operator errors and the number of operator calls, and perform processing and optimization according to the set threshold.

[0028] The present invention also provides a method for realizing SAS data analysis capability in a Python environment, using the above system, comprising the following steps:

[0029] S1: Regularly extract SAS execution logs from SAS, and perform statistics, preprocessing and analysis on the SAS program execution statement information;

[0030] S2: Use SAS to perform regular log analysis and filter the list of operators that need to be stored;

[0031] S3: According to the list of screened operators, convert and encapsulate the statements and parameters that meet the frequency requirements into Python operators, test them, and encapsulate the Python operators into the sastopy library;

[0032] When encapsulating the Python operator into the Sastopy library, the statements in SAS are converted into the operator form of the Python language and stored;

[0033] S4: Each python system uses the python language to call operators with SAS-specific data processing and analysis.

[0034] Optionally, the method for implementing SAS data analysis capabilities in a Python environment further includes the following steps:

[0035] S5: Operate, maintain and regularly update the sastopy library.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] (1) Among many data analysis tools and software, SAS (Statistical Analysis System) is widely used due to its powerful functions in risk management, credit assessment and customer relationship management, while Python has also attracted a lot of attention in the field of data analysis due to its flexibility and complete library ecosystem. The present invention encapsulates the frequently used functions in SAS into Python operators and establishes a Sastopy library to achieve the conversion and migration of commonly used SAS functions, thus providing a new solution for data processing and analysis tasks. This not only continues the advantages of SAS in the field of data analysis, but also simplifies the data processing process and promotes the integration of different technology stacks, so that business personnel can complete data analysis tasks more intuitively and conveniently, thereby improving work efficiency and flexibility.

[0038] (2) By calling the sastopy library, you can call SAS-specific data processing and analysis operators while using the Python language, enriching Python's data processing and analysis capabilities.

[0039] (3) The sastopy library builds a bridge between SAS and modeling platforms built based on Python, allowing various analytical modeling platforms built based on Python to cover SAS functions to a certain extent, further expanding the application scenarios and coverage of the platform.

[0040] (4) The present invention combines the characteristics of SAS and Python languages, and ensures that the Python operators in the library are consistent with the corresponding SAS statements in terms of usage and running results as much as possible, so that business personnel who are proficient in using SAS can use the operator functions quickly and efficiently.

[0041] (5) The Sastopy library of the present invention enables users to utilize Python's extensive ecosystem and library resources to optimize and automate tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A module flow chart of a system provided by an embodiment of the present invention;

[0043] Figure 2 A flow chart of a SAS log analysis module provided by an embodiment of the present invention;

[0044] Figure 3 A flowchart of a newly added operator module provided in an embodiment of the present invention;

[0045] Figure 4 A flowchart of a Python conversion module provided by an embodiment of the present invention;

[0046] Figure 5 Operator operation and maintenance flow chart provided for an embodiment of the present invention;

[0047] Figure 6 A comparison diagram of operator coverage provided by an embodiment of the present invention;

[0048] Figure 7 A comparison chart of operator accuracy provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0050] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0051] The existing technology has the following deficiencies: First, there is a lack of tools on the market that can directly convert SAS (Statistical Analysis System) code into Python. Second, there is no function library from SAS to other languages ​​that can achieve the migration and conversion of most commonly used SAS statements, so that other languages ​​have the ability to perform SAS data analysis. Third, there is a lack of a regular update mechanism and related operation and maintenance steps, which limits its reliability in long-term use.

[0052] In order to solve the problems existing in the prior art, the present invention provides a system for realizing SAS data analysis capabilities in a Python environment, such as Figure 1 As shown, the system includes the following modules:

[0053] The SAS log analysis module is configured to periodically extract SAS execution logs from SAS, and perform statistics, preprocessing and analysis on the SAS program execution statement information therein, so as to provide data support for subsequent processes.

[0054] Specifically, refer to Figure 2 ,SAS log analysis module includes a log collection unit, a data preprocessing unit, a script statement analysis unit and an analysis result generation unit;

[0055] (1) The log collection unit is configured to periodically extract all execution logs in SAS within a period of time; the SAS execution log contains system information, SAS program execution information, error and warning information, running time and resource usage information.

[0056] (2) After the data preprocessing unit is configured to complete the log collection, the main analysis object is the execution status of each SAS statement. First, the SAS log is preprocessed to extract the information required for subsequent analysis and transcription. The specific preprocessing includes the following steps: 1) Filter and retain log information with a level of info: Each line in the log data has its corresponding log level. For a normally running SAS program, the log level is info, so only the information with a log level of info needs to be retained; 2) Filter and retain the information of SAS program lines: The info-level log of SAS will contain information such as data sets, SAS statement execution, SAS library, SAS options, ODS output, and program running time. The present invention filters and retains the information of SAS program lines in the SAS statement execution information; 3) Eliminate the comments in the SAS program lines: In the program, the purpose of the comments is to explain the purpose and function of the code, and it will not be executed as a valid program line in the system, so the comments need to be eliminated in the statement statistics; 4) When encountering a log error, perform statistical rollback: When the log level is error in the SAS log, the statistical content corresponding to the error log needs to be rolled back to ensure that there is no statistical error data;

[0057] (3) The script statement analysis unit is configured to count and analyze the SAS statements appearing in the execution log after completing the data preprocessing;

[0058] Specifically, a typical SAS program consists of data steps, procedure steps, comments, global statements, macro statements, and functions. Comments are not considered valid program lines and have been removed in the preprocessing step. Write a python script to identify and count the five types of statements in SAS programs: data steps, procedure steps, global statements, macro statements, and functions. The following is the identification and counting method for each statement:

[0059] Data step: Recognizes the "data" keyword, which indicates the start of a data step. Recognizes the "run" keyword, which indicates the end of a data step. Counts the statements used in a data step.

[0060] Procedure step: Identify the "proc" keyword to indicate the start of a procedure step. Identify the "run" or "quit" keyword to indicate the end of a procedure step. Count the procedure step names that appear in the procedure step and the parameters and statements corresponding to each procedure step name.

[0061] Global statements: Identify lines that begin with keywords such as "options", "title", "footnote", "libname", etc. Count the global statements that appear in the program.

[0062] Macro statements: Identify the "%macro" keyword, which indicates the beginning of a macro. Identify the "%mend" keyword, which indicates the end of a macro. The macro program names and macro functions used in the statistics program.

[0063] Function: Identify the syntax of function calls, such as "function_name(arguments)". The function names and corresponding parameters used in the statistical program.

[0064] (4) The analysis result generation unit is configured to generate a statistical analysis report based on the SAS statement analysis results. The report content includes the calling status and frequency statistics of various statements.

[0065] A new operator module is added, which is configured to use the results of regular log analysis performed by SAS to filter the list of operators that need to be stored;

[0066] like Figure 3 As shown in the figure, the method of filtering the list of operators to be stored is as follows:

[0067] Scan and analyze SAS execution logs regularly to count the statements and parameters that meet the frequency requirements (for example, define a high frequency range, and the statements and parameters that meet the frequency requirements are high frequency statements and parameters);

[0068] Generate a pre-selected pool of update operators based on the statements and parameters that meet the frequency requirements obtained by scanning;

[0069] Determine whether the operator in the pre-selected pool already exists in the sastopy library;

[0070] If the corresponding operator does not exist in the sastopy library, determine through manual judgment and analysis whether the corresponding operator (including statements or statement blocks) in the pre-selected pool needs to be written into a new operator;

[0071] After manual judgment, if the corresponding operator in the pre-selected pool needs to be written into a new operator, the corresponding operator in the pre-selected pool is recorded in the operator list, thereby generating a list of operators that need to be stored.

[0072] In one embodiment, it also includes integrating user customized SAS transcription requirements to meet user personalized needs; if user customized SAS transcription requirements are integrated, it is necessary to generate an update operator pre-selection pool based on the scanned statements and parameters that meet the frequency of occurrence and the integrated user personalized needs.

[0073] The python conversion module is configured to convert and encapsulate statements and parameters (as well as user-customized SAS) that meet the frequency of occurrence into python operators according to the screening operator list, and test them, and encapsulate the python operators into the sastopy library; the present invention uses the results of regular analysis of SAS execution logs to regularly update the operator content in the sastopy library to ensure that these operators can meet the user's usage requirements.

[0074] Specifically, refer to Figure 4 , the python conversion module includes an operator research unit, an operator writing unit, an operator testing unit and an operator storage unit;

[0075] (1) The operator research unit is configured to conduct an in-depth study of the high-frequency statements and parameters (as well as user-customized SAS) that need to be transcribed based on the list of screened operators. This process involves an in-depth understanding of the details of the high-frequency SAS statements and parameters (as well as user-customized SAS), especially the frequency of use, specific values ​​and specific functions of the parameters.

[0076] (2) The operator writing unit is configured to ensure that the functions of the written operators are consistent with the functions of SAS high-frequency statements and parameters (or user-customized SAS requirements) during the operator writing process, and to carefully transcribe the high-frequency statements and parameters (and user-customized SAS), including the parameter type, default value, and specific function. For each operator, the input and output format and error handling mechanism are clearly defined to ensure that clear feedback is provided when the input data is inconsistent or the parameters are incorrect, so as to improve the usability, efficiency, and reliability of the operator.

[0077] (3) The operator testing unit is configured to test multiple identical data sets in SAS and Python environments during the operator testing phase to verify that the output of the Python operator is consistent with the output of the original SAS statement in terms of accuracy and completeness.

[0078] (4) The operator library unit is configured to encapsulate these functions into the library after completing the writing and testing of the operators. This process includes writing the library documentation, including the functional description, parameter type, instructions and usage examples of each operator, aiming to provide clear library entry guidance and support.

[0079] The sastopy library is configured to convert the statements in SAS into the operator form of the Python language and store them when encapsulating the Python operator into the sastopy library;

[0080] Specifically, the sastopy library is divided into three main parts: data step operators, process step operators, and function operators. Each part corresponds to different data processing and analysis functions. This structure is consistent with the division of SAS functions, so that users who are proficient in SAS can quickly find operators with corresponding functions when using the sastopy library.

[0081] (1) Data step operator (data module): The data step operator has the same function as the data step in SAS and is specifically used for data preprocessing and management. The following is an introduction to the data step operator.

[0082] The sas_drop operator matches the drop statement in the SAS data step and is used to delete the specified column from the data set.

[0083] The sas_keep operator matches the keep statement in the SAS data step and is used to retain the specified columns in the data set and delete all other columns.

[0084] The sas_merge operator matches the merge statement in the SAS data step and is used to merge two data sets for integration and association analysis.

[0085] The sas_rename operator matches the rename statement in the SAS data step and is used to rename columns in a data set.

[0086] The sas_where operator matches the where statement in the SAS data step and is used to filter rows in the data set based on certain conditions.

[0087] (2) Process step operators (proc module): Process step operators have the same functions as the process step (proc step) in SAS. They are specifically used for advanced data analysis and statistical calculations, including operators such as sorting, frequency statistics, and descriptive statistics. The following is an introduction to process step operators.

[0088] The sas_proc_sort operator matches the proc sort statement in a SAS procedure step and is used to sort a data set.

[0089] The sas_proc_freq operator matches the proc freq statement in the SAS procedure step and is used to calculate and output the frequency distribution of numerical variables.

[0090] The sas_proc_append operator matches the proc append statement in a SAS procedure step and is used to append one data set to another.

[0091] The sas_proc_transpose operator matches the proc transpose statement in a SAS procedure step and is used to transpose a data set.

[0092] The sas_proc_contents operator matches the proc contents statement in a SAS procedure step and is used to view and describe the metadata information of a data set.

[0093] The sas_proc_surveyselect operator matches the proc surveyselect statement in the SAS procedure step and is used to perform sample sampling operations on the data set. It supports simple random sampling and stratified sampling.

[0094] The sas_proc_means operator matches the proc means statement in the SAS procedure step and is used to generate descriptive statistics for variables.

[0095] The sas_proc_tabulate operator matches the proc tabulate statement in a SAS procedure step and is used to create summary tables and crosstabs, generating summary statistics based on specified categorical and numerical variables.

[0096] The sas_proc_summary operator matches the proc summary statement in the SAS procedure step and is used to generate summary statistical information. It can calculate multiple statistics for one or more numeric variables.

[0097] The sas_proc_rank operator matches the proc rank statement in a SAS procedure step and is used to rank the numeric variables in a data set and generate a rank for each observation in the data set.

[0098] The sas_proc_print operator matches the proc print statement in a SAS procedure step and is used to display the contents of a data set.

[0099] (3) Function operators (func module): Function operators have the same functions as functions in SAS and are specifically used to perform specific data operations and conversions, including character functions, numeric functions, and date functions. The following is an introduction to function operators.

[0100] The sas_cat operator matches the cat function in SAS and is used to merge input variables.

[0101] The sas_compress operator matches the compress function in SAS and is used to remove spaces or other characters from the string for each element in the specified column.

[0102] The sas_find operator matches the find function in SAS and is used to perform a string search for each element in the specified column starting from the specified position.

[0103] The sas_index operator matches the index function in SAS and is used to perform a string search on each element of the specified column.

[0104] The sas_length operator matches the length function in SAS and is used to calculate the character length of each element in the specified column.

[0105] The sas_strip operator matches the strip function in SAS and is used to remove leading and trailing spaces from each element of the specified column.

[0106] The sas_substr operator matches the substr function in SAS and is used to perform character slicing on each element of the specified column.

[0107] The sas_tranwrd operator matches the tranwrd function in SAS and is used to perform string replacement on each element of the specified column.

[0108] The sas_trim operator matches the trim function in SAS and is used to remove the trailing spaces of each element in the specified column.

[0109] The sas_upcase operator matches the upcase function in SAS and is used to change all the letters of each element in the specified column to uppercase.

[0110] The sas_abs operator matches the abs function in SAS and is used to calculate the absolute value of each element in the specified column.

[0111] The sas_floor operator matches the floor function in SAS and is used to round down each element of the specified column.

[0112] The sas_max operator matches the max function in SAS and is used to return the maximum value of the input variables.

[0113] The sas_min operator matches the min function in SAS and is used to return the minimum value in the input variables.

[0114] The sas_sum operator matches the sum function in SAS and is used to return the sum of the input variables.

[0115] The sas_round operator matches the round function in SAS and is used to round each element of the specified column to the specified number of digits.

[0116] The sas_day operator matches the floor function in SAS and is used to extract and return the day of a given date input.

[0117] The sas_intck operator matches the intck function in SAS and is used to calculate the time interval between two dates based on the specified interval type.

[0118] The sas_intnx operator matches the intnx function in SAS and is used to calculate and return the date that is increased or decreased by a specific number of intervals from the starting date based on the given time interval and alignment.

[0119] The sas_lag operator matches the lag function in SAS and is used to shift each element of the specified column down one position.

[0120] The sas_mdy operator matches the mdy function in SAS and is used to create a date value based on the given month, day, and year information.

[0121] The sas_month operator matches the month function in SAS and is used to extract and return the month of a given date based on the input date.

[0122] The sas_today operator matches the today function in SAS and is used to specify the number of days to add or subtract from today based on the current date.

[0123] The sas_year operator matches the year function in SAS and is used to extract and return the year of a given date input.

[0124] The sas_input operator matches the input function in SAS and is used to convert characters to numeric values.

[0125] The sas_missing operator matches the missing function in SAS and is used to check whether each element of the specified column is missing.

[0126] The sas_put operator matches the put function in SAS and is used to convert numeric values ​​to characters.

[0127] The operator operation and maintenance module is configured to monitor and maintain the sastopy library. The operator operation and maintenance module is configured to regularly analyze the call status of the sastopy library, count the number of operator errors and operator calls, and process and optimize according to the set thresholds.

[0128] For details, please refer to Figure 5 , Figure 5 This is the content of the operator operation and maintenance module.

[0129] (1) Call analysis: Regularly analyze the call status of the sastopy library, count the number of errors and calls of each operator, and generate corresponding data and statistical reports.

[0130] (2) Update the low-frequency operator list: Operators whose number of calls within a period of time is less than the preset threshold m are placed in the low-frequency list; operators whose number of calls within a period of time is not less than the preset threshold m and are in the low-frequency list are removed from the low-frequency list.

[0131] (3) Identification of operator optimization needs: Operators that meet at least one of the following conditions are included in the operator function optimization list: ① The number of error reports within a period of time is greater than the preset threshold n and is not in the low-frequency list; ② The user proposes an optimization requirement for a specific operator.

[0132] (4) Operator function optimization: Perform function optimization on operators in the operator list, including performance optimization, bug fixes, code optimization, etc., to meet the performance and function requirements of specific operators.

[0133] (5) Library version update and iteration: Update and iterate the sastopy version, introduce the optimized operators into the new version, and maintain the continuous improvement and update of the sastopy library.

[0134] The present invention also involves a comprehensive evaluation scheme to test the coverage and accuracy of operators. Five sample groups are randomly selected from the full log data, each containing 1000 process steps, to ensure the representativeness and diversity of the selected samples.

[0135] (1) Coverage

[0136] The coverage is evaluated based on several key dimensions: group, number of SAS process steps, number of full parameter support, number of partial parameter support, and number of unsupported parameters. The full coverage of each group is calculated by calculating the ratio of the number of full parameter support to the number of SAS process steps. The test results of each group of data are recorded in Figure 6 middle.

[0137] (2) Accuracy

[0138] The accuracy analysis considers the following dimensions: group, number of SAS process steps, and number of data consistency. The accuracy of each group is obtained by calculating the ratio of the number of data consistency to the number of SAS process steps. The test results of each group of data are recorded in Figure 7 middle.

[0139] The present invention also provides a method for realizing SAS data analysis capability in a Python environment, using the above system, comprising the following steps:

[0140] S1: Regularly extract SAS execution logs from SAS, and perform statistics, preprocessing and analysis on the SAS program execution statement information;

[0141] S2: Use SAS to perform regular log analysis and filter the list of operators that need to be stored;

[0142] S3: According to the list of screened operators, convert and encapsulate the statements and parameters that meet the frequency requirements into Python operators, test them, and encapsulate the Python operators into the sastopy library;

[0143] When encapsulating the Python operator into the Sastopy library, the statements in SAS are converted into the operator form of the Python language and stored;

[0144] S4: Each python system uses the python language to call operators with SAS-specific data processing and analysis.

[0145] The present invention enables users to have SAS-like data analysis capabilities in a python environment, and facilitates users to migrate SAS scripts to a python environment.

[0146] Preferably, the method further comprises the following steps:

[0147] S5: Operate, maintain and regularly update the sastopy library.

[0148] In summary, compared with the prior art, the present invention has the following advantages:

[0149] (1) Among many data analysis tools and software, SAS (Statistical Analysis System) is widely used due to its powerful functions in risk management, credit assessment and customer relationship management, while Python has also attracted a lot of attention in the field of data analysis due to its flexibility and complete library ecosystem. The present invention encapsulates the frequently used functions in SAS into Python operators and establishes a Sastopy library to achieve the conversion and migration of commonly used SAS functions, thus providing a new solution for data processing and analysis tasks. This not only continues the advantages of SAS in the field of data analysis, but also simplifies the data processing process and promotes the integration of different technology stacks, so that business personnel can complete data analysis tasks more intuitively and conveniently, thereby improving work efficiency and flexibility.

[0150] (2) By calling the sastopy library, you can call SAS-specific data processing and analysis operators while using the Python language, enriching Python's data processing and analysis capabilities.

[0151] (3) The sastopy library builds a bridge between SAS and modeling platforms built based on Python, allowing various analytical modeling platforms built based on Python to cover SAS functions to a certain extent, further expanding the application scenarios and coverage of the platform.

[0152] (4) The present invention combines the characteristics of SAS and Python languages, and ensures that the Python operators in the library are consistent with the corresponding SAS statements in terms of usage and running results as much as possible, so that business personnel who are proficient in using SAS can use the operator functions quickly and efficiently.

[0153] (5) The Sastopy library of the present invention enables users to utilize Python's extensive ecosystem and library resources to optimize and automate tasks.

[0154] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A system for implementing SAS data analysis capabilities in a Python environment, characterized in that: Includes the following modules: A SAS log analysis module is configured to periodically extract SAS execution logs from SAS, and perform statistics, preprocessing and analysis on SAS program execution statement information therein, wherein the SAS program execution statement includes data steps, process steps, comments, global statements, macro statements and functions; A new operator module is added, which is configured to use the results of regular log analysis performed by SAS to filter the list of operators that need to be stored; The python conversion module is configured to convert and encapsulate the statements and parameters that meet the frequency requirements into python operators according to the list of filtered operators, and then test and encapsulate the python operators into the sastopy library, which includes data step operators, process step operators, and function operators. The sastopy library is configured to convert the statements in SAS into the operator form of the Python language and store them when encapsulating the Python operator into the sastopy library; Operator operation and maintenance module, configured to monitor and maintain the sastopy library.

2. The system for realizing SAS data analysis capability in Python environment as claimed in claim 1, characterized in that: The SAS log analysis module includes a log collection unit, a data preprocessing unit, a script statement analysis unit, and an analysis result generation unit; The log collection unit is configured to periodically extract all execution logs in SAS within a period of time; The data preprocessing unit is configured to filter and retain log information of level info; filter and retain information of SAS program lines; and remove the comments in SAS program lines; When a log error occurs, perform statistical rollback; The script statement analysis unit is configured to count and analyze SAS statements appearing in the execution log; The analysis result generating unit is configured to generate a statistical analysis report by analyzing the results through SAS statements.

3. The system for realizing SAS data analysis capability in Python environment as claimed in claim 1, characterized in that: The method for filtering the list of operators to be stored is as follows: Scan SAS execution logs regularly to count statements and parameters that meet the frequency requirements; Generate a pre-selected pool of update operators based on the statements and parameters that meet the frequency requirements obtained by scanning; Determine whether the operator in the pre-selected pool already exists in the sastopy library; If the corresponding operator does not exist in the sastopy library, determine through manual judgment and analysis whether the corresponding operator in the pre-selected pool needs to be written as a new operator; After manual judgment, a list of operators that need to be stored is generated.

4. The system for realizing SAS data analysis capability in Python environment as claimed in claim 3, characterized in that: It also includes integrating user-customized SAS transcription requirements to meet user personalized needs; generating an update operator pre-selection pool based on the scanned statements and parameters that meet the frequency requirements and the integrated user personalized needs.

5. The system for realizing SAS data analysis capability in Python environment as claimed in claim 1, characterized in that: The python conversion module is configured to perform detailed transcription of statements and parameters that meet the frequency requirements, including the parameter type, default value, and specific function. During the operator testing phase, multiple identical data sets are tested in SAS and Python environments to verify that the output of the Python operator is consistent with the output of the original SAS statement in terms of accuracy and completeness.

6. The system for realizing SAS data analysis capability in Python environment as claimed in claim 1, characterized in that: The operator operation and maintenance module is configured to regularly analyze the calls to the sastopy library, count the number of operator errors and operator calls, and process and optimize according to the set thresholds.

7. A method for implementing SAS data analysis capabilities in a Python environment, characterized in that: The system according to any one of claims 1 to 6 comprises the following steps: S1: Regularly extract SAS execution logs from SAS, and perform statistics, preprocessing and analysis on the SAS program execution statement information therein. SAS program execution statements include data steps, procedure steps, comments, global statements, macro statements and functions; S2: Use SAS to perform regular log analysis and filter the list of operators that need to be stored; S3: According to the list of screened operators, convert and encapsulate the statements and parameters that meet the frequency requirements into Python operators, and test them. Encapsulate the Python operators into the sastopy library, which includes data step operators, process step operators, and function operators. When encapsulating the Python operator into the Sastopy library, the statements in SAS are converted into the operator form of the Python language and stored; S4: Each python system uses the python language to call operators with SAS-specific data processing and analysis.

8. The method for realizing SAS data analysis capability in Python environment as claimed in claim 7, characterized in that: The following steps are also included: S5: Operate, maintain and regularly update the sastopy library.

Images