SAS code automatic conversion method and device, equipment and storage medium
By identifying the function name and variable name in the SAS code, detecting the logical relationship of the data table, and using finite automata and BERT models to convert the SAS code into Hive and Z+ platform codes, the problem of logical inconsistency in the SAS code conversion process is solved, and the implementation of the Efficient and automatic data processing logic conversion ensures the accuracy and security of financial data analysis.
Patent Information
- Application Number
- CN202411177380.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-06
AI Technical Summary
The existing technology is difficult to automatically and efficiently convert SAS code into target platform code, resulting in inconsistent logic of financial data processing and affecting decision-making accuracy.
By receiving SAS code, identifying function names and variable names, detecting the logical relationship of the data table, using finite automata and syntax parsing trees to convert SAS code into Hive SQL code, and optimizing the translation model through the BERT model, finally generating the target Z+ platform code, ensuring that Automation and accuracy of the conversion process.
It improves the automation and efficiency of SAS code conversion, ensures consistency of data processing logic, reduces the risk of operational errors, and provides strong technical support for digital transformation in the financial field.
Smart Images

Figure CN119088398A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data conversion and compilation, and in particular to a SAS code automatic conversion method, device, equipment and storage medium. Background Art
[0002] In the modern financial industry, data analysis and processing capabilities have become an important support for corporate decision-making and operations. Financial institutions rely on complex data analysis to conduct risk assessment, customer segmentation, market forecasting, and anti-fraud detection. To this end, financial institutions usually use a variety of data processing tools and platforms, such as SAS (Statistical Analysis System) and various big data platforms (such as Hive, Z+ platform) to manage and analyze massive financial data.
[0003] The SAS platform has played an important role in the financial sector with its powerful data analysis capabilities and a wide range of application scenarios. However, with the explosive growth of data volume and the development of cloud computing technology, financial institutions are gradually turning to more scalable and efficient cloud-based big data platforms for data processing and storage. These platforms (such as Hive and Z+ platforms) can not only process large-scale data, but also support parallel computing and distributed storage, providing higher computing performance and lower costs for financial analysis.
[0004] A key challenge in this transformation is how to seamlessly migrate existing SAS code and analytical logic to these emerging big data platforms. Traditional SAS codes often contain complex logic and data processing steps that involve cleaning, transforming, analyzing, and visualizing financial data. In order to adapt to the characteristics of the new platform, these SAS codes need to be converted into the code form of the target platform, such as Hive SQL or Z+ platform code.
[0005] However, completing such conversions manually is not only time-consuming and error-prone, but may also lead to deviations in analytical logic, thus affecting the accuracy of financial decisions. Especially in the financial field, there are extremely high requirements for the accuracy and security of data processing, and any slight error may lead to significant economic losses. Therefore, developing a tool that can automatically and efficiently convert SAS code to target platform code is of great practical value to financial institutions. Summary of the invention
[0006] The main purpose of the present invention is to provide a SAS code automatic conversion method, device, equipment and storage medium, aiming to solve the technical problem of how to automatically and efficiently convert SAS code into target platform code to ensure the consistency of data processing logic and the accuracy of the conversion process.
[0007] To achieve the above object, the present invention provides a SAS code automatic conversion method, comprising:
[0008] Receiving a SAS code written based on a SAS platform, identifying and extracting a function name and a variable name of the SAS code;
[0009] Detect the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform, and generate table correspondence mapping;
[0010] Extract the data step part in the SAS code using a finite automaton, and parse the data step part into relational algebra using predefined grammar rules combined with the table correspondence mapping;
[0011] Calling the syntax parse tree of Hive, and converting the parsed relational algebra into Hive SQL code in combination with the table correspondence mapping;
[0012] The function names and variable names in the SAS code are converted into corresponding function names and variable names in the Hive platform, the target Hive code is generated by combining the Hive SQL code with the function names and variable names of the Hive platform, and the target Z+ platform code is converted and generated based on the target Hive code.
[0013] In one embodiment, converting and generating target Z+ platform code based on the target Hive code includes:
[0014] Build a parallel corpus, which includes SAS code, Hive code generated from SAS code, and Z+ platform code corresponding to Hive code, covering translation tasks in various scenarios;
[0015] A Transformer-based BERT model is used to perform model training using the parallel corpus, wherein the BERT model optimizes the translation model by learning the mapping rules between the SAS code and the target Hive code and the conversion rules between the target Hive code and the Z+ platform code;
[0016] During the model training process, regularization terms are added to adjust the loss function in the model training to obtain the final target translation model.
[0017] The SAS code is converted into the target Z+ platform code based on the target translation model.
[0018] In one embodiment, the BERT model based on Transformer uses the parallel corpus to perform model training, including:
[0019] Input the data in the parallel corpus into the Transformer-based BERT model for training, wherein the structure of the BERT model includes a multi-head attention mechanism, a feedforward neural network layer, and position encoding;
[0020] During the model training process, the learning rate decay strategy is applied to gradually reduce the learning rate as the training progresses;
[0021] Calculate the loss function in model training, optimize the model parameters through the back-propagation algorithm, and fine-tune the last preset number of layers of the BERT model.
[0022] In one embodiment, converting the SAS code into the target Z+ platform code based on the target translation model includes:
[0023] During the transformation process, the generated intermediate results are tested for outliers and anomalies are recorded and warnings are generated when they are detected;
[0024] After generating the preliminary Z+ platform code, verify the grammatical correctness of the preliminary Z+ platform code through a grammar checking tool, and automatically repair it or prompt the user to manually correct it when a grammatical error is detected;
[0025] Perform logic verification on the generated preliminary Z+ platform code to ensure that the data processing logic is consistent with the input SAS code, and verify the correctness of function and variable mapping through comparative analysis methods;
[0026] Execute the preliminary Z+ platform code in a simulation environment, record and analyze the operation log, detect potential runtime errors and logic anomalies, and generate detailed error reports and repair suggestions;
[0027] Verify the results of the simulation execution and compare whether the results of the simulation execution are consistent with the expected output. If they are inconsistent, mark the problem and prompt the user to review;
[0028] After completing the verification process and fixing errors, generate the final target Z+ platform code.
[0029] In one embodiment, converting the function names and variable names in the SAS code into corresponding function names and variable names in the Hive platform includes:
[0030] According to the pre-defined mapping rule table, find the corresponding names of the function names and variable names in the SAS code in the Hive platform;
[0031] According to the mapping rule table, the function names and variable names in the SAS code are replaced one by one and converted into corresponding function names and variable names in the Hive platform;
[0032] After the conversion is complete, perform a syntax check. If any naming or syntax issues that do not meet the Hive platform requirements are found, adjust the function names and variable names in the converted Hive platform.
[0033] In one embodiment, receiving a SAS code written based on a SAS platform, identifying and extracting a function name and a variable name of the SAS code, comprises:
[0034] Receive SAS code written based on the SAS platform and store it in memory;
[0035] Using a syntax parser to preliminarily parse the SAS code, construct a syntax tree of the code, and mark syntax elements therein, including function names and variables;
[0036] Analyze each function call node in the syntax tree, identify the nested function structure therein, and mark the nodes containing other function calls in the parameters as nested function calls;
[0037] For the identified nested function calls, recursively parse the inner functions, starting from the innermost function and gradually parse outwards, and store the parsed function names in the mapping table in sequence;
[0038] Recognize complex data structures in the syntax tree, including multidimensional arrays and nested data frames or data sets, and label the nodes of these complex data structures;
[0039] The complex data structure is parsed layer by layer, the variable names of the data elements of each layer are extracted, and the variable names are stored in a variable mapping table, and the variable names of each dimension are recorded in sequence.
[0040] In one embodiment, detecting the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform and generating a table correspondence mapping includes:
[0041] Extract metadata of data tables in the SAS platform, including table name, field name, data type, primary key and foreign key;
[0042] Extract metadata of the corresponding data table in the Hive platform, including table name, field name, data type, partition key, primary key, and foreign key;
[0043] Based on the logical relationship between primary keys and foreign keys, define the correspondence between SAS data tables and Hive data tables;
[0044] When the mapping relationship between the primary key and the foreign key is not sufficient to determine the corresponding relationship between the tables, the mapping relationship is defined based on the name similarity between the table name and the field name, and the string matching algorithm is used to calculate the similarity and determine whether there is a mapping relationship;
[0045] If the mapping relationship cannot be determined even after name matching, further analyze the structural similarity of the tables, including the number of fields, data type distribution, and hierarchical relationships of the tables, identify and record the existing mapping relationships, and generate a table correspondence mapping between the SAS data table and the Hive data table.
[0046] Furthermore, to achieve the above object, the present invention also provides a SAS code automatic conversion device, comprising:
[0047] A code receiving and parsing module receives SAS code written based on the SAS platform, identifies and extracts function names and variable names of the SAS code;
[0048] The data table mapping module detects the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform, and generates a table correspondence mapping;
[0049] A SAS code parsing module, which uses a finite automaton to extract a data step part in the SAS code, and uses a predefined grammar rule in combination with the table correspondence mapping to parse the data step part into relational algebra;
[0050] An SQL conversion module calls the syntax parsing tree of Hive and converts the parsed relational algebra into Hive SQL code in combination with the table correspondence mapping;
[0051] The code conversion and generation module converts the function names and variable names in the SAS code into the corresponding function names and variable names in the Hive platform, generates the target Hive code by combining the Hive SQL code with the function names and variable names of the Hive platform, and converts and generates the target Z+ platform code based on the target Hive code.
[0052] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a SAS code automatic conversion device, which includes a memory, a processor, and a SAS code automatic conversion program stored in the memory and executable on the processor, and when the SAS code automatic conversion program is executed by the processor, the steps of the SAS code automatic conversion method described above are implemented.
[0053] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a SAS code automatic conversion program is stored, and when the SAS code automatic conversion program is executed by a processor, the steps of the SAS code automatic conversion method described above are implemented.
[0054] Beneficial effect: The present invention relates to a SAS code automatic conversion method, which receives and parses SAS code written based on the SAS platform, identifies and extracts function names and variable names therein; detects the data table correspondence between the SAS platform and the Hive platform, and generates a table correspondence mapping; uses a finite automaton to parse the data step part in the SAS code, and combines predefined grammatical rules and table correspondence mapping to parse it into relational algebra; calls the syntax parsing tree of Hive to convert the parsed relational algebra into Hive SQL code; converts the function name and variable name in the SAS code into the corresponding item in the Hive platform, generates the target Hive code based on the generated Hive SQL code, and finally converts it into the target Z+ platform code. The present invention not only improves the automation and efficiency of the conversion process, but also ensures the consistency of data processing logic, and provides strong technical support for the digital transformation of the financial field. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0056] Figure 1 This is a flow chart of an embodiment of a method for automatic conversion of SAS codes according to the present invention;
[0057] Figure 2 It is a functional module schematic diagram of a preferred embodiment of the SAS code automatic conversion device of the present invention;
[0058] Figure 3 It is a structural schematic diagram of the equipment hardware operating environment involved in the embodiment of the SAS code automatic conversion equipment of the present invention. DETAILED DESCRIPTION
[0059] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0060] It should be noted that in the modern financial industry, data analysis and processing capabilities have become an important support for corporate decision-making and operations. Financial institutions rely on complex data analysis to conduct risk assessment, customer segmentation, market forecasting, and anti-fraud detection. To this end, financial institutions usually use a variety of data processing tools and platforms, such as SAS (Statistical Analysis System) and various big data platforms (such as Hive, Z+ platform) to manage and analyze massive financial data.
[0061] The SAS platform has played an important role in the financial sector with its powerful data analysis capabilities and a wide range of application scenarios. However, with the explosive growth of data volume and the development of cloud computing technology, financial institutions are gradually turning to more scalable and efficient cloud-based big data platforms for data processing and storage. These platforms (such as Hive and Z+ platforms) can not only process large-scale data, but also support parallel computing and distributed storage, providing higher computing performance and lower costs for financial analysis.
[0062] A key challenge in this transformation is how to seamlessly migrate existing SAS code and analytical logic to these emerging big data platforms. Traditional SAS codes often contain complex logic and data processing steps that involve cleaning, transforming, analyzing, and visualizing financial data. In order to adapt to the characteristics of the new platform, these SAS codes need to be converted into the code form of the target platform, such as Hive SQL or Z+ platform code.
[0063] However, completing such conversions manually is not only time-consuming and error-prone, but may also lead to deviations in analytical logic, thus affecting the accuracy of financial decisions. Especially in the financial field, there are extremely high requirements for the accuracy and security of data processing, and any slight error may lead to significant economic losses. Therefore, developing a tool that can automatically and efficiently convert SAS code to target platform code is of great practical value to financial institutions.
[0064] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the SAS code automatic conversion method provided by the present invention. It should be noted that although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that here.
[0065] like Figure 1 As shown, the SAS code automatic conversion method proposed by the present invention comprises the following steps:
[0066] S10, receiving a SAS code written based on a SAS platform, identifying and extracting a function name and a variable name of the SAS code;
[0067] In this embodiment, code files written by the SAS platform are obtained from the financial system and these codes are preliminarily analyzed. Specifically, the system scans the entire SAS code and automatically identifies all function names and variable names used in the code. Function names may include statistical functions such as MEAN, SUM, etc., while variable names may correspond to field names in financial data sets, such as account_balance, transaction_date, etc. By identifying and extracting these function names and variable names, the system can further parse and process the data processing logic in the code.
[0068] The system can set a step value (such as parsing 50 lines of code each time) to gradually scan the entire SAS code file and identify all function names and variable names through regular expressions or syntax parsers. The identification of function names can be limited to the function libraries commonly used in the field of financial statistics, and the extraction of variable names is analyzed based on the declaration part and usage context in the code.
[0069] In one specific implementation, the system receives SAS code directly from the user terminal, which may be written locally and uploaded to the system for conversion. The system parses the code in real time, automatically identifies all function names and variable names, and allows the user to make manual adjustments after identification to ensure accuracy. The system then analyzes each data processing step in the code, maps the function names and variable names to the corresponding target platform code, and optimizes performance in the process. For example, when the code involves complex financial derivative calculations, the system can identify these calculation functions and convert them into optimized equivalent functions on the target platform. This implementation is suitable for scenarios where financial data is processed in real time, such as stock market transaction analysis or the execution of real-time credit scoring models.
[0070] By automatically identifying and extracting function names and variable names in SAS code, the efficiency and accuracy of code conversion are significantly improved, the necessity of manual intervention is reduced, and the risk of operational errors is reduced. It is suitable for large-scale data processing scenarios in the financial field, and can ensure the consistency and accuracy of data processing logic between different platforms, and improve the execution performance on the target platform through optimized code, providing reliable technical support for financial data analysis.
[0071] S20, detecting the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform, and generating a table corresponding relationship mapping;
[0072] In this embodiment, data tables on the SAS and Hive platforms are compared, including attributes such as table name, field name, data type, primary key and foreign key, and a mapping table is generated based on the similarity or matching degree of these attributes to clarify the corresponding table of each table in the SAS platform in the Hive platform. For example, the table "customer_data" in the SAS platform may correspond to the table "hive_customer_data" in the Hive platform. The system generates a mapping relationship by analyzing the structure and content of the two tables.
[0073] The table relationship between the two platforms can be determined by string matching of table names and field names, logical analysis of primary keys and foreign keys, and comparison of data types. The step size can be selected to compare each field one by one, and the range can be set to calculate the similarity of all fields in the SAS table and the Hive table.
[0074] In one specific implementation, the system extracts the table structures of the SAS platform and the Hive platform from multiple databases of a financial institution to generate a cross-platform table structure mapping. In this implementation, the system enhances the accuracy of the mapping not only based on the similarity of table names and field names, but also by analyzing the content distribution and usage frequency of the data tables. For example, the system can analyze the usage patterns of the data table "account_balance" on the SAS platform and the data table "hive_account_balances" on the Hive platform in the historical records to determine the mapping relationship between them. After generating the mapping table, the system will give priority to converting tables that are used frequently to ensure timely processing of data. This implementation is suitable for the data migration needs of financial institutions during business integration or system upgrades.
[0075] In another specific implementation, the system batch processes data tables of multiple SAS platforms and automatically generates mapping relationships corresponding to the Hive platform. In this way, the system can automatically identify special table relationships, such as composite primary keys or nested table structures, and perform special processing. When generating the mapping, the system also considers the hierarchical relationships of data tables, such as the structure of parent-child tables or master-slave tables, to ensure that these complex relationships are not destroyed during the conversion process. After the mapping table is generated, the system applies it to batch migration and conversion of data to ensure that all related data tables maintain logical consistency and data integrity on both platforms. This implementation method is suitable for large-scale data processing scenarios, such as data lake construction of financial institutions or full-bank data migration projects.
[0076] By detecting the logical relationship between the data tables of the SAS platform and the Hive platform and generating table correspondence mapping, the problem of inconsistent table structure and damaged logical relationship in the cross-platform data migration process is effectively solved. The consistency of data during the conversion and migration process is ensured, the risk of data loss or mismatch is reduced, and the efficiency and accuracy of cross-platform data processing are improved.
[0077] S30, extracting the data step part in the SAS code by using a finite automaton, and parsing the data step part into relational algebra by using predefined grammar rules combined with the table correspondence mapping;
[0078] In this embodiment, the data step part in the SAS code is extracted and converted into a relational algebra form to achieve code conversion and execution on the target platform (such as Hive or Z+ platform). Finite automaton is a computational model for syntax parsing that can gradually read each character or symbol of the code and identify the start and end positions of the data step part according to predefined state transition rules. After extraction, the system uses predefined syntax rules and table correspondence mapping to convert the data processing logic in the data step into relational algebra. This algebraic form is a universal data processing expression that can be applied in different database systems.
[0079] Finite automata may recognize SAS code similar to "datamydata; set customer_data;" and parse the content of the data step according to predefined grammar rules. The system will further convert this code into relational algebra, such as "π_customer_data(σ_condition)" means to project the data table when a certain condition is met.
[0080] In one specific implementation, not only the data step part is extracted, but also the nested datastep structure can be processed. In the SAS code used by financial institutions, nested data processing logic often appears, such as performing an aggregation operation first and then filtering the aggregation result. In this case, the system uses a finite automaton to parse the nested structure layer by layer, converts each nested operation into a relational algebraic expression, and finally merges them into a complete algebraic expression to ensure the integrity and consistency of the logic. For example, a nested aggregation operation may be parsed as a combination of multiple connection operations. This implementation is suitable for conversion scenarios of financial report generation or multi-level data aggregation.
[0081] In other specific embodiments, the system can process complex data operations contained in the data step, such as sorting, joining, and union operations, while extracting and parsing the data step through a finite automaton. In this embodiment, the system converts these complex operations into corresponding operators in relational algebra in conjunction with table-to-table relationship mapping. For example, when processing the integration of multiple financial data sources, the system can parse the joint query of multiple data tables into a combination of union and join operations in relational algebra, and generate an efficient execution plan through an optimization algorithm. This embodiment is suitable for scenarios of financial data integration and cross-platform data linkage analysis.
[0082] By using finite automata to extract and parse the data step part of the SAS code, the system can convert complex data processing logic into relational algebra expressions, thereby achieving automatic conversion and optimized execution of cross-platform codes. This significantly improves the efficiency and accuracy of data processing, especially when dealing with financial scenarios involving multiple data sources or complex data operations. It can ensure the consistency and integrity of data processing logic during the conversion process, providing financial institutions with flexible and efficient data migration and analysis tools.
[0083] S40, calling the syntax parse tree of Hive, and converting the parsed relational algebra into Hive SQL code in combination with the table correspondence mapping;
[0084] In this embodiment, the SAS code that has been parsed into the form of relational algebra is converted into Hive SQL code through the syntax parse tree of the Hive platform. The syntax parse tree of Hive is a data structure used to convert relational algebra expressions into specific SQL queries. During the conversion process, the system will map the table names, field names, etc. in the relational algebra to the corresponding entities in the Hive platform in combination with the previously generated table correspondence mapping to generate SQL code that can be correctly executed on the Hive platform. For example, a relational algebra expression may be converted into a HiveSQL query containing multiple JOIN, SELECT, and WHERE clauses.
[0085] You can use Hive's syntax parse tree to parse relational algebra expressions, such as "π_customer_data(σ_condition)", and convert it to "SELECT * FROM hive_customer_data WHERE condition;". During the conversion process, the system maps the table name "customer_data" in the relational algebra to the table name "hive_customer_data" in the Hive platform according to the table mapping.
[0086] In one specific implementation, the system processes multi-level relational algebraic expressions and generates corresponding multiple segments of Hive SQL code. In this way, the system can decompose a complex nested relational algebraic expression into multiple relatively simple SQL subqueries, and finally combine them into a complete SQL query. For example, in the process of generating financial reports containing nested subqueries and union operations, the system will parse the nested relational algebraic expressions layer by layer, convert each layer of expression into a subquery, and then combine these subqueries into a complete Hive SQL statement to ensure the accuracy and efficiency of the report generation process. This implementation is suitable for complex financial report generation and multi-level data analysis scenarios.
[0087] In other specific implementations, the system not only uses Hive's syntax parse tree to convert relational algebra expressions into Hive SQL code, but also optimizes the SQL code after it is generated. In the actual operation of financial institutions, the generated SQL queries often involve large-scale data sets, and direct execution may cause performance problems. In order to improve query efficiency, after generating the SQL code, the system will automatically adjust the structure of the SQL query in combination with the execution plan analysis function of the Hive platform. For example, the system may rearrange the order of JOIN operations, or convert certain subqueries into temporary tables to reduce repeated calculations. This implementation method is suitable for financial data query and real-time analysis scenarios with high performance requirements.
[0088] Through the above steps, the conversion process from SAS code to Hive platform code can be completed automatically and efficiently. It not only ensures the consistency and correctness of the data processing logic during the conversion process, but also solves the problem of inconsistent table names and field names across platforms by combining table correspondence mapping.
[0089] S50, converting the function names and variable names in the SAS code into corresponding function names and variable names in the Hive platform, generating a target Hive code by combining the Hive SQL code with the function names and variable names of the Hive platform, and converting and generating a target Z+ platform code based on the target Hive code.
[0090] In this embodiment, the function names and variable names in the SAS code are mapped to the corresponding items in the Hive platform, and the target Hive code is generated. These mappings include common function names, such as the MEAN function in SAS may be mapped to the AVG function in Hive, and the direct mapping of variable names or adjustment according to the context. The generated Hive code is a code that is further optimized based on Hive SQL and can be executed directly on the Hive platform. Subsequently, the system generates the corresponding Z+ platform code based on the target Hive code to ensure the consistency and compatibility of the code on multiple platforms. This process includes multiple mappings and conversions of function and variable names, which ensures the accuracy and effectiveness of the conversion from SAS code to the final Z+ platform code.
[0091] For example, the function LOG in the SAS code may be mapped to the LN function in the Hive platform, and the variable name such as customer_id may be mapped to hive_customer_id in the Hive platform. After the system generates the Hive SQL code, it will further convert it into the target Hive code according to the mapping relationship, and then generate the Z+ platform code based on the target Hive code, which may include mapping the Hive function and variable names to the corresponding items of the Z+ platform again.
[0092] In one specific implementation, not only are the function names and variable names in the SAS code mapped and replaced, but these mappings are also customized and extended. For example, for some SAS functions that do not have direct corresponding functions on the Hive platform, the system can call custom function libraries to implement equivalent functions. The generated target Hive code calls these custom functions to ensure that the data processing logic is consistent with the original SAS code. Subsequently, the system converts the target Hive code into Z+ platform code. During this process, the custom function library is also mapped or ported to the Z+ platform to ensure that the converted code can be correctly executed on the Z+ platform. This implementation is suitable for scenarios in financial institutions where data processing logic is highly customized, such as the migration of unique risk pricing models or customer segmentation algorithms.
[0093] In other specific implementations, the system supports batch processing and optimization of function names and variable names when converting SAS code to Hive platform code. After generating the target Hive code, the system automatically adjusts the mapping rules of certain functions and variables according to the characteristics of the Z+ platform. For example, when generating Z+ platform code, the system may merge or rearrange certain functions to improve the execution efficiency of the code on the Z+ platform. In this process, the system not only maps function and variable names, but also performs certain optimization processing on the code structure, such as merging multiple steps into one step to reduce the computational burden at runtime. This implementation method is suitable for scenarios of large-scale financial data processing, especially in batch data analysis tasks that require efficient execution.
[0094] By mapping the function names and variable names in the SAS code to the corresponding items in the Hive platform and Z+ platform and generating the target code, the cross-platform data processing logic conversion and optimization is achieved. This ensures the consistency and compatibility of the code on different platforms, reduces the complexity and error risk of manual conversion, and improves the efficiency of code execution.
[0095] The present invention relates to a SAS code automatic conversion method, which receives and parses the SAS code written based on the SAS platform, identifies and extracts the function name and variable name therein; detects the data table correspondence between the SAS platform and the Hive platform, and generates a table correspondence mapping; uses a finite automaton to parse the data step part in the SAS code, and combines predefined grammatical rules and table correspondence mapping to parse it into relational algebra; calls the syntax parsing tree of Hive to convert the parsed relational algebra into Hive SQL code; converts the function name and variable name in the SAS code into the corresponding item in the Hive platform, generates a target Hive code based on the generated Hive SQL code, and finally converts it into a target Z+ platform code. The present invention not only improves the automation and efficiency of the conversion process, but also ensures the consistency of data processing logic, and provides strong technical support for the digital transformation of the financial field.
[0096] In one embodiment, in the above S50, converting and generating the target Z+ platform code based on the target Hive code includes:
[0097] S501, constructing a parallel corpus, wherein the parallel corpus includes SAS code, Hive code generated from the SAS code, and Z+ platform code corresponding to the Hive code, covering translation tasks in various scenarios;
[0098] S502, a Transformer-based BERT model is used to perform model training using the parallel corpus, wherein the BERT model optimizes the translation model by learning a mapping rule between the SAS code and the target Hive code and a conversion rule between the target Hive code and the Z+ platform code;
[0099] S503, during the model training process, adding a regularization term, adjusting the loss function in the model training, and training to obtain a final target translation model;
[0100] S504: Convert the SAS code into the target Z+ platform code based on the target translation model.
[0101] In this embodiment, the parallel corpus is a database or data set containing comparative code examples that show how to generate Hive code from SAS code and further convert Hive code to Z+ platform code. Through these comparative examples, the system can learn the patterns and rules of code conversion between different platforms. Parallel corpus components:
[0102] SAS code: Original SAS code, used for tasks such as processing and analyzing financial data. These codes may include operations such as data query, calculation, and statistical analysis.
[0103] Hive Code: These are SAS codes that are converted and run on the Hive platform. Hive codes are usually related to big data processing and can be executed in distributed systems.
[0104] Z+ platform code: The final generated code is suitable for the Z+ platform. The Z+ platform may have different functional features and syntax requirements, so further conversion is required.
[0105] The data in the corpus covers a variety of application scenarios in the financial field, such as risk assessment, customer analysis, market forecasting, etc. This ensures that the model can handle a wide range of conversion tasks and is not limited to a certain application scenario. For example, in a financial scenario, the corpus may contain a SAS code segment for credit scoring and its converted Hive code and Z+ platform code. By learning these examples, the system can understand how to automatically convert SAS code into Z+ platform code in similar tasks.
[0106] BERT (Bidirectional Encoder Representations from Transformers) is a deep learning model based on the Transformer architecture that is good at handling language-related tasks. In this technical solution, the BERT model is used to learn the complex mapping relationship in code conversion, understand how to generate Hive code from SAS code, and further generate Z+ platform code.
[0107] During the model training process, the BERT model will continuously obtain control examples from the parallel corpus to learn how to convert SAS code to Hive code, and how to further convert Hive code to Z+ platform code. Through this training, the model gradually masters the rules and patterns of code conversion between platforms. For example, the BERT model may learn that the DATA step used in SAS code usually corresponds to the CREATE TABLE operation in Hive, and these operations may require further syntax adjustments or function mappings on the Z+ platform. Through repeated training, the model can automatically handle similar conversion tasks.
[0108] Regularization is a technique used to prevent model overfitting. By imposing certain constraints on model parameters (such as L1 or L2 regularization), the model can avoid overfitting to the training data, thereby improving the generalization ability on new data. The loss function is an indicator that measures the gap between the model's predicted results and the actual results. By adjusting the loss function, the system can better optimize the model's training process so that it can more accurately learn the conversion rules between SAS code, Hive code, and Z+ platform code. Example: During the training process, if the model performs poorly on certain complex code conversion tasks, the system can guide the model to better learn these tasks by adding regularization terms or adjusting the loss function, avoiding excessive bias in training data on specific tasks.
[0109] The target translation model is a trained and optimized BERT model that has mastered the complex conversion rules for generating Hive code from SAS code and then generating Z+ platform code from Hive code. Once the model training is completed, the system can directly use the model to process new SAS code and automatically convert it into code suitable for running on the Z+ platform. This process utilizes the conversion rules learned by the model during training to ensure the accuracy and efficiency of the conversion.
[0110] In one specific implementation, the parallel corpus not only includes historical data, but is also continuously updated with new conversion tasks. When developers at financial institutions submit new SAS code for conversion, the system stores the conversion results in the corpus and regularly performs incremental training on the BERT model. This dynamically updated parallel corpus ensures that the model can adapt to new business needs and technological changes. During the training process, the system adjusts the regularization terms for different data sets and flexibly sets the learning rate to ensure that the model can be stably trained with data of different sizes. Through this continuous learning and optimization, the system can quickly generate efficient and accurate Z+ platform code when faced with new financial data processing needs. This implementation is particularly suitable for real-time data processing and multi-platform linkage analysis scenarios in financial institutions.
[0111] Through the above steps, this embodiment can automatically learn and optimize the conversion rules between SAS code and Hive code and Z+ platform code, thereby achieving efficient and accurate code conversion. This method significantly improves the generalization ability of the model, reduces manual intervention, and can adapt to the needs of various financial scenarios. Adding regularization terms and adjusting loss functions during the training process ensures the stability and accuracy of the model under different data scales and complexities, providing reliable technical support for the automation and intelligence of financial data processing.
[0112] In one embodiment, in S502 above, the BERT model based on Transformer is trained using the parallel corpus, including:
[0113] S5021, inputting the data in the parallel corpus into a Transformer-based BERT model for training, wherein the structure of the BERT model includes a multi-head attention mechanism, a feedforward neural network layer, and a position encoding;
[0114] S5022, during the model training process, apply the learning rate decay strategy, and gradually reduce the learning rate as the training progresses;
[0115] S5023, calculate the loss function in model training, optimize the model parameters through the back propagation algorithm, and fine-tune the last preset number of layers of the BERT model.
[0116] In this embodiment, the SAS code, Hive code and Z+ platform code in the parallel corpus are input into the BERT model for training. The BERT model uses these control data to learn how to map and convert between SAS code and Hive code, and between Hive code and Z+ platform code.
[0117] BERT model structure:
[0118] Multi-Head Attention: One of the core mechanisms of the BERT model, it can capture the relationship between different parts of the input data and process them in parallel. The multi-head attention mechanism allows the model to focus on other related parts when processing each part of the data, improving the model's ability to understand complex transformation rules.
[0119] Feedforward Neural Networks (FFNN): The feedforward neural network layer in the BERT model is used to perform further nonlinear transformation on the output of the attention mechanism to improve the expressiveness of the model.
[0120] Positional Encoding: Because the Transformer model itself has no sequence information, positional encoding is used to add sequence information to the input data to help the model understand the order of the data, which is very important in code conversion.
[0121] During model training, the system applies a learning rate decay strategy, which means that at the beginning of training, the model uses a larger learning rate to adjust parameters faster, and as training progresses, the learning rate gradually decreases to avoid excessive fluctuations in model parameters and ensure that the model eventually converges to a better solution. By calculating the loss function, the system can evaluate the performance of the model and optimize the model's parameters through the back-propagation algorithm. During this process, the system fine-tunes the last few layers of the BERT model to further improve the model's translation capabilities.
[0122] Through the above steps, this embodiment can take advantage of the Transformer architecture to efficiently learn and process complex code conversion tasks. Through the learning rate decay strategy, the model can gradually converge to a better solution, and back propagation and fine-tuning improve the conversion accuracy of the model.
[0123] In one embodiment, in the above S504, converting the SAS code into the target Z+ platform code based on the target translation model includes:
[0124] S5041, during the conversion process, performing outlier detection on the generated intermediate results, and recording and generating warnings when an anomaly is detected;
[0125] S5042, after generating the preliminary Z+ platform code, verify the grammatical correctness of the preliminary Z+ platform code through a grammar checking tool, and automatically repair it or prompt the user to manually correct it when a grammatical error is detected;
[0126] S5043, performing logic verification on the generated preliminary Z+ platform code to ensure that the data processing logic is consistent with the input SAS code, and verifying the correctness of function and variable mapping through comparative analysis method;
[0127] S5044, executing the preliminary Z+ platform code in a simulation environment, recording and analyzing the operation log, detecting potential runtime errors and logical anomalies, and generating a detailed error report and repair suggestions;
[0128] S5045, verifying the result of the simulation execution, comparing the result of the simulation execution with the expected output to see if they are consistent, if not, marking the problem and prompting the user to review;
[0129] S5046, after completing the verification process and correcting the errors, generate the final target Z+ platform code.
[0130] In this embodiment, during the conversion process, the system will perform outlier detection on the generated intermediate results. If anomalies are detected, the system will automatically record these anomalies and generate warning information. Outlier detection is usually used to identify logical or data errors that may occur during the conversion process to ensure the rationality and accuracy of the intermediate results.
[0131] After the preliminary Z+ platform code is generated, the system verifies the syntactic correctness of the code through a syntax checker. If syntax errors are found, the system will attempt to automatically fix them; if the automatic fix fails, the system will prompt the user to make manual corrections. Syntax checking is a key step to ensure that the generated code can be executed correctly. The system performs logic verification on the generated preliminary Z+ platform code to ensure that the data processing logic is consistent with the original SAS code. This step verifies the correctness of function and variable mappings through comparative analysis methods to ensure that there are no logical errors or inconsistencies in the conversion process.
[0132] The system will execute the preliminary Z+ platform code in a simulation environment, record and analyze the operation log to detect potential runtime errors and logical anomalies. The system will generate detailed error reports and repair suggestions to help users understand and solve the problem. The system will verify the results of the simulation execution and compare whether the results are consistent with the expected output. If the results are inconsistent, the system will mark the problem and prompt the user to review it. This step ensures that the final generated code can meet the expected functional requirements.
[0133] After all verification processes are completed and errors are corrected, the system will generate the final target Z+ platform code. This code has been checked and verified multiple times to ensure its execution performance and correctness on the Z+ platform.
[0134] This embodiment ensures the accuracy and consistency of the conversion from SAS code to Z+ platform code by performing outlier detection, syntax checking, logic verification, and simulation execution on the generated intermediate results. The multi-level verification process ensures that each conversion step is strictly checked, reducing the risk of code errors or logical inconsistencies.
[0135] In one embodiment, in the above S50, converting the function names and variable names in the SAS code into corresponding function names and variable names in the Hi ve platform includes:
[0136] S505, searching for corresponding names of function names and variable names in the SAS code in the Hive platform according to a predefined mapping rule table;
[0137] S506, replacing the function names and variable names in the SAS code one by one according to the mapping rule table, and converting them into corresponding function names and variable names in the Hive platform;
[0138] S507, after the conversion is completed, a syntax check is performed. If any naming or syntax problems that do not meet the requirements of the Hive platform are found, the function names and variable names in the converted Hive platform are adjusted.
[0139] In this embodiment, the system searches for the corresponding names of the function names and variable names in the SAS code in the Hive platform according to the predefined mapping rule table. The mapping rule table contains the correspondence between common SAS functions and Hive functions, and SAS variables and Hive variables. These rules may be established based on business logic, grammatical structure or previous conversion experience.
[0140] The system replaces the function names and variable names in the SAS code one by one according to the corresponding names found, and converts them into the corresponding function names and variable names in the Hive platform. This step ensures that all functions and variables involved in the code conversion process can match the syntax and functions of the Hive platform.
[0141] After the replacement is completed, the system will perform a syntax check on the converted Hive code. If it is found that the converted function name or variable name does not conform to the naming rules of the Hive platform or has syntax problems, the system will adjust it to ensure that the generated Hive code can run correctly.
[0142] In one specific implementation, the system supports defining custom mapping rules in the mapping rule table, allowing users to adjust the mapping relationship according to specific business needs. For example, some financial institutions may have specific naming conventions or custom functions on the Hive platform, which can be added in the mapping rule table. During the actual conversion process, the system will find and replace function names and variable names in the SAS code according to these custom rules. After the conversion, the system will call the built-in syntax checker of the Hive platform to perform a comprehensive check on the generated code. If naming or syntax issues that do not meet the platform requirements are found, the system will generate a detailed error report and provide automatic adjustment suggestions to help users quickly fix the problem
[0143] This embodiment ensures the accuracy and executability of code conversion by converting function names and variable names in SAS code into corresponding names in the Hive platform and combining syntax checking and adjustment.
[0144] In one embodiment, in the above S10, receiving a SAS code written based on a SAS platform, and identifying and extracting a function name and a variable name of the SAS code include:
[0145] S101, receiving SAS code written based on the SAS platform and storing it in a memory;
[0146] Using a syntax parser to preliminarily parse the SAS code, construct a syntax tree of the code, and mark syntax elements therein, including function names and variables;
[0147] S102, analyzing each function call node in the syntax tree, identifying the nested function structure therein, and marking the node containing other function calls in the parameters as a nested function call;
[0148] S103, recursively parse the inner functions of the identified nested function calls, parse from the innermost function to the outermost function, and store the parsed function names in a mapping table in sequence;
[0149] S104, identifying complex data structures in the syntax tree, including multidimensional arrays and nested data frames or data sets, and marking nodes of these complex data structures;
[0150] S105, parsing the complex data structure layer by layer, extracting the variable names of the data elements of each layer, and storing them in a variable mapping table, recording the variable names of each dimension in sequence.
[0151] In this embodiment, the system first receives the SAS code obtained from the SAS platform and loads it into the memory for subsequent processing. This step ensures that the code data can be quickly accessed during the processing.
[0152] The system uses a syntax parser to perform preliminary parsing of SAS code. The parser analyzes the structure of the code and generates a syntax tree. The syntax tree is a structured representation of the code that shows how the various elements in the code are related to each other. In the process of building the syntax tree, the system marks the key syntax elements in the code, such as function names and variable names.
[0153] The system analyzes each function call node in the syntax tree one by one to identify which nodes contain nested function calls (that is, the parameters of the function contain other function calls). The system marks these nested calls for special treatment in subsequent processing.
[0154] For marked nested function calls, the system will recursively parse its inner functions. The parsing process starts from the innermost function and gradually parses outward to ensure that each layer of nested functions is correctly understood and processed. After parsing is completed, the system stores these function names in a mapping table in sequence for use in subsequent processing.
[0155] The system recognizes complex data structures in the syntax tree, such as multidimensional arrays, nested data frames, or datasets. These structures often involve complex data operations, and the system marks these nodes for special attention in subsequent processing.
[0156] The system parses the complex data structure in the syntax tree layer by layer, processing from the outermost layer to the innermost layer in turn, and extracts the variable names of the data elements in each layer. The system stores these variable names in the variable mapping table and records the variable names of each dimension in sequence to ensure that subsequent processing can correctly identify and operate these data.
[0157] In this embodiment, by identifying and extracting the function names and variable names of the SAS code, the system can effectively handle complex financial data code conversion tasks and ensure the correct identification and mapping of all elements in the code conversion process.
[0158] In one embodiment, the above S20 includes:
[0159] S201, extracting metadata of a data table in the SAS platform, including table name, field name, data type, primary key, and foreign key;
[0160] S202, extracting metadata of the corresponding data table in the Hive platform, including table name, field name, data type, partition key, primary key, and foreign key;
[0161] S203, based on the logical relationship between the primary key and the foreign key, define the corresponding relationship between the SAS data table and the Hive data table;
[0162] S204, when the mapping relationship between the primary key and the foreign key is not sufficient to determine the corresponding relationship between the tables, a mapping relationship is defined based on the name similarity between the table name and the field name, and a string matching algorithm is used to calculate the similarity and determine whether a mapping relationship exists;
[0163] S205, if the mapping relationship cannot be determined even after name matching, further analyze the structural similarity of the tables, including the number of fields, data type distribution, and hierarchical relationships of the tables, identify and record the existing mapping relationships, and generate a table correspondence mapping between the SAS data table and the Hive data table.
[0164] In this embodiment, the system extracts metadata from the data table of the SAS platform, and these metadata include table name, field name, data type, primary key and foreign key. Metadata is the descriptive information of the data table, which can help the system understand the structure and properties of the data table. The system also extracts metadata from the data table of the Hive platform. These metadata include not only the table name, field name, data type, but also the partition key unique to the Hive platform. Similar to the SAS platform, metadata can help the system identify the structure and properties of the Hive data table.
[0165] Using the extracted metadata, the system first attempts to define the correspondence between SAS data tables and Hive data tables through the logical relationship between primary keys and foreign keys. This step ensures that the association between data tables is correctly mapped by matching the primary keys and foreign keys of the tables.
[0166] If the logical relationship between the primary key and the foreign key is not sufficient to determine the correspondence between the tables, the system will further analyze the name similarity between the table name and the field name. The system uses a string matching algorithm to calculate the name similarity and determines whether there is a mapping relationship between the SAS data table and the Hive data table based on the similarity.
[0167] If the name similarity is not enough to determine the mapping relationship, the system will further analyze the structural similarity of the tables. This includes the number of fields, data type distribution, and hierarchical relationships of the tables. By analyzing these structural features, the system can identify and record the possible correspondence between SAS data tables and Hive data tables.
[0168] Finally, the system summarizes all the identified correspondences and generates a table correspondence mapping between SAS data tables and Hive data tables. This mapping table will be used in the subsequent data migration and conversion process to ensure the consistency of data structure between the SAS platform and the Hive platform.
[0169] In one specific implementation, when processing cross-platform data conversion, the system first extracts metadata from the SAS platform and the Hive platform through a multi-stage mapping process, and uses an advanced string matching algorithm to calculate the similarity between table names and field names. If the system cannot determine the correspondence only through the logical relationship between the primary key and the foreign key, it will further perform a similarity analysis of the table structure. The system will determine the mapping relationship between them based on the number of fields in the table, the distribution of data types, and the hierarchical relationship of the table. This multi-level mapping strategy ensures that the correspondence between data tables is accurately reflected on different platforms. Finally, the table correspondence mapping table generated by the system will be used in the data conversion tool to guide the cross-platform data migration process. This method is particularly suitable for financial system code migration and cross-platform integration tasks with complex data structures.
[0170] This embodiment detects the logical relationship between the SAS platform and the Hive platform data tables, and the system can generate an accurate table correspondence mapping table. It comprehensively utilizes metadata analysis, primary key and foreign key logical relationships, string similarity calculation, and table structure similarity analysis to ensure the correct mapping of cross-platform data tables and significantly improve the accuracy and efficiency of cross-platform data migration.
[0171] The present invention also provides a SAS code automatic conversion device, referring to Figure 2 , Figure 2 This is a functional module diagram of a preferred embodiment of the SAS code automatic conversion device of the present invention. The SAS code automatic conversion device comprises:
[0172] A code receiving and parsing module receives SAS code written based on the SAS platform, identifies and extracts function names and variable names of the SAS code;
[0173] The data table mapping module detects the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform, and generates a table correspondence mapping;
[0174] A SAS code parsing module, which uses a finite automaton to extract a data step part in the SAS code, and uses a predefined grammar rule in combination with the table correspondence mapping to parse the data step part into relational algebra;
[0175] An SQL conversion module calls the syntax parsing tree of Hive and converts the parsed relational algebra into Hive SQL code in combination with the table correspondence mapping;
[0176] The code conversion and generation module converts the function names and variable names in the SAS code into the corresponding function names and variable names in the Hive platform, generates the target Hive code by combining the Hive SQL code with the function names and variable names of the Hive platform, and converts and generates the target Z+ platform code based on the target Hive code.
[0177] The specific implementation of the SAS code automatic conversion device of the present invention is basically the same as the embodiments of the above-mentioned SAS code automatic conversion method, and will not be repeated here.
[0178] The present invention also provides a SAS code automatic conversion device, such as Figure 3 As shown, the SAS code automatic conversion device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0179] Those skilled in the art will understand that Figure 3 The hardware structure of the SAS code automatic conversion device shown in the figure does not constitute a limitation on the SAS code automatic conversion device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0180] like Figure 3 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a SAS code automatic conversion program. The operating system is a program for managing and controlling SAS code automatic conversion equipment and software resources, and supports the operation of the network communication module, the user interface module, the SAS code automatic conversion program, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.
[0181] exist Figure 3In the hardware structure of the SAS code automatic conversion device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the SAS code automatic conversion program stored in the memory 1005 and perform the same operation as the SAS code automatic conversion method.
[0182] The specific implementation of the SAS code automatic conversion device of the present invention is basically the same as the embodiments of the above-mentioned SAS code automatic conversion method, and will not be repeated here.
[0183] In addition, an embodiment of the present invention further provides a computer storage medium on which a SAS code automatic conversion program is stored. When the SAS code automatic conversion program is executed by a processor, the steps of the SAS code automatic conversion method described above are implemented.
[0184] The specific implementation of the computer storage medium of the present invention is basically the same as the embodiments of the above-mentioned SAS code automatic conversion method, and will not be repeated here.
[0185] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.
Claims
1. A SAS code automatic conversion method, characterized in that: The following steps are involved: Receiving a SAS code written based on a SAS platform, identifying and extracting a function name and a variable name of the SAS code; Detect the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform, and generate table correspondence mapping; Extract the data step part in the SAS code using a finite automaton, and parse the data step part into relational algebra using predefined grammar rules combined with the table correspondence mapping; Calling the syntax parse tree of Hive, and converting the parsed relational algebra into Hive SQL code in combination with the table correspondence mapping; The function names and variable names in the SAS code are converted into corresponding function names and variable names in the Hive platform, the target Hive code is generated by combining the Hive SQL code with the function names and variable names of the Hive platform, and the target Z+ platform code is converted and generated based on the target Hive code.
2. The SAS code automatic conversion method according to claim 1, characterized in that: The target Z+ platform code is converted and generated based on the target Hive code, including: Build a parallel corpus, which includes SAS code, Hive code generated from SAS code, and Z+ platform code corresponding to Hive code, covering translation tasks in various scenarios; A Transformer-based BERT model is used to perform model training using the parallel corpus, wherein the BERT model optimizes the translation model by learning the mapping rules between the SAS code and the target Hive code and the conversion rules between the target Hive code and the Z+ platform code; During the model training process, regularization terms are added to adjust the loss function in the model training to obtain the final target translation model. The SAS code is converted into the target Z+ platform code based on the target translation model.
3. The SAS code automatic conversion method according to claim 2, characterized in that: The Transformer-based BERT model uses the parallel corpus for model training, including: Input the data in the parallel corpus into the Transformer-based BERT model for training, wherein the structure of the BERT model includes a multi-head attention mechanism, a feedforward neural network layer, and position encoding; During the model training process, the learning rate decay strategy is applied to gradually reduce the learning rate as the training progresses; Calculate the loss function in model training, optimize the model parameters through the back-propagation algorithm, and fine-tune the last preset number of layers of the BERT model.
4. The SAS code automatic conversion method according to claim 2, characterized in that: Converting the SAS code into the target Z+ platform code based on the target translation model includes: During the transformation process, the generated intermediate results are tested for outliers and anomalies are recorded and warnings are generated when they are detected; After generating the preliminary Z+ platform code, verify the grammatical correctness of the preliminary Z+ platform code through a grammar checking tool, and automatically repair it or prompt the user to manually correct it when a grammatical error is detected; Perform logic verification on the generated preliminary Z+ platform code to ensure that the data processing logic is consistent with the input SAS code, and verify the correctness of function and variable mapping through comparative analysis methods; Execute the preliminary Z+ platform code in a simulation environment, record and analyze the operation log, detect potential runtime errors and logic anomalies, and generate detailed error reports and repair suggestions; Verify the results of the simulation execution and compare whether the results of the simulation execution are consistent with the expected output. If they are inconsistent, mark the problem and prompt the user to review; After completing the verification process and fixing errors, generate the final target Z+ platform code.
5. The SAS code automatic conversion method according to claim 1, characterized in that: Convert the function names and variable names in the SAS code to the corresponding function names and variable names in the Hive platform, including: According to the pre-defined mapping rule table, find the corresponding names of the function names and variable names in the SAS code in the Hive platform; According to the mapping rule table, the function names and variable names in the SAS code are replaced one by one and converted into corresponding function names and variable names in the Hive platform; After the conversion is complete, perform a syntax check. If any naming or syntax issues that do not meet the Hive platform requirements are found, adjust the function names and variable names in the converted Hive platform.
6. The SAS code automatic conversion method as described in claim 1, characterized in that: Receiving SAS code written based on the SAS platform, identifying and extracting function names and variable names of the SAS code, including: Receive SAS code written based on the SAS platform and store it in memory; Using a syntax parser to preliminarily parse the SAS code, construct a syntax tree of the code, and mark syntax elements therein, including function names and variables; Analyze each function call node in the syntax tree, identify the nested function structure therein, and mark the nodes containing other function calls in the parameters as nested function calls; For the identified nested function calls, recursively parse the inner functions, starting from the innermost function and gradually parse outwards, and store the parsed function names in the mapping table in sequence; Recognize complex data structures in the syntax tree, including multidimensional arrays and nested data frames or data sets, and label the nodes of these complex data structures; The complex data structure is parsed layer by layer, the variable names of the data elements of each layer are extracted, and the variable names are stored in a variable mapping table, and the variable names of each dimension are recorded in sequence.
7. The SAS code automatic conversion method according to claim 1, characterized in that: Detect the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform, and generate table correspondence mapping, including: Extract metadata of data tables in the SAS platform, including table name, field name, data type, primary key and foreign key; Extract metadata of the corresponding data table in the Hive platform, including table name, field name, data type, partition key, primary key, and foreign key; Based on the logical relationship between primary keys and foreign keys, define the correspondence between SAS data tables and Hive data tables; When the mapping relationship between the primary key and the foreign key is not sufficient to determine the corresponding relationship between the tables, the mapping relationship is defined based on the name similarity between the table name and the field name, and the string matching algorithm is used to calculate the similarity and determine whether there is a mapping relationship; If the mapping relationship cannot be determined even after name matching, further analyze the structural similarity of the tables, including the number of fields, data type distribution, and hierarchical relationships of the tables, identify and record the existing mapping relationships, and generate a table correspondence mapping between the SAS data table and the Hive data table.
8. A SAS code automatic conversion device, characterized in that: The SAS code automatic conversion device comprises: A code receiving and parsing module receives SAS code written based on the SAS platform, identifies and extracts function names and variable names of the SAS code; The data table mapping module detects the logical relationship between the data table in the SAS platform and the corresponding table in the Hive platform, and generates a table correspondence mapping; A SAS code parsing module, which uses a finite automaton to extract a data step part in the SAS code, and uses a predefined grammar rule in combination with the table correspondence mapping to parse the data step part into relational algebra; An SQL conversion module calls the syntax parsing tree of Hive and converts the parsed relational algebra into Hive SQL code in combination with the table correspondence mapping; The code conversion and generation module converts the function names and variable names in the SAS code into the corresponding function names and variable names in the Hive platform, generates the target Hive code by combining the Hive SQL code with the function names and variable names of the Hive platform, and converts and generates the target Z+ platform code based on the target Hive code.
9. A SAS code automatic conversion device, characterized in that: The SAS code automatic conversion device comprises a memory, a processor, and a SAS code automatic conversion program stored in the memory and executable on the processor. When the SAS code automatic conversion program is executed by the processor, the steps of the SAS code automatic conversion method according to any one of claims 1 to 7 are implemented.
10. A computer storage medium, characterized in that: The storage medium stores a SAS code automatic conversion program, and when the SAS code automatic conversion program is executed by the processor, the steps of the SAS code automatic conversion method according to any one of claims 1 to 7 are implemented.
