Automatic script generation method and device based on metadata, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses an automatic script generation method and device based on metadata, equipment and a medium. Analyzing the production library table metadata by using a preset template to obtain a plurality of data files; splicing the data files by utilizing a preset rule to obtain script core data; and assembling the script core data by utilizing a preset configuration file to obtain an automatic script. Meanwhile, based on a batch automation tool, an efficient and reliable workflow is created, and automation of DDL, initialization, deployment, number checking and other development work is achieved; by using the preset client to access the production library table metadata, the security and performance can be significantly improved, the application development and maintenance are simplified, and the risk to the production environment is reduced; the automatic tool can effectively compare an actual verification result with an expected result, and generates a detailed verification report to find problems in time.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a metadata-based automated script generation method, device, equipment and medium. Background Art

[0002] A big data warehouse is a system used to store and analyze massive amounts of data and provide functions such as data visualization, machine learning, and data mining. It usually uses distributed storage and processing technology to cope with rapidly growing and diverse data needs, helping companies better understand the value of data and make more effective decisions. It is used by companies to store and process analytical data, and combined with intelligent BI reporting tools, it plays a significant role in the company's senior decision-making, customer management, and daily operations. Big data warehouses play a vital role in financial technology. They integrate and store massive amounts of financial data from various sources, and through analysis and mining, they provide strong data support for applications such as precision marketing, risk control, anti-fraud, and personalized financial services, thereby promoting the innovation and development of financial technology.

[0003] As the business of an enterprise grows, the database tables and processing links of the entire data warehouse become increasingly numerous and complex. However, in the process of enterprise data warehouse development, based on its own data governance data warehouse upgrade, or to meet regulatory and national sensitive data requirements, the transformation of the big data warehouse has become a necessary daily optimization and transformation matter.

[0004] In actual use, based on national security, the transformation project includes multiple databases, each database has multiple reports that need to be processed, and a lot of manpower is needed to analyze, design, develop, and verify each table or task. In these processes, development personnel repeatedly access the library table metadata, write code, test code, put the code online, verify data, etc.; the speed is slow, prone to errors, inefficient, and costly; the entire processing process is very time-consuming and difficult to cope with the rapid growth and diversified needs of massive data. In addition, human operations are prone to omissions, resulting in data errors or omissions, which affect the accuracy of the final analysis results. Summary of the invention

[0005] The present invention provides a metadata-based automatic script generation method, device, computer equipment and medium to solve the technical problems of inaccurate matching of semantic matching models, serious lack of matching ability, and subsequent poor question-answering effects.

[0006] In a first aspect, a metadata-based automated script generation method is provided, comprising:

[0007] Obtain production database table metadata by accessing the client;

[0008] Parsing the production library table metadata using a preset template to obtain multiple data files;

[0009] The data files are spliced ​​using preset rules to obtain script core data;

[0010] The script core data is assembled using a preset configuration file to obtain an automated script.

[0011] In a second aspect, a metadata-based automatic script generation device is provided, comprising:

[0012] The acquisition module is used to obtain the metadata of production database tables by accessing the client;

[0013] A parsing module, used to parse the production library table metadata using a preset template to obtain multiple data files;

[0014] A splicing module, used to splice the data files using preset rules to obtain script core data;

[0015] The assembly module assembles the script core data using a preset configuration file to obtain an automated script.

[0016] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned metadata-based automated script generation method when executing the computer program.

[0017] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned metadata-based automated script generation method are implemented.

[0018] In the scheme implemented by the above-mentioned metadata-based automated script generation method, device, computer equipment and storage medium, batch automation tools can be used to realize automated operation, access and obtain production data, thereby outputting script files; at the same time, based on batch automation tools, an efficient and reliable workflow is created to realize the automation of development work such as DDL, initialization, deployment, and core count; using a preset client to access the production library table metadata can significantly improve security and performance, simplify application development and maintenance, and reduce risks to the production environment; obtaining a data parsing template and using it to parse the production library table metadata can present the information originally hidden in the database in a structured and readable manner, generate a data file in the target format, facilitate data analysis and processing, and eliminate data format differences between different systems, simplifying the data exchange process; automated scripts can be used to realize the automation of development work such as DDL, initialization, deployment, and core count, and for the existing development processes that require a large amount of manpower to repeatedly complete the transformation one by one, a large amount of manpower investment can be saved in these development processes; through the input-to-output comparison method, the automated tool can effectively compare the actual verification results with the expected results, and generate a detailed verification report to quickly understand the data verification results and find problems in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 It is a schematic diagram of an application environment of a metadata-based automatic script generation method in an embodiment of the present invention;

[0021] Figure 2 It is a flowchart of a method for automatically generating scripts based on metadata in one embodiment of the present invention;

[0022] Figure 3 It is a first structural diagram of a metadata-based automatic script generation method in one embodiment of the present invention;

[0023] Figure 4 is a second structural diagram of a metadata-based automatic script generation method in an embodiment of the present invention;

[0024] Figure 5 is a schematic structural diagram of an automatic script generation device based on metadata in one embodiment of the present invention;

[0025] Figure 6is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0026] Figure 7 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The metadata-based automatic script generation method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, Figure 1 It is a schematic diagram of an application environment of an automated script generation method based on metadata in an embodiment of the present invention; wherein, the client communicates with the server through a network. The server can use client data; an intelligent automation engine implemented by a server, this engine can obtain table metadata in the database server in real time; in the embodiment of the present invention, this specific processing method is applied to the core number automation work of the data warehouse script, and thereby realizes the automatic unified processing of the library tables that need to be transformed; through this automated processing, the accuracy of data processing can be significantly improved, and omissions that may be caused by manual operations can be prevented. At the same time, the efficiency of data processing can be greatly improved, and the labor cost in the data processing process can be reduced. Specifically, the method parses the data warehouse script, identifies the library tables that need to be transformed, and automatically generates corresponding code snippets, thereby realizing the automatic transformation of these library tables. This method can not only effectively improve the efficiency and accuracy of data processing, but also reduce the workload of data processing personnel, improve the overall quality of data processing, and ultimately provide effective technical support for the data core number work of national secret projects.

[0029] See also Figure 2 As shown, Figure 2 The flowchart of a method for automatically generating a script based on metadata in an embodiment of the present invention includes the following steps:

[0030] S1. Obtain production database table metadata by accessing the preset client.

[0031] The metadata-based automated script generation method provided by the present invention can be applied to intelligent automated engines such as staff or assistants that perform batch processing in various application scenarios. The intelligent automated engine is usually implemented through a server, which can obtain table metadata in the database server in real time. In an embodiment of the present invention, a specific automated processing method is applied to data warehouse scripts and core data automation work to automatically and uniformly process a batch of library tables that need to be modified, so as to improve the accuracy of data processing, prevent omissions, and at the same time improve the efficiency of data processing and reduce the labor cost of data processing.

[0032] In an embodiment of the present invention, the client is a provider of data tables, provides the data tables required by this embodiment, and is responsible for organizing various raw data, such as user behavior data, product information data, etc., into a structured table form in a standardized format; the access refers to connecting to the production database server through a pre-configured application or client, and using a database query language or API call to read table metadata information such as table structure, column name, data type, etc. from the database; the production library table metadata refers to a collection of information that describes the structure and content of the database table in the production environment; this information includes table name, field name, field type, field length, primary key, foreign key, index, etc., which clearly shows the structure and content of the data table, making it convenient for automated processing tools to understand and use this data; the acquisition of production library table metadata by accessing the preset client refers to using a pre-configured client program to connect to the production database server, and using a database query language to retrieve information such as table structure, column name, data type, and finally extracting these metadata information for subsequent automated processing or analysis.

[0033] In the embodiment of the present invention, the step of obtaining production library table metadata by accessing a preset client includes:

[0034] Establish a secure connection between the client and the preset database server;

[0035] Sending a request to obtain table metadata information to the database server through the secure connection;

[0036] Using the client to receive the table metadata information returned by the database server according to the acquisition request;

[0037] The client is used to parse the table metadata information into a preset format to obtain production library table metadata.

[0038] In the embodiment of the present invention, a secure connection is established between a client and a preset database server, and multiple security measures need to be taken, such as using an encryption protocol to protect the data transmission process, using an identity authentication mechanism to verify the identity of the client, and configuring an access control policy to limit the access rights of the client to ensure that only authorized users and applications can access the database; the database server is a computer system specifically used to store, manage and retrieve data. It provides interfaces and tools to enable applications to access and operate data in the database and ensure the security, integrity and consistency of the data; the secure connection refers to a network connection that establishes a secure channel in the network world. This channel must not only ensure the rapid transmission of data, but also ensure the security and integrity of data during transmission; the secure channel refers to the client and server using security mechanisms such as encryption protocols and identity authentication to ensure that data is securely protected during transmission to prevent malicious theft or tampering; the client sends a request to the database server to obtain table metadata through a secure connection. The request is similar to a letter, which contains information requesting to obtain a specific data table. The server After receiving the client request, the server establishes specific table metadata according to the information of the specific data table. The request includes necessary information such as the target table name and the database name to clearly indicate the metadata that the server needs to obtain; the request for obtaining the table metadata information sent to the database server through the secure connection needs to follow certain steps and specifications, and uses a pre-defined database connection interface to connect to the target database server. After the connection is successful, the client constructs a query statement, which is used to extract the required table metadata information from the database information architecture and request to return related fields, such as column name, data type, length, constraint conditions, etc.; the use of the client to receive the table metadata information returned by the database server according to the acquisition request means that after receiving the request, the server executes the query and returns a result set, which includes various fields of the table metadata and their corresponding values. The client receives the result set returned by the server and parses it. The client parses the result set and extracts the table metadata information of each field. The extracted table metadata information is usually organized into a structured data structure, such as a data table or JSON (JavaScript Object Notation) format, which is convenient for subsequent application use;The use of the client to parse the table metadata information into a preset format to obtain the production library table metadata refers to the table metadata information obtained by the client from the database server, which is usually presented in the original database format, such as in the form of a table or JSON format. In order to facilitate subsequent use and analysis, the client needs to parse these original data into a preset format to obtain the production library table metadata. The parsing process involves mapping each field in the original data to the corresponding field in the preset format. In addition, the parsing process also requires format conversion of some data. After the parsing is completed, the client will obtain a structured data structure, which contains all the information of the production library table metadata, such as table name, column name, data type, length, constraint conditions, etc.; this structured data structure can be set in various formats as needed, such as data table, JSON object, XML (Extensible Markup Language) document, etc., depending on the preset format requirements. ;

[0039] In detail, in order to obtain the metadata of the production library table safely and efficiently, it is first necessary to establish a secure connection between the client and the preset database server to ensure the confidentiality and integrity of data transmission. In this embodiment, SSL (Secure Sockets Layer) or TLS (Transport Layer Security) can be used to prevent data from being stolen or tampered with during transmission. After establishing the secure connection, the client can send a request to obtain table metadata information to the database server through the connection. After receiving the request, the database server will query the corresponding metadata information according to the request content and return it to the client in a structured format, such as JSON. After receiving the metadata information returned by the server, the client will use a pre-defined parser to parse the data into a preset format, for example, the metadata in JSON format can be converted into a Python dictionary or other data structure to obtain the metadata of the production library table. This method not only ensures the security of data transmission, but also simplifies the process of data acquisition and parsing, thereby improving the overall efficiency.

[0040] In the embodiment of the present invention, using a preset client to access production library table metadata can significantly improve security and performance, simplify application development and maintenance, and reduce risks to the production environment.

[0041] S2. Obtain a data parsing template, and use the data parsing template to parse the production library table metadata to obtain multiple data files.

[0042] In the embodiment of the present invention, please refer to Figure 3 As shown, Figure 3It is a first structural diagram of an automatic script generation method based on metadata in an embodiment of the present invention: the data parsing template includes rules and formats for parsing production library table metadata, and can accurately identify and interpret various information in the metadata; the acquisition of the data parsing template requires storing the required template statements in a template file. In this embodiment, a DDL statement template, an HQL script template, an initialization SHELL script template, and a deployment command document template can be used, and the client connection information, table metadata, data path, project information, and Hive connection information are stored in a data source file; the template file and the data source file are data-fused to obtain a data parsing template; then the data parsing template is obtained using a toolbox of simulation functions. In this embodiment, the Bash command of Python software can be used to read the file where the template is located to complete the acquisition; parsing refers to the process of analyzing, understanding and interpreting data or code, with the purpose of converting raw data or code into an understandable and operable form, such as converting text into semantic information, or converting code into executable instructions, thereby providing a basis for subsequent processing.

[0043] In the embodiment of the present invention, the production library table metadata is parsed using the data parsing template to obtain multiple data files, including:

[0044] Parse the production database table metadata according to the definition rules in the data parsing template to obtain the target format data;

[0045] A corresponding blank data file is created according to the target format data, and the target format data is filled into the blank data file to obtain a plurality of data files.

[0046] In an embodiment of the present invention, the definition rules may include data field mapping, data type conversion, formatting, cleaning, etc.; the process of parsing the production library table metadata according to the definition rules in the data parsing template to obtain the target format data is a process of converting structured original data into target format data, similar to translating one language into another. The data parsing template plays the role of a dictionary, defines the mapping relationship between the fields of the original data and the target data and the data format conversion rules. The parsing process first needs to find the corresponding fields in the original data according to the rules defined in the template, and convert the data types in the original data into the data types in the target data according to the data type conversion rules in the template; the blank data file refers to a file pre-created according to the target data format, but the content is empty, which is used to fill the parsed data in the subsequent generation of the final data file; the process of creating a corresponding blank data file according to the target format data and filling the target format data into the blank data file to obtain multiple data files is a process of writing data from the memory to the disk. A blank data file is created according to the structure and data type in the target format data, such as creating a CSV (Command File). SeparatedValues, comma separated values) file, and set the file header information, including field name and data type, etc., fill the content in the target format data into the blank data file line by line, determine the value of each field according to the structure in the target format data and write it to the corresponding column, and format the data according to the data type in the target format data. The filling process needs to be verified according to the structure and data type of the target format data to ensure that the data format written into the data file is correct and meets the expected format requirements, thereby ensuring the reliability and correctness of the entire operation.

[0047] In detail, according to the rules defined in the data parsing template, such as field mapping, data type conversion, data cleaning, etc., parse the metadata of the production library table and convert it into data in the target format. According to the target data format, create a blank data file; this blank data file contains the basic structure of the target data format, but the content is empty; fill the parsed target format data into this blank data file, and finally generate a data file. This process can realize data migration and data analysis, and store or use it in a specific format.

[0048] In the embodiment of the present invention, the step of parsing the production library table metadata according to the definition rules in the data parsing template to obtain the target format data includes:

[0049] Parse the metadata of the production library table through the preset function to obtain the initial parsed data;

[0050] Derivation of the initial parsing data is performed through a preset list to obtain derived parsing data;

[0051] The derived and parsed data is converted into a target data format to obtain target format data.

[0052] In the embodiment of the present invention, the preset function can be implemented by response.json() in the Python program; the parsing of the production library table metadata by the preset function to obtain the initial parsed data is a process of converting the original data into a processable format, which is similar to decrypting a letter written in code into readable text. The preset function is like a decryption key, which contains the rules and methods for parsing the original data. The parsing process first needs to pass the production library table metadata as an input parameter to the preset function, and then the function will parse according to the preset rules. For example, the function will use the preset rules to identify the field names in the original data and map the field names to the field names in the target data structure. The function will convert the data types in the original data according to the preset rules. The function will assemble the parsed data into a structured data structure by replacing the data type in the target data structure; the preset list can use a Python dictionary or list, and the conversion includes data type conversion and format conversion; the deduction of the initial parsed data through the preset list to obtain the derived parsed data is a process of reasoning and expansion based on known information. The deduction process first needs to take the initial parsed data as input, and then deduce according to the rules in the preset list. The preset list may define that the value of a field is converted according to the values ​​of other fields. In this case, the deduction process will convert the value of the field according to the values ​​of other fields in the initial parsed data. The deduction process will also create new fields and assign values ​​according to the definition of the newly added fields in the preset list.

[0053] In detail, this step can be implemented by Python program. The production library table metadata to be parsed is specified by the parameter table_name, and the parameter table_name is passed to api_url. The HTTP GET request is constructed by using the requests library, and the table_name parameter is embedded in the request URL. The request will be sent to api_url and the server response will be obtained. The received response content will be parsed into a Python dictionary or list by the response.json() method. This step can adopt JSON format and use Python's list derivation to filter out the target table metadata in the returned JSON data, retaining only the required data to obtain the derived parsed data; according to the predefined target format, the extracted derived parsed data is converted to obtain the target format data; the information in the database can be converted into data in the target format, which makes the data easier to read and process, thereby improving data utilization, simplifying data integration and analysis processes, and ultimately achieving more convenient data management and application.

[0054] In an embodiment of the present invention, a data parsing template is obtained and used to parse the production library table metadata, so that the information originally hidden in the database can be presented in a structured and readable manner, and a data file in a target format can be generated, which can facilitate data analysis and processing, and at the same time can eliminate data format differences between different systems and simplify the data exchange process.

[0055] S3, splicing the data files to obtain script core data;

[0056] In the embodiment of the present invention, the splicing refers to combining the various parts of the data file in a specified order and manner according to predefined rules to finally form a complete data set; the preset rule is a data processing logic that specifies the sorting in the splicing process. In actual use, the preset rule can be set according to the required sorting or position relationship.

[0057] In the embodiment of the present invention, the data files are spliced ​​to obtain script core data, including:

[0058] Convert the data file into a consistent data type to obtain a formatted data file;

[0059] Sorting the format data files according to preset rules to obtain standard data files;

[0060] The standard data files are spliced ​​to obtain script core data.

[0061] In an embodiment of the present invention, the consistent data type can be converted using the json.load() function. The conversion of the data file into the consistent data type to obtain the format data file is a process of unifying the data types in the data file. It is necessary to determine the data type that should be used for each field in the data file according to the preset format requirements, perform type conversion on the data in the data file according to the data type in the data file and the preset format requirements, write the converted data into a new data file, and format the data according to the preset format requirements; the splicing can use the concat() function to splice the standard data files together, sort the format data files according to the preset rules to obtain the standard data files, sort the formatted data files according to the predefined sorting rules to obtain the standard data files arranged in a specific order; this sorting rule may be based on one or more fields, such as time, ID or name, and ascending or descending order; then, these sorted standard data files are spliced ​​together to form a complete data set containing all data, that is, the script core data.

[0062] In detail, this step can be implemented using Python programs. By using the json.load() function to read the data file, the read data file is analyzed, necessary information is extracted, all the extracted information is combined into a unified dictionary list, and the dictionary list is converted into a DataFrame using the read_csv() function of the pandas library; the DataFrame is sorted using the sort_values() function of the pandas library, and the sorting order corresponds to the DDL statement template, HQL script template, initialization SHELL script template, and deployment command document template; finally, the concat() function of the pandas library is used to splice multiple DataFrames together to obtain the core data of the script; converting data files to consistent data types and sorting and splicing them can significantly improve the quality, readability, and analysis efficiency of the data. These advantages can help users process data more efficiently and obtain more valuable analysis results.

[0063] In the embodiment of the present invention, the data files are spliced ​​to obtain script core data; this can facilitate subsequent script processing and analysis, improve data utilization, and reduce potential error risks, ultimately improving the reliability and accuracy of the script.

[0064] S4. Assemble the script core data using a preset configuration file to obtain an automated script.

[0065] In the embodiment of the present invention, please refer to Figure 4 As shown, Figure 4 It is a second structural diagram of the metadata-based automated script generation method in one embodiment of the present invention; the preset configuration file can be a JSON file; the preset configuration file defines various information required for the data processing task, including the path of the data source, the data format, the type conversion rules of the configuration column, the column name and sorting order for sorting data, the output file path and name of the processing result, the DDL statement template, the HQL script template, the initialization SHELL script template and the deployment command document template.

[0066] In the embodiment of the present invention, the use of a preset configuration file to assemble the script core data to obtain an automated script includes:

[0067] Get a pre-built configuration file;

[0068] Using simulation software to load the configuration file to obtain configuration information;

[0069] The assembly script core data is assembled into an automation script according to the configuration information.

[0070] In an embodiment of the present invention, the configuration file includes various parameters and configuration information required for the automation script, such as key information such as data source path, processing steps, output format, parameter values, etc. The simulation software can use json.load in the Python program; the simulation software plays the role of reading and parsing the configuration file. It can read the configuration file and convert its content into structured data that can be used by the program. The automation script includes a DDL statement HQL script, an initialization SHELL script, a deployment command document, and a data verification HQL script; assembling the assembly script core data into an automation script according to the configuration information means that the program assembles the pre-assembled script core data in a specific way using the configuration information, and assembles the script core data according to the preset rules and logic according to the configuration information to finally generate a complete automation script.

[0071] In detail, the json.load in the Python program is used to read the configuration file, and the read configuration file content is converted into a format that can be used by the script. According to the configuration information, the core data of the assembly script is assembled into an automation script; the automation script is used to realize automated operations, access and obtain production data; generate DDL statement HQL scripts, initialization SHELL scripts, deployment command documents and data verification HQL scripts; the automation script can be used to realize the automation of development work such as DDL, initialization, deployment, and core count; among them, the DDL statement HQL script can automatically generate commands for creating, modifying or deleting database table structures according to preset rules and configurations. The HQL statement of the DDL statement is constructed and adjusted according to actual needs to ensure that the generated statement can be executed accurately, thereby realizing the automatic management of the database table structure; the advantage of the HQL script of the DDL statement is that it can greatly improve the efficiency of database table structure management, reduce the risk of errors caused by manual operations, and improve the readability and maintainability of the code. At the same time, it can be flexibly configured according to actual needs to meet the automation needs of different scenarios; the initialization SHELL script can automatically perform a series of initialization operations according to preset rules and configurations, such as creating directories, configuring environment variables, installing dependent software, etc., thereby providing the necessary basic environment for the operation of applications or systems and ensuring the initial The advantage of the initialization SHELL script is that it can simplify the system initialization process, improve the efficiency and reliability of system deployment, and can be flexibly configured according to actual needs to meet the initialization needs of different scenarios; the deployment command document records the deployment process of the system or application, including the required commands, configuration parameters, and operation steps, and presents it in the form of executable scripts or documents, which is convenient for users to complete the deployment operation quickly and accurately, ensuring the standardization and consistency of the deployment process; the advantage of the deployment command document is that it can improve the deployment efficiency, ensure the repeatability and traceability of the deployment process, and can easily manage and update versions to adapt to the system The data validation HQL script can automatically generate HQL statements for checking and validating data according to preset rules and configurations, and execute these statements to detect whether there are errors or anomalies in the data, such as data type errors, data missing, data duplication, etc., so as to ensure the reliability of data quality; the advantage of the data validation HQL script is that it can greatly improve the efficiency of data validation and the coverage of data validation, and it can be flexibly configured according to actual needs to meet the data validation needs of different scenarios; for the existing development processes that require a large amount of manpower to repeatedly complete the transformation one by one, it can save a lot of manpower investment in these development processes.

[0072] In the embodiment of the present invention, after assembling the script core data using the preset configuration file to obtain the automation script, the method further includes:

[0073] Convert the automation script according to a preset script format to obtain a formatted script file;

[0074] Create a script expected results file;

[0075] The formatted script file is compared with the expected result file to obtain the verification result.

[0076] In an embodiment of the present invention, the automation script refers to the obtained data verification HQL script; the converting of the automation script according to a preset script format to obtain a formatted script file refers to converting the generated automation script into a file conforming to a specific format; the establishing of a script expected result file refers to pre-defining the expected result after the execution of the automation script, such as the expected output file content, the format of the output file, the value of a specific variable, a program execution completion flag, and the return value of a key operation step, etc. The expected result file is usually stored in a text or table format, contains a detailed description of the expected results, and covers as many possible execution paths and potential abnormal situations as possible. These expected results may be based on previous test data, document descriptions or manual evaluations, reflecting the exact results that the script should achieve or produce; the comparing of the formatted script file according to the expected result file to obtain a verification result refers to comparing the output result generated by the actually executed formatted script file with the pre-established expected result file to determine whether the script is executed as expected and to evaluate the accuracy of the script.

[0077] The preset script format refers to defining a preset script format. This step adopts the JSON format for storing script information; the expected result file contains the expected data verification result.

[0078] In detail, define a JSON script format, use Python's json module to convert the automation script into JSON format, and generate a formatted script file; create an expected result file, which contains the expected data quantity and parameter indicators of the data verification result, and compare the data type, data value, and data structure between the formatted script file generated by the automation script and the expected result file, and obtain the verification result by comparing whether the two comparison items are the same; the obtained data verification HQL script needs to repeatedly visit the production for the next step of data verification and result collection, which requires a certain amount of manpower; through the input-to-output comparison method, the automation tool can effectively compare the actual verification results with the expected results, and generate a detailed verification report to quickly understand the data verification results and find problems in time.

[0079] In one embodiment, a metadata-based automated script generation device is provided, and the metadata-based automated script generation device corresponds one-to-one to the metadata-based automated script generation method in the above embodiment. Figure 5 As shown, Figure 5 1 is a schematic diagram of a structure of an automatic script generation device based on metadata in one embodiment of the present invention; the automatic script generation device based on metadata includes an acquisition module 101, a parsing module 102, a splicing module 103 and an assembling module 104. The functional modules are described in detail as follows:

[0080] The acquisition module 101 is used to obtain production library table metadata by accessing the client;

[0081] The parsing module 102 is used to parse the production library table metadata using a preset template to obtain multiple data files;

[0082] A splicing module 103 is used to splice the data files using preset rules to obtain script core data;

[0083] The assembly module 104 assembles the script core data using a preset configuration file to obtain an automated script.

[0084] In one embodiment, the acquisition module 101, when acquiring production library table metadata by accessing the client, is used to:

[0085] Establish a secure connection between the client and the preset database server;

[0086] Sending a request to obtain table metadata information to the database server through the secure connection;

[0087] Using the client to receive the table metadata information returned by the database server according to the acquisition request;

[0088] The client is used to parse the table metadata information into a preset format to obtain production library table metadata.

[0089] In one embodiment, the parsing module 102, when parsing the production library table metadata using a preset template to obtain multiple data files, is used to:

[0090] Parse the production database table metadata according to the definition rules in the data parsing template to obtain the target format data;

[0091] Create a corresponding blank data file according to the target format data, and fill the target format data into the blank data file to obtain multiple data files;

[0092] In one embodiment, the parsing module 102, when parsing the production library table metadata according to the definition rules in the data parsing template to obtain the target format data, is used to:

[0093] Parse the metadata of the production library table through the preset function to obtain the initial parsed data;

[0094] Derivation of the initial parsing data is performed through a preset list to obtain derived parsing data;

[0095] The derived and parsed data is converted into a target data format to obtain target format data.

[0096] In one embodiment, the splicing module 103, when executing the splicing of the data files using the preset rules to obtain the script core data, is used to:

[0097] Convert the data file into a consistent data type to obtain a formatted data file;

[0098] Sorting the format data files according to preset rules to obtain standard data files;

[0099] The standard data files are spliced ​​to obtain the script core data

[0100] In one embodiment, the assembling module 104, when assembling the script core data using a preset configuration file to obtain an automated script, is used to:

[0101] Get a pre-built configuration file;

[0102] Using simulation software to load the configuration file to obtain configuration information;

[0103] The assembly script core data is assembled into an automation script according to the configuration information.

[0104] In one embodiment, the assembling module 104, when assembling the script core data using a preset configuration file to obtain an automated script, is further configured to:

[0105] Convert the automation script according to a preset script format to obtain a formatted script file;

[0106] Create a script expected results file;

[0107] The formatted script file is compared with the expected result file to obtain the verification result.

[0108] For the specific definition of the metadata-based automated script generation device, please refer to the definition of the metadata-based automated script generation method above, which will not be repeated here. Each module in the above-mentioned metadata-based automated script generation device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0109] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, Figure 6 1 is a schematic diagram of the structure of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a metadata-based automated script generation method are implemented.

[0110] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, Figure 7 1 is another structural diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a method for automatically generating scripts based on metadata are implemented.

[0111] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0112] Receive user questions, and perform question recall processing from a preset standard question library according to the user questions to obtain multiple candidate questions;

[0113] Sorting the plurality of candidate questions according to the semantic matching degree between the user question and the candidate question to obtain a first sorting result of the candidate questions;

[0114] Determine entity alignment between the user question and each of the candidate questions respectively;

[0115] According to the entity alignment, the first sorting result is adjusted to obtain a second sorting result of the candidate questions, in which the candidate questions without entity alignment are sorted behind the candidate questions with entity alignment, and the sorting of the candidate questions with entity alignment according to the first sorting result remains unchanged;

[0116] In the sequence of the second sorting results, a preset number of candidate questions are sequentially intercepted from the first ones as matching questions;

[0117] A matching answer corresponding to the matching question is selected to respond to the user question.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0119] Receive user questions, and perform question recall processing from a preset standard question library according to the user questions to obtain multiple candidate questions;

[0120] Sorting the plurality of candidate questions according to the semantic matching degree between the user question and the candidate question to obtain a first sorting result of the candidate questions;

[0121] Determine entity alignment between the user question and each of the candidate questions respectively;

[0122] According to the entity alignment, the first sorting result is adjusted to obtain a second sorting result of the candidate questions, in which the candidate questions without entity alignment are sorted behind the candidate questions with entity alignment, and the sorting of the candidate questions with entity alignment according to the first sorting result remains unchanged;

[0123] In the sequence of the second sorting results, a preset number of candidate questions are sequentially intercepted from the first ones as matching questions;

[0124] A matching answer corresponding to the matching question is selected to respond to the user question.

[0125] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0127] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0128] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A metadata-based automated script generation method, characterized in that: include: Obtain production database table metadata by accessing the client; Parsing the production library table metadata using a preset template to obtain multiple data files; Splicing the data files to obtain script core data; The script core data is assembled using a preset configuration file to obtain an automated script.

2. The metadata-based automated script generation method according to claim 1, characterized in that: The step of obtaining the production library table metadata by accessing the client includes: Establish a secure connection between the client and the preset database server; Sending a request to obtain table metadata information to the database server through the secure connection; Using the client to receive the table metadata information returned by the database server according to the acquisition request; The client is used to parse the table metadata information into a preset format to obtain production library table metadata.

3. The metadata-based automated script generation method according to claim 1, characterized in that: The production library table metadata is parsed using the data parsing template to obtain multiple data files, including: Parse the production database table metadata according to the definition rules in the data parsing template to obtain the target format data; A corresponding blank data file is created according to the target format data, and the target format data is filled into the blank data file to obtain a plurality of data files.

4. The metadata-based automated script generation method according to claim 3, characterized in that: The step of parsing the production library table metadata according to the definition rules in the data parsing template to obtain the target format data includes: Parse the metadata of the production library table through the preset function to obtain the initial parsed data; Derivation of the initial parsing data is performed through a preset list to obtain derived parsing data; The derived and parsed data is converted into a target data format to obtain target format data.

5. The metadata-based automated script generation method according to claim 1, characterized in that: The data files are spliced ​​to obtain script core data, including: Convert the data file into a consistent data type to obtain a formatted data file; Sorting the format data files according to preset rules to obtain standard data files; The standard data files are spliced ​​to obtain script core data.

6. The metadata-based automated script generation method according to claim 1, characterized in that: The method of assembling the script core data by using the preset configuration file to obtain the automation script includes: Get a pre-built configuration file; Using simulation software to load the configuration file to obtain configuration information; The assembly script core data is assembled into an automation script according to the configuration information.

7. The metadata-based automated script generation method according to claim 1, characterized in that: After assembling the script core data by using the preset configuration file to obtain the automation script, the method further includes: Convert the automation script according to a preset script format to obtain a formatted script file; Create a script expected results file; The formatted script file is compared with the expected result file to obtain the verification result.

8. An automatic script generation device based on metadata, characterized in that: include: The acquisition module is used to obtain the metadata of production database tables by accessing the client; A parsing module, used to parse the production library table metadata using a preset template to obtain multiple data files; A splicing module, used to splice the data files using preset rules to obtain script core data; The assembly module assembles the script core data using a preset configuration file to obtain an automated script.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the metadata-based automatic script generation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the metadata-based automatic script generation method according to any one of claims 1 to 7 is implemented.

Images