Structured query statement extraction method, device, equipment, storage medium and program product

Abstract

A structured query statement extraction method, device, equipment, storage medium and program product, the method comprising: constructing a virtual environment for running a Python script according to a Python library dictionary and an environment variable dictionary of the Python script to be processed; reconstructing an original syntax tree corresponding to the Python script according to the Python library dictionary and a conditional component list of the Python script, the reconstructed syntax tree comprising a variable assignment node and an SQL catcher node; traversing each parameter value of each variable assignment node of the reconstructed syntax tree to obtain multiple value combinations, converting a syntax tree formed by each value combination into a new Python script, running the new Python script in the virtual environment, and grouping SQL codes formed by each running result to form a structured query statement extraction result of the Python script to be processed. In this way, the generalization, security and integrity of SQL extraction can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method, apparatus, device, storage medium, and program product for extracting structured query statements. Background Technology

[0002] In data development and operations, a combination of Python (a high-level, general-purpose, interpreted programming language) and Structured Query Language (SQL) code is often used. However, when building and managing data lineages, the core goal is to answer the questions: "Where did this data come from? What processing did it undergo? Where did it go?" A large amount of critical data transformation logic (including SQL) is hidden in Python scripts, resulting in an incomplete data lineage diagram. It's like a "static map" containing only database table and field names, but without showing how the data is actually processed. Therefore, to fill this largest gap in the modern data stack—the lineage blind spot—and upgrade the data lineage from a static map containing only "database objects" to a dynamic and complete panoramic operational map covering "data processing logic" (code), extracting SQL from Python scripts becomes a crucial step.

[0003] Currently, the main methods for extracting SQL statements from Python scripts include static analysis and dynamic analysis. However, static analysis has poor generalization and some SQL fragments may be missed; while dynamic analysis requires additional performance overhead and poses security risks, making it unsuitable for scenarios with high security requirements.

[0004] Therefore, improving the generalizability, security, and completeness of SQL statement extraction has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for extracting structured query statements, which can improve the generalizability, security, and completeness of SQL statement extraction.

[0006] Firstly, this application provides a method for extracting structured query statements, the method comprising:

[0007] Based on the original syntax tree corresponding to the Python script to be processed, the Python library dictionary, environment variable dictionary, and condition component list of the Python script are determined. The Python library dictionary is used to represent the mapping relationship between the Python libraries used by the Python script and the aliases of the Python libraries. The environment variable dictionary is used to represent the mapping relationship between the environment variables used by the Python script and the names of the environment variables.

[0008] Based on the Python library dictionary and environment variable dictionary, construct a virtual environment for running Python scripts;

[0009] The original syntax tree is reconstructed based on the Python library dictionary and the list of conditional components to obtain the reconstructed syntax tree. The reconstructed syntax tree includes variable assignment nodes and structured query catcher nodes. The structured query catcher nodes are used to indicate the output structured query code.

[0010] Traverse each variable assignment node of the reconstructed syntax tree and obtain the value of each parameter, resulting in multiple combinations of values. Convert the syntax tree formed by each combination of values ​​into a new Python script, run it in a virtual environment, and obtain the running result under each combination of values.

[0011] The structured query statement code formed by each execution result is used to form the structured query statement extraction result of the Python script to be processed.

[0012] In one embodiment, a virtual environment for running Python scripts is constructed based on a Python library dictionary and an environment variable dictionary, including: determining at least one database connector corresponding to the Python script based on the Python library dictionary; determining the remaining dependent libraries in the Python library dictionary besides the at least one database connector; determining at least one environment variable corresponding to the Python script based on the environment variable dictionary, and updating the virtual environment variable corresponding to each environment variable to a virtual value; the virtual value is associated with the meaning of the corresponding environment variable; and constructing the virtual environment for running Python scripts based on the remaining dependent libraries, at least one virtual value, and a pre-built structured query capture tool.

[0013] In one embodiment, the original syntax tree is reconstructed based on the Python library dictionary and the conditional component list to obtain a reconstructed syntax tree. This includes: determining at least one database connector corresponding to the Python script based on the Python library dictionary, and updating the node corresponding to each database connector in the at least one database connector to a structured query statement catcher node; constructing a conditional branch list corresponding to the Python script based on the conditional component list, and inserting a variable assignment node at the position corresponding to each branch component included in the conditional branch list; the variable assignment node includes a preset parameter; and reconstructing the original syntax tree based on the conditional branches after inserting multiple variable assignment nodes and multiple structured query statement catcher nodes to obtain a reconstructed syntax tree.

[0014] In one embodiment, constructing a conditional branch list corresponding to the Python script based on the conditional component list includes: for each conditional component included in the conditional component list, constructing a branch component corresponding to the conditional component based on the parameters, values, logic, and position of the conditional component; and obtaining a conditional branch list corresponding to the Python script based on the branch component corresponding to each conditional component.

[0015] In one embodiment, a branch component corresponding to a condition component is constructed based on the parameters, values, logic, and position of the condition component, including: determining the value range that satisfies the values ​​and logic of the condition component; sampling within and outside the value range based on the value range, and determining a list of branch values ​​for the parameters of the condition component based on the sampling results; and generating a branch component corresponding to the condition component based on the position, parameters, and list of branch values ​​of the condition component.

[0016] In one embodiment, based on the original syntax tree corresponding to the Python script to be processed, the Python library dictionary, environment variable dictionary, and condition component list of the Python script are determined, including: determining the Python libraries, environment variables, and condition components used by the Python script based on the node type of the original syntax tree corresponding to the Python script to be processed; and determining the Python library dictionary corresponding to the Python library, the environment variable dictionary corresponding to the environment variable, and the condition component list corresponding to the condition component based on the structure of the original syntax tree.

[0017] Secondly, this application provides a structured query statement extraction apparatus, the apparatus comprising:

[0018] The determination module is used to determine the Python library dictionary, environment variable dictionary, and condition component list of the Python script based on the original syntax tree corresponding to the Python script to be processed. The Python library dictionary is used to represent the mapping relationship between the Python libraries used by the Python script and the aliases of the Python libraries, and the environment variable dictionary is used to represent the mapping relationship between the environment variables used by the Python script and the names of the environment variables.

[0019] The building block is used to construct a virtual environment for running Python scripts based on a dictionary of Python libraries and a dictionary of environment variables.

[0020] The building block is also used to reconstruct the original syntax tree based on the Python library dictionary and the list of conditional constructs, resulting in a reconstructed syntax tree. The reconstructed syntax tree includes variable assignment nodes and structured query catcher nodes. The structured query catcher nodes are used to indicate the output structured query code.

[0021] The processing module is used to traverse the variable assignment node of the reconstructed syntax tree and obtain the parameter values ​​of each parameter, resulting in multiple combinations of values. It then converts the syntax tree formed by each combination of values ​​into a new Python script, runs it in a virtual environment, and obtains the running results under each combination of values.

[0022] The processing module is also used to extract the results of the structured query statements formed by each type of execution result into a structured query statement for the Python script to be processed.

[0023] Thirdly, this application provides a computer device including a processor and a memory, the memory storing a computer program; when the processor executes the computer program, it implements the steps in the method provided in the first aspect.

[0024] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0025] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the method provided in the first aspect.

[0026] The aforementioned method, apparatus, device, storage medium, and program product for extracting structured query statements include a computer device that, based on the original syntax tree corresponding to the Python script to be processed, determines the Python library dictionary, environment variable dictionary, and condition component list of the Python script. The Python library dictionary represents the mapping relationship between the Python libraries used by the Python script and their aliases, and the environment variable dictionary represents the mapping relationship between the environment variables used by the Python script and their names. A virtual environment for running the Python script is constructed based on the Python library dictionary and environment variable dictionary. The original syntax tree is reconstructed based on the Python library dictionary and condition component list to obtain a reconstructed syntax tree. The reconstructed syntax tree includes variable assignment nodes and structured query statement capture nodes. Each parameter value of each variable assignment node in the reconstructed syntax tree is traversed to obtain multiple value combinations. The syntax tree formed by each value combination is converted into a new Python script, which is then run in the virtual environment to obtain the running results under each value combination. The structured query statement code formed by each running result is used to assemble the structured query statement extraction result of the Python script to be processed. This method, by including variable assignment nodes and structured query capture nodes in the reconstructed syntax tree, allows for the reduction of missing SQL fragments and improved SQL statement extraction by traversing each parameter value of each variable assignment node and converting the syntax tree formed by each value combination into a new Python script. Furthermore, constructing a virtual environment for running the Python script decouples the execution of structured query statements from database execution, enhancing the generalization and flexibility of SQL statement extraction. Finally, the inclusion of SQL capture nodes in the reconstructed syntax tree allows for the direct capture of SQL code through the SQL capture mechanism when running the corresponding new Python script in the virtual environment, improving the security of SQL statement extraction. In short, this method improves the generalization, security, and completeness of SQL statement extraction. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a structured query statement extraction method provided in an embodiment of this application;

[0029] Figure 2 This is a flowchart illustrating another method for extracting structured query statements provided in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of a structured query statement extraction device provided in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] In data development and operations, Python and SQL code are often used together. This makes extracting SQL from Python scripts an important step when building and managing data lineage.

[0034] Currently, methods for extracting SQL statements from Python scripts mainly include static analysis and dynamic analysis. Static methods include regular expression matching, syntax tree parsing, and Object-Relational Mapping (ORM) reverse engineering. Regular expression matching uses complex regular expression rules to match SQL code segments; this method is not only difficult to develop and maintain but also suffers from poor flexibility and generalization. Syntax tree parsing constructs a syntax tree by parsing Python syntax and then extracts SQL code as variables or parameters from key elements in the syntax tree. This method struggles to handle dynamically concatenated statements and conditional branches, and may miss branching SQL statements. ORM reverse engineering is used for specific libraries such as SQLAlchemy (a Python SQL toolkit and ORM system), parsing their SQL construction methods to reverse engineer and reconstruct the SQL code. This method only supports specific libraries and cannot support large-scale database scenarios. Dynamic methods primarily involve constructing database listeners or log capture tools to read the actual SQL code during a specific stage of database execution. While these methods can capture the real SQL at runtime, their reliance on actual script execution incurs additional performance overhead for the database. Furthermore, these methods face the security risk of leaking sensitive information such as production data. Therefore, they cannot be used in fields with stringent data security requirements, such as banking and insurance.

[0035] Based on this, embodiments of this application provide a method, apparatus, device, storage medium, and program product for extracting structured query statements. The computer device constructs a virtual environment for running the Python script based on the Python library dictionary and environment variable dictionary corresponding to the Python script to be processed. It reconstructs the original syntax tree based on the Python library dictionary and the list of conditional components corresponding to the Python script to be processed, obtaining a syntax tree (reconstructed syntax tree) including variable assignment nodes and structured query statement catcher nodes. The structured query statement catcher nodes are used to indicate the output structured query statement code. Each parameter value of each variable assignment node in the reconstructed syntax tree is traversed to obtain multiple value combinations. The syntax tree formed by each value combination is converted into a new Python script, which is then run in the virtual environment to obtain the running result under each value combination. The structured query statement code formed by each running result is used to compose the structured query statement extraction result of the Python script to be processed. This method, by including variable assignment nodes and structured query capture nodes in the reconstructed syntax tree, allows for the reduction of missing SQL fragments and improved SQL statement extraction by traversing each parameter value of each variable assignment node and converting the syntax tree formed by each value combination into a new Python script. Furthermore, constructing a virtual environment for running the Python script decouples the execution of structured query statements from database execution, enhancing the generalization and flexibility of SQL statement extraction. Finally, the inclusion of SQL capture nodes in the reconstructed syntax tree allows for the direct capture of SQL code through the SQL capture mechanism when running the corresponding new Python script in the virtual environment, improving the security of SQL statement extraction. In short, this method improves the generalization, security, and completeness of SQL statement extraction.

[0036] Optionally, the computer equipment mentioned above can be a terminal or a server. Terminals mentioned here may include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, projection devices, etc. Portable wearable devices may include smartwatches, smart bracelets, head-mounted displays, etc. Head-mounted displays may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Servers mentioned here may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services, etc., without limitation.

[0037] The structured query statement extraction method provided in the embodiments of this application will be described in detail below.

[0038] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for extracting structured query statements provided in an embodiment of this application. This method can be executed by a computer device. Figure 1 As shown, this method for extracting structured query statements may include, but is not limited to, the following steps:

[0039] S101. Based on the original syntax tree corresponding to the Python script to be processed, determine the Python library dictionary, environment variable dictionary, and condition component list of the Python script.

[0040] The Python library dictionary represents the mapping between the Python libraries used by the Python script and their aliases, while the environment variable dictionary represents the mapping between the environment variables used by the Python script and their names.

[0041] The syntax tree, also known as the abstract syntax tree (AST), is an abstract representation of the syntax structure of source code. It represents the syntax structure of a programming language in a tree-like form, with each node in the tree representing a structure in the source code.

[0042] Among them, the Python library dictionary refers to a dictionary structure containing information about installed Python packages and modules; the environment variable dictionary is a collection of key-value pairs containing operating system environment variables; and the conditional component list usually refers to a configuration list in the build tool that determines whether to include certain components, dependencies, or files based on conditions.

[0043] In an optional implementation, before step S101, the computer device may further parse the Python script to be processed to obtain the original syntax tree corresponding to the Python script. Optionally, the computer device may use a syntax parsing tool to parse the Python script to be processed to obtain the original syntax tree corresponding to the Python script. Optionally, the syntax parsing tool may be the Python standardized ast module, which provides a method for converting Python code into a syntax tree. In the implementation where the standardized ast module is used to parse the Python script to be processed, the computer device may first import the ast module, and then use the ast.parse() function (used to parse the Python source code string and convert it into an abstract syntax tree) to parse the source code string to obtain an AST object.

[0044] S102. Based on the Python library dictionary and environment variable dictionary, construct a virtual environment for running Python scripts.

[0045] A virtual environment is an isolated Python runtime environment that creates an isolated space for a Python project, containing: an independent Python interpreter, an independent package installation directory, and independent environment configuration. By using virtual environments, you can ensure that Python scripts run in the same way anywhere, avoiding problems caused by environment differences.

[0046] S103. Based on the Python library dictionary and the list of conditional components, reconstruct the original syntax tree to obtain the reconstructed syntax tree. The reconstructed syntax tree includes variable assignment nodes and structured query catcher nodes. The structured query catcher nodes are used to indicate the output structured query code.

[0047] Each variable assignment node includes a preset parameter; the preset parameter's value is an integer. Therefore, it can be understood that the reconstructed syntax tree is a syntax tree containing multiple preset parameters and SQL catcher nodes.

[0048] In one alternative implementation, the structured query catcher corresponding to the structured query catcher node can be pre-built by a computer device in the following way: mimicking the common structure of various database connectors in Python, the SQL execution function is modified to output SQL strings and virtual execution results.

[0049] Optionally, the process of building an SQL catcher using computer equipment may include, but is not limited to, the following steps:

[0050] Step 1: Construct a virtual cursor class, including functions such as execute, executemany, close, rollback, and commit.

[0051] The `execute` function performs the following functions: (1) reads the SQL request and parameters; (2) replaces the variables in the SQL request with the values ​​in the parameters; (3) writes the replaced SQL request to the SQL extraction file; and (4) returns the virtual SQL execution result. The virtual SQL execution result can be achieved through rule-based, neural network-based, or large model-based methods.

[0052] The `executemany` function reads the list of SQL requests and parameters, and calls the `execute` function sequentially according to the order of the SQL requests.

[0053] The close function closes the cursor object.

[0054] The rollback function rolls back the current transaction.

[0055] The commit function commits the current transaction.

[0056] Among them, functions such as close, rollback, and commit can be set to empty functions.

[0057] Step 2: Build the connection class, including functions such as cursor, close, rollback, and commit.

[0058] The cursor function generates a cursor class object defined in step one above.

[0059] Functions such as close, rollback, and commit can be set to empty functions.

[0060] Step 3: Build the engine class, including functions such as connect and execute.

[0061] The `connect` function generates a `connection` class object defined in step two above.

[0062] The execute function generates a cursor class object defined in step one above and executes the execute function of that object.

[0063] Step 4: Build the session class, including functions such as execute, close, rollback, and commit.

[0064] The execute function generates a cursor class object defined in step one above and executes the execute function of that object.

[0065] Functions such as close, rollback, and commit can be set to empty functions.

[0066] S104. Traverse the reconstructed syntax tree, assign each parameter value to each variable node, obtain multiple value combinations, convert the syntax tree formed by each value combination into a new Python script, run it in a virtual environment, and obtain the running results under each value combination.

[0067] S105. Take the structured query statement code formed by each running result and extract the results from the structured query statement of the Python script to be processed.

[0068] The process of steps S104 to S105 is described in detail below. Steps S104 to S105 may include, but are not limited to, the following steps:

[0069] Step 1: Initialize each parameter value of each variable assignment node in the reconstructed syntax tree to 0, obtain the value combinations, and construct the SQL code set S.

[0070] Step 2: Combine the values ​​to form a syntax tree (denoted as T(k1,k2,...,k...)). c ,...,k n Convert the script into a new Python script and run the new Python script in a virtual environment to obtain the results.

[0071] Step 3: Deduplicate the execution results to obtain the processed results, and insert the processed results into the SQL code set S.

[0072] Step 4: Adjust the parameter values ​​of each variable assignment node to obtain the value combinations, and repeat steps 2 to 4 until all parameter values ​​of each variable assignment node have been traversed to obtain the final SQL code set.

[0073] Step 5: Extract the final SQL code set as the SQL statement result of the Python script to be processed.

[0074] In this embodiment, the computer device can determine the Python library dictionary, environment variable dictionary, and condition component list of the Python script based on the original syntax tree corresponding to the Python script to be processed. The Python library dictionary is used to represent the mapping relationship between the Python libraries used by the Python script and the aliases of the Python libraries, and the environment variable dictionary is used to represent the mapping relationship between the environment variables used by the Python script and the names of the environment variables. A virtual environment for running the Python script is constructed based on the Python library dictionary and the environment variable dictionary. The original syntax tree is reconstructed based on the Python library dictionary and the condition component list to obtain a reconstructed syntax tree. The reconstructed syntax tree includes variable assignment nodes and structured query statement capture nodes. Each parameter value of each variable assignment node in the reconstructed syntax tree is traversed to obtain multiple value combinations. The syntax tree formed by each value combination is converted into a new Python script, which is run in the virtual environment to obtain the running result under each value combination. The structured query statement code formed by each running result is used to form the structured query statement extraction result of the Python script to be processed. This method, by including variable assignment nodes and structured query capture nodes in the reconstructed syntax tree, allows for the reduction of missing SQL fragments and improved SQL statement extraction by traversing each parameter value of each variable assignment node and converting the syntax tree formed by each value combination into a new Python script. Furthermore, constructing a virtual environment for running the Python script decouples the execution of structured query statements from database execution, enhancing the generalization and flexibility of SQL statement extraction. Finally, the inclusion of SQL capture nodes in the reconstructed syntax tree allows for the direct capture of SQL code through the SQL capture mechanism when running the corresponding new Python script in the virtual environment, improving the security of SQL statement extraction. In short, this method improves the generalization, security, and completeness of SQL statement extraction.

[0075] In one alternative implementation, Figure 1Step S102 in the structured query extraction method shown, which is the way the computer device constructs a virtual environment for running Python scripts based on the Python library dictionary and environment variable dictionary, can be: based on the Python library dictionary, determining at least one database connector corresponding to the Python script; determining the remaining dependent libraries in the Python library dictionary other than at least one database connector; based on the environment variable dictionary, determining at least one environment variable corresponding to the Python script, and updating the virtual environment variable corresponding to each environment variable to a virtual value; associating the virtual value with the meaning of the corresponding environment variable; and constructing a virtual environment for running Python scripts based on the remaining dependent libraries, at least one virtual value, and a pre-built structured query capture tool.

[0076] In some embodiments, the computer device determines at least one database connector corresponding to a Python script based on a Python library dictionary. This may include: determining the name and alias of at least one database connector used by the Python script based on the Python library dictionary. The name and alias of each database connector may be denoted as {m conn :alias conn}, where m conn This refers to the name of the database connector, alias. conn This refers to an alias for the database connector.

[0077] In some embodiments, when a computer device updates the virtual environment variable corresponding to each environment variable to a virtual value, the virtual value can be generated in any of the following ways: rule-based, neural network-based, large model-based, etc.

[0078] For example, if the environment variable is a database configuration (such as url, port, user, keyword, etc.), the corresponding virtual environment variable can be a virtual string; if the environment variable is a date, the corresponding virtual environment variable can be a common date (such as 20241231, etc.).

[0079] In some embodiments, the computer device also creates a Python environment directory and configures the correct version of Python, pip, and other tools using tools such as venv (Python's official virtual environment tool) and conda (a cross-platform package and environment management tool).

[0080] Using this implementation method, a virtual environment for running Python scripts can be quickly constructed based on the Python library dictionary and environment variable dictionary, providing an extraction environment for subsequent SQL statement extraction.

[0081] In one alternative implementation, Figure 1 Step S103 in the structured query extraction method shown, namely, the computer device reconstructs the original syntax tree based on the Python library dictionary and the list of conditional components to obtain the reconstructed syntax tree, can be achieved as follows: Based on the Python library dictionary, determine at least one database connector corresponding to the Python script, and update the node corresponding to each database connector in the at least one database connector to a structured query capture node; Based on the list of conditional components, construct a list of conditional branches corresponding to the Python script, and insert variable assignment nodes at the positions corresponding to each branch component included in the list of conditional branches; the variable assignment nodes include preset parameters; Based on the multiple conditional branches after inserting variable assignment nodes and the multiple structured query capture nodes, reconstruct the original syntax tree to obtain the reconstructed syntax tree.

[0082] In some embodiments, the computer device determines at least one database connector corresponding to a Python script based on a Python library dictionary. This may include: determining the name and alias of at least one database connector used by the Python script based on the Python library dictionary. The name and alias of each database connector may be denoted as {m conn :alias conn}, where m conn This refers to the name of the database connector, alias. conn This refers to an alias for the database connector.

[0083] In some embodiments, the computer device updates the node corresponding to each database connector in at least one database connector to a structured query statement catcher node. This can be done based on the name or alias of each database connector in at least one database connector.

[0084] In some embodiments, the computer device constructs a list of conditional branches corresponding to a Python script based on a list of conditional components by: for each conditional component included in the list of conditional components, constructing a branch component corresponding to the conditional component based on the parameters, values, logic, and position of the conditional component; and obtaining a list of conditional branches corresponding to the Python script based on the branch components corresponding to each conditional component.

[0085] Optionally, before constructing the branch component corresponding to the conditional component based on the parameters, values, logic, and position of the conditional component, the computer device may also extract the parameters corresponding to the conditional component (denoted as p). c ), value (denoted as v) c ), logic (denoted as o) c) and position (denoted as i) c ).

[0086] Optionally, the computer device extracts the parameter p corresponding to the conditional component. c Value v c , logic o c Then, the parameter p can be combined into a dictionary. c Value v c With logic o c Save as a triple (p c v c o c (Where, the value v is taken as...) c The data types can include, but are not limited to, Python's native data types such as numbers, strings, booleans, and lists; logical 0 c This can include, but is not limited to, Python's native logical operators such as =, <, >, and in.

[0087] For example, for the code "a in [1,2,3]", the triple (p c v c o c = ('a', [1,2,3], 'in'); for the code "a > 10", the triple (p) = ('a', [1,2,3], 'in'); c v c o c = ('a', 10, '>').

[0088] In some embodiments, the computer device constructs branch components corresponding to conditional components based on their parameters, values, logic, and positions. This can be achieved by: determining the value range that satisfies the values ​​and logic corresponding to the conditional component; sampling within and outside the value range, and determining a list of branch values ​​for the parameters corresponding to the conditional component based on the sampling results; and generating the branch components corresponding to the conditional component based on its position, parameters, and the list of branch values. This allows for a simple and quick determination of the branch components corresponding to each conditional component, thereby enabling the generation of a list of conditional branches for the Python script based on the branch components corresponding to each conditional component.

[0089] Optionally, the computer device determines the value range and logical range of the component that satisfies the condition, which can be achieved by using a mathematical solver to determine the value v corresponding to the component that satisfies the condition. c With logic o c The range of values. For example, the data solver is, for instance, the Z3 solver (a solver that finds one of the feasible solutions to a set of constraints by checking the satisfiability of the logical expression).

[0090] Optionally, when the computer device samples within and outside the value range based on the value range, the sampling method can be any one of the sampling methods based on the value range, such as random sampling, rule-based sampling, neural network sampling, or large language model sampling.

[0091] Optionally, the computer device determines the list of branch values ​​for the parameters corresponding to the conditional component based on the sampling results. This can be achieved by using the list of sampling results as the list of branch values ​​for the parameters corresponding to the conditional component. The list of branch values ​​can be denoted as b. c .

[0092] For example, suppose for a triple (p c v c o c Given ('a', [1,2,3], 'in'), the computer device can first determine the value range that satisfies the logical condition as [1,2,3]. Then, the computer device can sample within and outside this value range to obtain the sampling result, i.e., the branch value list b. c =[1,4]. Assume that for the triple (p... c v c o c ) = ('a', 10, '>'), the computer device can first determine the range of values ​​that satisfy the logic as

[10] , and then the computer device can sample within and outside the range of values ​​to obtain the sampling result, that is, the branch value list b. c =[9,10].

[0093] Optionally, the computer device generates a branch component corresponding to the condition component based on the position, parameters, and branch value list of the condition component. This can be achieved by: setting the position corresponding to the condition component (denoted as i) as... c ), parameter p c and the branch value list b c The resulting triplet is identified as the branch component corresponding to condition component c. For example, the branch component corresponding to condition component c can be (i... c p c b c ).

[0094] In some embodiments, when the computer device inserts a variable assignment node at the position corresponding to each branch component included in the conditional branch list, the variable assignment node can be denoted as p. c =b c [k c ]. Where, k c k is a parameter c are integers and 0≤k c ≤length(b c)-1, length(b c The ) indicates the length of the conditional branch. For example, if b c [k c If ]=1, the node to which the variable is assigned a value will be "p". c =1".

[0095] For example, suppose that for the branch component (i) corresponding to the condition component c c p c b c Computer equipment can be used in i c Insert variable assignment node p c =b c [k c ].

[0096] Using this implementation method, a syntax tree (i.e., a reconstructed syntax tree) containing preset parameters and SQL catcher nodes can be determined, thereby providing a foundation for subsequent SQL statement extraction.

[0097] In one alternative implementation, Figure 1 Step S101 in the structured query extraction method shown, namely, the way the computer device determines the Python library dictionary, environment variable dictionary, and condition component list of the Python script based on the original syntax tree corresponding to the Python script to be processed, can be: determining the Python libraries, environment variables, and condition components used by the Python script based on the node type of the original syntax tree corresponding to the Python script to be processed; and determining the Python library dictionary corresponding to the Python library, the environment variable dictionary corresponding to the environment variable, and the condition component list corresponding to the condition component based on the structure of the original syntax tree.

[0098] Optionally, the Python libraries used by the Python script may include those imported using import and from...import... methods.

[0099] Optionally, the environment variables used by the Python script may include those read using methods such as os.environ.get.

[0100] Optionally, the conditional constructs used in Python scripts may include if...else, for loops, while loops, etc.

[0101] In some embodiments, the computer device determines the Python libraries, environment variables, and conditional components used by the Python script based on the node types of the original syntax tree corresponding to the Python script to be processed. This can be achieved by traversing the nodes in the original syntax tree in a top-down order and determining the Python libraries, environment variables, and conditional components used by the Python script based on the node types of each node. In this way, the computer device can extract the key components needed for subsequent steps by performing a depth-first traversal of the original syntax tree in a top-down order and matching the nodes of the syntax tree.

[0102] In some embodiments, when a computer device determines the list of conditional components corresponding to conditional components based on the structure of the original syntax tree, it may do so based on the position of the conditional component in the original syntax tree.

[0103] By using this implementation method, key components (i.e., Python libraries, environment variables, and conditional components used by Python scripts) can be extracted from the original syntax tree, providing a data foundation for subsequently building a virtual environment for running Python scripts and reconstructing the syntax tree.

[0104] The SQL statement extraction method provided in the embodiments of this application will be described in general below. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating another method for extracting structured query statements provided in an embodiment of this application. This method can be executed by a computer device. Figure 2 As shown, this method for extracting structured query statements may include, but is not limited to, the following steps:

[0105] S201. Parse the Python script to be processed to obtain the original syntax tree corresponding to the Python script.

[0106] In one alternative implementation, the computer device can use a syntax parsing tool to parse the Python script to be processed, and obtain the original syntax tree corresponding to the Python script to be processed.

[0107] S202. Perform a depth-first traversal of the original syntax tree in a top-down order, and determine the Python libraries, environment variables and conditional constructs of the Python script based on the nodes of the original syntax tree.

[0108] Optionally, the Python libraries used by the Python script may include those imported using import and from...import... methods; the environment variables used by the Python script may include those read using os.environ.get methods; and the conditional constructs used by the Python script may include if...else, for loops, while loops, etc.

[0109] S203. Based on the structure of the original syntax tree, determine the Python library dictionary corresponding to the Python library, the environment variable dictionary corresponding to the environment variable, and the list of condition components corresponding to the condition components.

[0110] The Python library dictionary represents the mapping between the Python libraries used by the Python script and their aliases, while the environment variable dictionary represents the mapping between the environment variables used by the Python script and their names.

[0111] S204. For each condition component included in the condition component list, construct the corresponding branch component based on the parameters, values, logic, and position of the condition component.

[0112] In one alternative implementation, the computer device constructs a branch component corresponding to each condition component in the condition component list, based on the parameters, values, logic, and position of the condition component. This may include, but is not limited to, the following steps:

[0113] Step 1: For the list of conditional components (denoted as D) c For each conditional component c∈D c Extract the parameters p involved. c Value v c With logic o c And store them as triples (p) in dictionary form. c v c o c ).

[0114] Step 2, for triples (p) c v c o c Based on the value v c , logic o c Construct parameter p c Branch value list b c p c The method for constructing the branch values ​​is as follows: first, determine the values ​​that satisfy v using a mathematical solver. c o cThe value range is then determined, and samples are taken from both within and outside the value range to obtain the branch value list b. c .

[0115] Step 3: Position i of conditional component c c Parameter p c The branch value list b obtained in step two above c Form a triple (i c p c b c ), the triple (i c p c b c ) is the branch component corresponding to the condition component c.

[0116] S205. Based on the branch component corresponding to each condition component, obtain the list of condition branches corresponding to the Python script.

[0117] S206. Construct an SQL catcher, which is used to indicate the output of structured query statement code.

[0118] In an optional implementation, the relevant description of step S206 can be found in the relevant description of step S103 above, and will not be repeated here.

[0119] S207. Based on the Python library dictionary and environment variable dictionary, construct a virtual environment for running Python scripts.

[0120] In an alternative implementation, the relevant description of step S207 can be found in the previous description of the computer device constructing a virtual environment for running Python scripts based on the Python library dictionary and environment variable dictionary, and will not be repeated here.

[0121] S208. Based on the Python library dictionary and the list of conditional components, the original syntax tree is reconstructed to obtain the reconstructed syntax tree; the reconstructed syntax tree includes variable assignment nodes and structured query statement catcher nodes.

[0122] In an alternative implementation, the relevant description of step S208 can be found in the aforementioned description of how the computer device reconstructs the original syntax tree based on the Python library dictionary and the list of conditional components to obtain the reconstructed syntax tree, which will not be repeated here.

[0123] S209. Convert the reconstructed syntax tree into a new Python script and run it in a virtual environment. By adjusting the parameter values ​​of the variable assignment nodes, the branch traversal is completed, and the structured query statement extraction result of the Python script to be processed is obtained.

[0124] In an optional implementation, the relevant description of step S209 can be found in the foregoing detailed description of steps S104 and S105, and will not be repeated here.

[0125] In this embodiment, since the reconstructed syntax tree includes variable assignment nodes and structured query statement catcher nodes, by traversing each parameter value of each variable assignment node in the reconstructed syntax tree and converting the syntax tree composed of each value combination into a new Python script, it is beneficial to reduce the omission of SQL fragments in the subsequent process, thereby improving the completeness of SQL statement extraction. By constructing a virtual environment for running Python scripts, the execution of structured query statements and database execution can be decoupled, improving the generalization and flexibility of SQL statement extraction. Since the reconstructed syntax tree includes SQL catcher nodes, running the corresponding new Python script in the virtual environment can directly capture SQL code through the SQL catcher, improving the security of SQL statement extraction. In other words, by adopting this method, the generalization, security, and completeness of SQL statement extraction can be improved.

[0126] It should be understood that, although Figure 1 and Figure 2 The steps in the flowchart shown are displayed sequentially as indicated by the arrows; however, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order in which these steps are performed, and they can be executed in other orders. Furthermore, Figure 1 and Figure 2 At least some of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0127] Based on the same inventive concept, this application also provides a structured query statement extraction device for implementing the structured query statement extraction method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more structured query statement extraction device embodiments provided below can be found in the limitations of the structured query statement extraction method described above, and will not be repeated here.

[0128] Please see Figure 3 , Figure 3 This is a schematic diagram of a structured query statement extraction device provided in an embodiment of this application. Figure 3 As shown, the device may include, but is not limited to:

[0129] The determination module 301 is used to determine the Python library dictionary, environment variable dictionary, and condition component list of the Python script based on the original syntax tree corresponding to the Python script to be processed. The Python library dictionary is used to represent the mapping relationship between the Python libraries used by the Python script and the aliases of the Python libraries. The environment variable dictionary is used to represent the mapping relationship between the environment variables used by the Python script and the names of the environment variables.

[0130] Module 302 is used to build a virtual environment for running Python scripts based on a dictionary of Python libraries and a dictionary of environment variables.

[0131] Module 302 is also used to reconstruct the original syntax tree based on the Python library dictionary and the list of conditional constructs to obtain a reconstructed syntax tree; the reconstructed syntax tree includes variable assignment nodes and structured query catcher nodes; the structured query catcher nodes are used to indicate the output structured query code;

[0132] The processing module 303 is used to traverse the variable assignment node of the reconstructed syntax tree and obtain the parameter values ​​of each parameter, to obtain multiple combinations of values, to convert the syntax tree formed by each combination of values ​​into a new Python script, to run in a virtual environment, and to obtain the running results under each combination of values.

[0133] The processing module 303 is also used to extract the results of the structured query statement code formed by each running result into the structured query statement of the Python script to be processed.

[0134] In some embodiments, when the construction module 302 constructs a virtual environment for running Python scripts based on a Python library dictionary and an environment variable dictionary, it specifically performs the following steps: based on the Python library dictionary, it determines at least one database connector corresponding to the Python script; it determines the remaining dependent libraries in the Python library dictionary besides the at least one database connector; based on the environment variable dictionary, it determines at least one environment variable corresponding to the Python script, and updates the virtual environment variable corresponding to each environment variable to a virtual value; the virtual value is associated with the meaning of the corresponding environment variable; and based on the remaining dependent libraries, at least one virtual value, and a pre-built structured query capture tool, it constructs a virtual environment for running Python scripts.

[0135] In some embodiments, when the construction module 302 reconstructs the original syntax tree based on the Python library dictionary and the conditional component list to obtain a reconstructed syntax tree, it specifically performs the following steps: based on the Python library dictionary, it determines at least one database connector corresponding to the Python script and updates the node corresponding to each database connector in the at least one database connector to a structured query statement catcher node; based on the conditional component list, it constructs a conditional branch list corresponding to the Python script and inserts a variable assignment node at the position corresponding to each branch component included in the conditional branch list; the variable assignment node includes a preset parameter; based on the multiple conditional branches after inserting variable assignment nodes and the multiple structured query statement catcher nodes, it reconstructs the original syntax tree to obtain a reconstructed syntax tree.

[0136] In some embodiments, when the construction module 302 constructs a conditional branch list corresponding to a Python script based on the conditional component list, it specifically performs the following steps: for each conditional component included in the conditional component list, constructs a branch component corresponding to the conditional component based on the parameters, values, logic, and position of the conditional component; and obtains a conditional branch list corresponding to the Python script based on the branch component corresponding to each conditional component.

[0137] In some embodiments, when the construction module 302 constructs a branch component corresponding to a condition component based on the parameters, values, logic, and position of the condition component, it is specifically used to: determine the value range that satisfies the values ​​and logic corresponding to the condition component; sample within and outside the value range based on the value range, and determine the branch value list of the parameters corresponding to the condition component based on the sampling results; and generate the branch component corresponding to the condition component based on the position, parameters, and branch value list of the condition component.

[0138] In some embodiments, when determining the Python library dictionary, environment variable dictionary, and condition component list of the Python script based on the original syntax tree corresponding to the Python script to be processed, the determining module 301 is specifically used to: determine the Python libraries, environment variables, and condition components used by the Python script based on the node type of the original syntax tree corresponding to the Python script to be processed; and determine the Python library dictionary corresponding to the Python library, the environment variable dictionary corresponding to the environment variable, and the condition component list corresponding to the condition component based on the structure of the original syntax tree.

[0139] It is understood that the steps that can be implemented by each module in the device and the beneficial effects that can be achieved can be referred to the description in the aforementioned embodiments of the structured query statement extraction method, and will not be repeated here.

[0140] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a structured query extraction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0141] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one exemplary embodiment, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described structured query extraction method.

[0143] In one exemplary embodiment, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described structured query extraction method.

[0144] In one exemplary embodiment, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the structured query extraction method described above.

[0145] It should be noted that the data involved in this application (including but not limited to Python scripts to be processed, raw syntax trees, Python library dictionaries, environment variable dictionaries, and condition component lists, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for extracting structured query statements, characterized in that, The method includes: Based on the original syntax tree corresponding to the Python script to be processed, the Python library dictionary, environment variable dictionary, and condition component list of the Python script are determined; the Python library dictionary is used to represent the mapping relationship between the Python libraries used by the Python script and the aliases of the Python libraries, and the environment variable dictionary is used to represent the mapping relationship between the environment variables used by the Python script and the names of the environment variables; Based on the Python library dictionary and the environment variable dictionary, construct a virtual environment for running the Python script; Based on the Python library dictionary and the list of conditional components, the original syntax tree is reconstructed to obtain a reconstructed syntax tree; the reconstructed syntax tree includes variable assignment nodes and structured query statement catcher nodes; the structured query statement catcher nodes are used to indicate the output structured query statement code; By traversing each parameter value of each variable assignment node in the reconstructed syntax tree, multiple value combinations are obtained. The syntax tree formed by each value combination is converted into a new Python script, which is then run in the virtual environment to obtain the running results under each value combination. The structured query statement code formed by each running result is used to extract the structured query statement result of the Python script to be processed.

2. The method according to claim 1, characterized in that, The step of constructing a virtual environment for running the Python script based on the Python library dictionary and the environment variable dictionary includes: Based on the Python library dictionary, at least one database connector corresponding to the Python script is determined; Identify the remaining dependent libraries in the Python library dictionary, excluding at least one of the database connectors; Based on the environment variable dictionary, at least one environment variable corresponding to the Python script is determined, and the virtual environment variable corresponding to each environment variable is updated to a virtual value; the virtual value is associated with the meaning of the corresponding environment variable. Based on the remaining dependency libraries, at least one of the virtual values, and a pre-built structured query capture tool, a virtual environment for running the Python script is constructed.

3. The method according to claim 1, characterized in that, The process of reconstructing the original syntax tree based on the Python library dictionary and the list of conditional components to obtain a reconstructed syntax tree includes: Based on the Python library dictionary, at least one database connector corresponding to the Python script is determined, and the node corresponding to each database connector in the at least one database connector is updated to a structured query statement catcher node; Based on the list of conditional components, a list of conditional branches corresponding to the Python script is constructed, and a variable assignment node is inserted at the position corresponding to each branch component included in the list of conditional branches; the variable assignment node includes preset parameters; Based on the conditional branches following multiple variable assignment nodes and the multiple structured query capture nodes, the original syntax tree is reconstructed to obtain a reconstructed syntax tree.

4. The method according to claim 3, characterized in that, The step of constructing the conditional branch list corresponding to the Python script based on the conditional component list includes: For each condition component included in the condition component list, a branch component corresponding to the condition component is constructed based on the parameters, values, logic, and position of the condition component. Based on the branch component corresponding to each of the condition components, a list of condition branches corresponding to the Python script is obtained.

5. The method according to claim 4, characterized in that, The construction of branch components corresponding to the conditional components based on their parameters, values, logic, and positions includes: Determine the value range and logical value range of the component that satisfies the conditions; Based on the value range, sampling is performed within and outside the value range, and based on the sampling results, a list of branch values ​​for the parameters corresponding to the conditional component is determined. Based on the position of the conditional component, the parameter, and the branch value list, a branch component corresponding to the conditional component is generated.

6. The method according to any one of claims 1 to 5, characterized in that, Based on the original syntax tree corresponding to the Python script to be processed, the Python library dictionary, environment variable dictionary, and condition component list of the Python script are determined, including: Based on the node type of the original syntax tree corresponding to the Python script to be processed, determine the Python libraries, environment variables, and conditional components used by the Python script; Based on the structure of the original syntax tree, determine the Python library dictionary corresponding to the Python library, the environment variable dictionary corresponding to the environment variable, and the condition component list corresponding to the condition component.

7. A structured query statement extraction device, characterized in that, The device includes: The determination module is used to determine the Python library dictionary, environment variable dictionary, and condition component list of the Python script based on the original syntax tree corresponding to the Python script to be processed; the Python library dictionary is used to represent the mapping relationship between the Python libraries used by the Python script and the aliases of the Python libraries, and the environment variable dictionary is used to represent the mapping relationship between the environment variables used by the Python script and the names of the environment variables; A building module is used to construct a virtual environment for running the Python script based on the Python library dictionary and the environment variable dictionary; The building module is further configured to reconstruct the original syntax tree based on the Python library dictionary and the list of conditional components to obtain a reconstructed syntax tree; the reconstructed syntax tree includes variable assignment nodes and structured query statement catcher nodes; the structured query statement catcher nodes are used to indicate the output of structured query statement code; The processing module is used to traverse each parameter value of each variable assignment node in the reconstructed syntax tree, obtain multiple value combinations, convert the syntax tree formed by each value combination into a new Python script, run it in the virtual environment, and obtain the running result under each value combination. The processing module is also used to extract the structured query statement code formed by each running result into the structured query statement of the Python script to be processed.

8. A device, characterized in that, include: A memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images