Python code static analysis method and device

Abstract

The present disclosure relates to a Python code static analysis method and device, the method comprising: obtaining source code of a Python project, and converting the source code into an abstract syntax tree; generating an abstract object, the abstract object being used to simulate an object created in a running process of the source code; binding the abstract object to a corresponding variable based on semantic information contained in the abstract syntax tree and Python syntax, to obtain a pointing analysis result of the Python project; obtaining a speculated variable according to a node type of a node in the abstract syntax tree, and obtaining a variable type speculation result of the Python project by speculating a type of each speculated variable; and judging whether the Python project violates a code programming rule and touches an object type error based on the pointing analysis result and the variable type speculation result, to obtain a static analysis result of the Python project. The present disclosure can supplement variable types that cannot be reasoned in the prior art based on a speculation method of syntax and semantics, and comprehensive Python code static analysis is achieved.

Description

TECHNICAL FIELD

[0001] The application relates to a Python code static analysis method and device, and belongs to the field of computer software testing. BACKGROUND

[0002] With the improvement of computer hardware resources, operation performance is no longer the most important standard for selecting a programming language. Whether a programming language can quickly iterate business, facilitate writing and quickly deploy has gradually become the first consideration of Internet companies and developers. Therefore, dynamic type languages have begun to attract more and more attention and favor. Among many dynamic type languages, Python has become the preferred dynamic type language of many developers because of its simple and easy-to-learn syntax. Python has extremely rich and powerful standard libraries and third-party libraries. With the help of class libraries, developers can quickly implement functions to meet various business needs. However, compared with mainstream static type languages such as C, C++ and Java, the flexible language characteristics of Python bring great challenges to the static analysis of Python.

[0003] Although dynamic languages represented by Python have absolute advantages in development speed, the quality of Python code is greatly affected by the level of developers, which has greatly threatened the reliability and maintainability of Python programs. As a dynamic type language lacking type information at the compilation stage, Python can only discover potential type errors when running the code, which is a great hidden danger to the stability of the system.

[0004] In order to improve the reliability of programs, more and more developers or teams begin to use static program analysis tools to help discover potential errors in the code. Static analysis can scan the code without executing the code, and detect the code that is inconsistent with the rules set in advance through various algorithms. Compared with dynamic analysis, static analysis tools can scan all the code of a project, even including the code that will not be executed during running, which is difficult for dynamic analysis to achieve such coverage. However, the detection rules of the widely used Python static analysis tools are mainly on code style, and the reason is that the type system is missing. Therefore, it is necessary to use pointer analysis and type inference technology to further improve the detection accuracy.

[0005] Pointer analysis

[0006] Points-to analysis is a compile-time analysis technique that aims to identify the memory locations that a variable can point to during the execution of a program. This technique is often a component of other more complex program analysis techniques and improves the precision of the analysis by incorporating the specific features of the programming language. Like other static program analysis techniques, the precision of points-to analysis depends heavily on the sensitivity of the analysis, including flow sensitivity, path sensitivity, and context sensitivity. For object-oriented languages, there are also field sensitivity and object sensitivity.

[0007] Flow sensitivity. Flow-sensitive analysis techniques take into account the execution order of the code, and their input is an ordered sequence of code. On the other hand, flow-insensitive analysis does not consider the execution order of the code, and its input is an unordered set of code statements.

[0008] Context sensitivity. For context-sensitive analysis, the context information (such as parameter information) of the calling point is passed to the called method each time a method is called. Context-insensitive analysis does not consider the context in which the method call point is located.

[0009] Field sensitivity. In field-sensitive analysis, the field information (member information) of each instance object is saved separately. On the other hand, field-insensitive analysis considers all instance objects of a class to have the same field information.

[0010] Generally, the more sensitive the analysis algorithm considers, the more accurate the analysis result is, and the more time or space overhead it has. In consideration of the above, the present invention will use a flow-sensitive points-to analysis algorithm. In addition, for object-oriented languages, context sensitivity has been considered one of the most important factors to ensure high precision of the algorithm, and for Python, analysis of instance objects is more meaningful than analysis of types, so field sensitivity is also one of the factors to ensure high precision of points-to analysis for Python.

[0011] For dynamic languages like Python, the lack of type information at compile time increases the difficulty of program analysis and limits the ability of tools to analyze type-related rules, so statically supplementing type information for Python variables can greatly expand the content that program analysis tools can analyze.

[0012] Type inference

[0013] Type inference refers to automatically inferring the type of a variable without running the code, which is initially widely applied in the design of functional programming languages. With the increasing demand for program analysis of dynamic type languages, type inference is also applied to static program analysis.

[0014] In technology, type inference can be divided into constraint-based reasoning methods and machine learning-based probabilistic methods. Python is famous for its rich and easy-to-use standard library and third-party libraries, but the introduction of third-party libraries will lead to the lack of analysis code, so the analysis accuracy of reasoning methods that rely on complete data flow information will decrease with the increase of code involving third-party libraries. For machine learning, the quality and quantity of the training set affect the quality of the final result to some extent. For Python, the lack of sufficient training sets limits the application of machine learning in Python type inference. In summary, the reasoning-based method cannot solve the data flow loss problem caused by the lack of code in theory, and except for providing manually annotated type annotation files, further increasing the number of inferable variables can only rely on rules to speculate. SUMMARY

[0015] In view of the lack of sufficient machine learning training sets for Python, the unreliability of probabilistic methods contradicts the goal of the tool to have as low a false positive rate as possible, and because the abstract objects used in the analysis point to save type information, the present application discloses a Python code static analysis method and device, which supplements the variable types that cannot be reasoned in the prior art based on the speculation method of syntax and semantics, and realizes comprehensive Python code static analysis.

[0016] The technical content of the present application includes:

[0017] A Python code static analysis method, the method comprising:

[0018] Obtaining the source code of a Python project, and converting the source code into an abstract syntax tree;

[0019] Generating an abstract object, the abstract object being used to simulate the object created in the running process of the source code;

[0020] Based on the abstract syntax tree and the semantic information implied by the Python syntax, binding the abstract object to the corresponding variable to obtain the analysis result of the Python project;

[0021] According to the node type of the node in the abstract syntax tree, obtaining the speculated variable, and obtaining the variable type speculation result of the Python project by speculating the type of each speculated variable;

[0022] Based on the analysis result of the pointing and the variable type speculation result, it is judged whether the Python project violates the code programming rules and touches the object type error, so as to obtain the static analysis result of the Python project.

[0023] Further, the abstract object includes a basic type, a code region type, a built-in object type and other types; the basic type includes a number, a boolean type, a string, a tuple, a list, a set, a dictionary and a null value; the code region type includes a program package definition, a user class definition and a branch block; the built-in object type includes a built-in function and a built-in class; and the other types include an instance object of a class, an unknown object, a multiple object, a method of a bound parameter and an undefined object.

[0024] Further, the abstract object is bound to the corresponding variable based on the abstract syntax tree and the semantic information implied by the Python grammar, so as to obtain the pointing analysis result of the Python project, including:

[0025] A pointing analysis assignment operation rule is generated; the pointing analysis assignment operation rule represents the assignment operation rule of the assignment statement in the Python project;

[0026] Based on the pointing analysis assignment operation rule, the flow sensitivity, context sensitivity and domain sensitivity pointing analysis is performed on each file in the Python project, so as to obtain the syntax analysis result of the Python project; wherein, when the file references other files and the other files have not been subjected to the pointing analysis, the pointing analysis of the file is completed after the pointing analysis of the other files is completed.

[0027] Further, the pointing analysis assignment operation rule includes:

[0028] For the assignment statement with the basic type expression on the right side, if the basic type expression points to a first abstract object, then the left side variable also points to the first abstract object;

[0029] and / or,

[0030] For the assignment statement with the variable on the right side, if the right side variable points to a second abstract object, then the left side variable also points to the second abstract object;

[0031] and / or,

[0032] For the assignment statement with the domain of the variable on the right side, if the right side variable points to a third abstract object, and a fourth abstract object is a domain of the third abstract object, then the left side variable points to the fourth abstract object;

[0033] and / or,

[0034] For the assignment statement whose right side is a container expression with subscript index, if the right side variable points to the fifth abstract object, and the sixth abstract object is the value of the fifth abstract object at index, then the left side variable points to the sixth abstract object;

[0035] and / or,

[0036] For the assignment statement whose right side is a binary expression with operator op acting on the first expression and the second expression, if the first expression points to the seventh abstract object, the second expression points to the eighth abstract object, and the ninth abstract object is the value of the op operation of the seventh abstract object and the eighth abstract object, then the left side variable points to the ninth abstract object;

[0037] and / or,

[0038] For the assignment statement whose right side is a conditional judgment, according to the tenth abstract object pointed to by the conditional expression, bind the corresponding right side expression to the left side variable;

[0039] and / or,

[0040] For the assignment statement whose right side is a normal method call, get the method object pointed to by all arguments, actual call, bind the formal parameter to the object pointed to by the argument, and return the value to the left side variable;

[0041] and / or,

[0042] For the assignment statement whose right side is a class constructor method call, on the basis of the pointing analysis assignment operation rule of the assignment statement whose right side is a normal method call, initialize the self object, bind the first formal parameter to the self object, and do not bind the return value but bind the self object to the left side variable;

[0043] and / or,

[0044] For the assignment statement whose right side is an instance method call, on the basis of the pointing analysis assignment operation rule of the assignment statement whose right side is a class constructor method call, do not initialize the self object and get the self object from the pointing of the instance object t, and bind the return value to the left side variable.

[0045] Further, the variable name of the speculated variable is obtained according to the node type of the node in the abstract syntax tree, and the variable type speculation result of the Python project is obtained by speculating the type of each speculated variable, comprising:

[0046] a variable speculation type table is constructed; the variable speculation type table comprises a type speculation table of a property, a type speculation table of a subscript syntax, and a type speculation table of a binary operation;

[0047] Initialize the variable type relation table;

[0048] Traverse the abstract syntax tree;

[0049] When the node type of a node is attribute, subscript syntax or binary operator, obtain the variable name of a speculated variable, infer the variable type of the speculated variable according to the variable speculation type table, and add the variable type to the variable type relation table to obtain the type list of the speculated variable at the node;

[0050] When the node type of a node is the definition of a module, class or function, take the intersection of the elements in the type list of each speculated variable, and when the element in the intersection is 1, update the type list of the speculated variable using the element in the intersection;

[0051] Until the end of the traversal, obtain the variable type speculation result of the Python project based on the current variable type relation table.

[0052] Further, the code programming rules include:

[0053] Importing all members of a module using from xxx import * is prohibited;

[0054] And / or,

[0055] When the document string is more than one line, the last three quotation marks should be in a separate line;

[0056] And / or,

[0057] When using slicing on a sequence, do not use negative values other than -1 for slicing;

[0058] And / or,

[0059] Use a comprehension instead of a repetitive logical operation to construct a sequence;

[0060] And / or,

[0061] When the method of a class does not need to access the instance, use @staticmethod to decorate;

[0062] And / or,

[0063] Do not modify the sys.path list in the code;

[0064] And / or,

[0065] When using the try…except… structure to protect the code, use the finally… structure after the exception to ensure the release of the operation object;

[0066] And / or,

[0067]

[0067] In the case where the number of members and the content are unchanged, use tuple instead of list;

[0068] and / or,

[0069] Use generator comprehension instead of list comprehension;

[0070] and / or,

[0071] Use the format method, the "%" operator and the join method instead of "+" and "+ =" operators to complete the string formatting.

[0072] Further, the object type error includes: AttributeError error, the AttributeError error indicates the error caused by calling the attribute that does not exist in the object;

[0073] The judgment whether the Python project touches AttributeError error based on the pointing analysis result and the variable type speculation result, comprising:

[0074] Based on the pointing analysis result and the variable type speculation result, obtain the object instance pointed by the called variable, and detect whether the object instance exists calling attribute;

[0075] In the case where the calling attribute exists, it is determined as a legal operation;

[0076] In the case where the calling attribute does not exist, it is detected whether the object instance is a self-defined type object;

[0077] In the case where the object is not a self-defined type object, it is determined as a legal operation;

[0078] In the case where the object is a self-defined type object, it is detected whether the parent class of the object instance has the called attribute;

[0079] In the case where the parent class of the object instance has the called attribute, it is determined as a legal operation;

[0080] In the case where the parent class of the object instance does not have the called attribute, it is detected whether the parent class of the object instance belongs to a third-party library;

[0081] In the case where the parent class of the object instance belongs to a third-party library, it is determined as a legal operation;

[0082] In the case where the parent class of the object instance does not belong to a third-party library, it is determined as an AttributeError error.

[0083] Further, the object type error includes a TypeError error;

[0084] The TypeError error includes:

[0085] An inadmissible binary operation occurs between two object instances;

[0086] And,

[0087] An object other than an integer and a type defined index Method is used as a subscript of a list, tuple or string;

[0088] And,

[0089] A non-callable object is called in the form of a method call;

[0090] And,

[0091] The number of parameters of a method call is incorrect;

[0092] And,

[0093] And a type that will not support hashing is put into a set or as a key of a dictionary.

[0094] A Python code static analysis device, the device comprises:

[0095] A syntax tree construction module, configured to acquire source code of a Python project, and convert the source code into an abstract syntax tree;

[0096] An object generation module, configured to generate abstract objects, the abstract objects being used to simulate objects created in a running process of the source code;

[0097] A pointing analysis module, configured to bind abstract objects to corresponding variables based on semantic information contained in the abstract syntax tree and Python syntax, so as to obtain a pointing analysis result of the Python project;

[0098] A type speculation module, configured to acquire speculated variables according to node types of nodes in the abstract syntax tree, and obtain a variable type speculation result of the Python project by speculating a type of each of the speculated variables;

[0099] A result inference module, configured to judge whether the Python project violates a code programming rule and touches an object type error based on the pointing analysis result and the variable type speculation result, so as to obtain a static analysis result of the Python project.

[0100] An electronic device, characterized in that the electronic device comprises a processor and a memory storing computer program instructions; the processor implements the method of any one of the preceding method embodiments when executing the computer program instructions.

[0101] A computer readable storage medium, having stored thereon computer program instructions, characterized in that the program instructions, when executed by a processor, implement the method of any one of the preceding method embodiments.

[0102] Compared with the prior art, the present application has at least the following advantages:

[0103] 1) The flow-sensitive, context-sensitive and domain-sensitive analysis method is combined to quickly and accurately obtain the relationship of each variable and the type at each time point.

[0104] 2) Detailed rules are constructed, a large number of built-in functions and class definitions are processed, and the analysis and type inference algorithm is supplemented.

[0105] 3) The checking of the semantic rules of 10 python coding specifications and the syntax rules of 2 is built-in, which has wide coverage and high readability of the report. BRIEF DESCRIPTION OF DRAWINGS

[0106] Figure 1 The flowchart of the Python code static analysis method of the present application.

[0107] Figure 2 The block diagram of the Python code static analysis device of the present application. DETAILED DESCRIPTION

[0108] The present application will be described in detail below in combination with the drawings and examples, and it should be pointed out that the described examples are only intended to facilitate the understanding of the present application and do not limit the present application in any way.

[0109] In order to ensure that the algorithm can analyze a large-scale Python project in a short time, the present application adopts a flow-sensitive, context-sensitive and domain-sensitive combined analysis algorithm in the analysis of the direction, and additionally uses a syntax and semantic-based type inference algorithm to expand the type of the variable on the basis of the analysis result of the direction. Finally, on the basis of the analysis of the direction and the type inference, the tool will analyze the file according to the programming rules of the code set in advance and record the errors.

[0110] As shown in Figure 1 The Python code static analysis method of the present application comprises the following steps.

[0111] Step 1: Obtain the source code of the Python project, and convert the source code into an abstract syntax tree.

[0112] Abstract Syntax Tree (AST) is a common internal data structure used in compiler front-end, which is a tree representation of the abstract syntax structure of source code. As the product of syntax analysis, it can losslessly represent the syntax structure of the program, and each node on the tree retains the code information in the source code, which can provide necessary context information for static analysis.

[0113] In the process of CPython, the official interpreter of Python, AST is an intermediate expression from Python source code to bytecode, and in the Python standard library, the official provides the ast module, which allows developers to view and modify AST objects through API.

[0114] In an example, in order to better analyze the code and better feedback and positioning of the detection result, some supplements are made to the content of the syntax tree based on the AST object generated by the Python standard library, including: (1) adding the parent node reference of all syntax tree nodes. (2) Adding the corresponding source file line number to all syntax tree nodes. (3) Binding the comment information to the syntax tree. (4) Adding the corresponding Python code text description to the syntax tree node.

[0115] Step 2: Generate abstract objects.

[0116] The ultimate goal of pointer analysis is to identify the memory locations that the pointer may point to during runtime. Since Python language does not have pointer operations, Python-oriented pointer analysis aims to study the objects that variables may point to during runtime. Since static analysis does not run the source code of the program, abstract objects are needed to simulate the objects created during the program. According to different categories, the invention divides abstract objects into four categories, including: basic types, code region types, built-in object types and other types.

[0117] Basic type.

[0118] Basic types include Python's standard data types, including numbers (integers and floating-point numbers), booleans, strings, tuples, lists, sets, dictionaries, and in addition to null (None), corresponding to IntObject, FloatObject, BoolObject, StringObject, TupleObject, ListObject, DictObject, SetObject and NoneObject, a total of 9 kinds of abstract object types.

[0119] Code region type.

[0120] Code region type refers to the modeling of a code segment containing one or more code statements, including module, class definition, method definition, branch block. For any one code region type, a local variable table is maintained to record the abstract objects pointed to by local variables. ModuleDefinition In Python language, each Python file is a module, which contains the definition of Python objects and Python statements. Class definitions, method definitions, and local variables in a module can be referenced by other modules (Python files).

[0121] PackageDefinition In Python language, a folder with __init__.py file is a Python package. According to the reference semantics of Python, a package is a special module, and the content of __init__.py file can be regarded as the content of the package module. In addition, all modules in the package can be regarded as local variables of the package module. FunctionDefinition is a method definition that binds the corresponding syntax tree for method invocation.

[0122] ClassDefinition is a user class definition, which is a special FunctionDefinition. In addition to binding the corresponding syntax tree node, it also binds the __init__() method (i.e. the constructor method) in the class for method invocation. Since Python language allows multiple inheritance, it also saves a list to store the ClassDefinition objects corresponding to the parent classes.

[0123] BodyDefinition is a branch block, which contains the branch code block under if, for, with, and try statements, and is mainly used for merging local variables after the branch ends.

[0124] Built-in object type

[0125] Built-in object type is the modeling of python library calls, including python built-in functions and built-in class definitions.

[0126] BuiltinFunctionDefinition is a built-in function, which is different from FunctionDefinition. Instead of having a corresponding syntax tree object, it saves a directly executable method.

[0127] BuiltinClassDefinition is a special BuiltinFunctionDefinition. Currently, the only object of this class is the class definition of Python standard types, i.e. int(), float(), str(), bool(), tuple(), list(), dict(), set().

[0128] OtherType

[0129] OtherType is the sum of custom or unanalyzable types.

[0130] ClassObject is the instance object of a class. It is bound to the corresponding ClassDefinition object and stores the attribute name of the instance object and its corresponding abstract object.

[0131] UnknownObject is an unknown object. Since static analysis does not run the program, for external input parameters, unidentified third-party library information, and other contents, the analysis cannot determine the object pointed to by the variable, at which time an object of this class is needed instead.

[0132] MultipleObject is a multiple object. Since the current pointing analysis uses path-insensitive analysis, at the confluence of the path, there is a situation where a variable may point to multiple variables. This class stores a list to save all possible objects.

[0133] BoundMethodObject is a bound parameter method. Python allows internal methods to be bound to parameters to form a closure and exposed. This class saves the corresponding FunctionDefinition object and additionally saves the parameters and their objects.

[0134] UndefinedObject is an undefined object. This class only appears when a variable is not defined in all branches.

[0135] Step 3: Based on the abstract syntax tree and the semantic information implied by the Python syntax, bind the abstract object to the corresponding variable to obtain the pointing analysis result of the Python project.

[0136] By analyzing the abstract syntax tree and according to the semantic information contained in different Python syntax, the abstract objects are propagated to the corresponding variables using flow-sensitive method, context-sensitive method and domain-sensitive method. Among them, the flow-sensitive method outputs different results for different execution order of the code, the context-sensitive method outputs different results for the code

especially functions

especially classes and structures

[0137] The symbol v represents the assigned object. In Python, the assigned object can be not only a general variable, but also a field of an object (such as the Python statement obj.field = 1) or a container accessed by an index (such as the Python statement arr[0] = 1). Although the three types of assignment have some differences in implementation, they are similar in terms of mapping. In addition, the symbol represents only a general variable. The basic type object of Python is the main starting point of object propagation, and the symbol represents the expression corresponding to the basic type, such as the literal 1, 'hello', or fixed syntax such as [1], {key:value}, and the symbol e represents an arbitrary expression. pts(e, o, scope) represents that the expression e points to the abstract object o in the scope scope, and in the case of a default code scope, it represents the scope in which the statement itself is located.

[0138] The abstract object instance stores value information, such as for the list object [1, 2], the abstract object has the following form.

[0139] ListObject([IntObject(value=1), IntObject(value=2)])

[0140] Therefore, the pointing analysis of the present application also supports reading and calculating specific values through index syntax (including index and slice) and operation (including unary operation, binary operation, and Boolean operation). However, the value stored by the object can be unknown, and the inaccuracy of the object stored by the container can cause the operation to fail, so when an illegal operation occurs, the result is UnknownObject. Since the index syntax and the operation are both based on the value processing of Python, only index and binary operation are listed in Table 1 as examples. The symbol op represents a binary operator, which indicates that the binary operation is performed on the values stored in and and the corresponding abstract object is returned.

[0141] For function calls, Table 1 lists the three most common types, namely the call of the constructor, the call of the normal method, and the call of the instance object. dispatch() indicates the location of the actual call method object, init represents the constructor, m.p represents the parameter of the method, The return value expression of the representative method. The three calling methods are slightly different in parameter passing and return value. The order of actual parameters and formal parameters of the ordinary method call is consistent (in fact, there are two special parameter passing methods, variable parameters and keyword parameters, in Python, which will be specially processed in algorithm implementation); The first parameter of the calling method of the instance is the instance object itself, that is, the first actual parameter of the method call corresponds to the second formal parameter of the method definition, and so on; The parameter passing method of the constructor is the same as the instance object, but according to the semantics of the constructor, the abstract object corresponding to the class object of the class object needs to be created as the first parameter of the method and the return value, create(Class, o) indicates to create a ClassObject object of the type Class.

[0142] Table 1 points to the analysis assignment operation rule table

[0143]

[0144] In table 1, e is an expression, base is a basic type, v is a variable, pts(a, b) indicates that a points to b, is another variable, o is an object, field is a field, m is an actual method object, t is an actual parameter, m.p is a formal parameter, self is the instance itself, ret is the return value, and init is the class constructor.

[0145] Step 4: According to the node type of the node in the abstract syntax tree, the speculated variable is obtained, and the variable type speculation result of the Python project is obtained by speculating the type of each speculated variable.

[0146] The present application completes the speculation of the variable in the Python project by traversing the abstract syntax tree. Specifically, the present step includes the following sub-steps.

[0147] Step 4.1: Construct a variable speculation type table.

[0148] The variable speculation type table of the present application includes: a type speculation table of attributes, a type speculation table of subscript syntax and a type speculation table of binary operations.

[0149] 1) Type speculation table of attributes.

[0150] The type speculation table of attributes in Table 2 lists the attribute names that all Python basic types have. The attributes that the classes defined in the project have can be collected in the process of code analysis. When there is attribute calling code such as obj.field, if the type of variable obj is unknown, it will be checked whether there is a type that has an attribute named field. If such a type exists, the type is a possible speculation type. If there are multiple such types, all of them are possible speculation types.

[0151] Table 2 Type speculation table of attributes

[0152]

[0153] 2) Type speculation table of subscript syntax.

[0154] In other programming languages, subscript syntax is often a special syntax unique to arrays, but in Python, both list type (list) and dictionary type (dict) can be operated using this syntax. The types of subscripts are divided into two categories, slice type and index type. Slice refers to the operation of obtaining a sequence fragment from the source sequence. The code example of slice type is collection[start: end: step], where variable collection can be list, tuple or string type; variables start, end, step can be int type. For index type subscript, the code example is collection[index]. If the type of variable index is not an integer, the type of variable collection is dict type, otherwise it is one of list, dict, tuple and string types.

[0155] 3) Type speculation table of binary operation

[0156] In Python programming, binary operation is the operation that is most likely to cause TypeError, but the type restriction of binary operation can also be used to speculate the unknown variable. The core idea of this set of rules is to know the type of the left object of binary operation, and then speculate the possible type of the right object according to the specific operator type. Table 3 describes all the speculation rules under this syntax. When the type of the left object is int, if the operator is multiplication, the type of the right object can be one of int, float, string, tuple, list, set, and if the operator is other than multiplication, the type of the right object can be int or float.

[0157] Table 3 Type speculation table of binary operation

[0158]

[0159] Step 4.2: initialize the variable type relationship table.

[0160] The variable type relationship table is used for the variable inference that may exist in each node of the abstract syntax tree, and the variable inference exists in the type list of the variable.

[0161] Step 4.3: traverse the abstract syntax tree; when the node type of the node is attribute, subscript syntax or binary operator, obtain the variable name of the speculated variable, infer the variable type of the speculated variable according to the variable speculation type table, and add the variable type to the variable type relationship table to obtain the type list of the speculated variable on the node.

[0162] Based on the type list, the present application maintains a speculation queue for each variable. When a speculation-related syntax (attribute, subscript syntax or binary operator) is encountered, the variable name of the speculated variable can be obtained. Then, all speculation types of the speculated variable can be obtained in combination with the above-mentioned variable speculation type table.

[0163] Step 4.4: when the node type of the node is the definition of a module, a class or a function, take the intersection of the elements in the type list of each speculated variable, and when the element of the intersection is 1, update the type list of the speculated variable using the intersection element.

[0164] At the end of the current code scope, take the intersection of all sets in the queue, and if the number of final set elements is 1, take the element as the final speculation type of the variable and update it to the variable type table, otherwise do not update the variable type.

[0165] Step 4.5: until the end of the traversal, obtain the variable type speculation result of the Python project based on the current variable type relationship table.

[0166] After the traversal of the abstract syntax tree ends, the type inference of each variable in the corresponding scope can be obtained based on the current variable type relationship table, so that the analysis result of the pointer can be well supplemented.

[0167] Step 5: based on the analysis result of the pointer and the variable type speculation result, judge whether the Python project violates the code programming rules and touches the object type error, to obtain the static analysis result of the Python project.

[0168] Based on the above technical achievements, the present application selects a number of programming specification rules that affect code quality and TypeError and AttributeError, two common Python errors, as detection rules.

[0169] Step 5.1: Constructing the programming specification rules.

[0170] In the detection logic of the rules, the rules can be roughly divided into two categories: syntax rules and semantic rules. Syntax rules refer to rules that can detect code violations directly through the syntax information of the abstract syntax tree, such as prohibiting the use of a certain syntax format for code development; semantic rules refer to detection objects that contain variable detection, such as prohibiting the modification of a certain specific object or prohibiting operations between certain types of objects, and the semantic information contained in the variable cannot be known through the information of the abstract syntax tree alone.

[0171] In order to prove that the finally implemented tool can not only analyze the code from the syntax level, but also detect the code from the semantic level, the present application selects ten representative rules from each category of rules as the final detection target, and the rules are described as follows.

[0172] Rule 1: Prohibit importing all members of a module using the from xxx import * method.

[0173] Rule 2: When the docstring is more than one line, the last three quotes should be on a separate line.

[0174] Rule 3: When using slicing on a sequence, it is not recommended to use negative values other than -1 for slicing.

[0175] Rule 4: Use comprehension instead of repeated logical operations to construct sequences.

[0176] Rule 5: When the method of a class does not need to access the instance, use @staticmethod to decorate.

[0177] Rule 6: Avoid modifying the sys.path list in the code.

[0178] Rule 7: When using the try…except… structure to protect the code, the finally… structure needs to be used after the exception to ensure the release of the operation object.

[0179] Rule 8: In scenarios where the number and content of members do not change, try to use tuples instead of lists.

[0180] Rule 9: Try to use generator comprehension instead of list comprehension.

[0181] Rule 10: Use the format method, "%" operator, and join method instead of "+" and "+=" operators to complete string formatting.

[0182] In Python, the type-related errors mainly manifest as AttributeError and TypeError, so the present application also takes these two types of errors as one of the detection targets of the tool.

[0183] AttributeError means the error caused by calling a non-existent attribute of an object. Unlike static typed languages, Python allows dynamically adding new attributes to an object, i.e., two objects of the same type may have different attributes.

[0184] Therefore, for the detection of AttributeError, it is not enough to rely on type information alone, and it must rely on pointer analysis to analyze the specific object instance. First, the specific object instance pointed to by the called variable needs to be obtained, and it is detected whether the called attribute exists. If not, it is detected whether it is an object of a custom type; if so, it is detected whether the called attribute exists in its parent class; if not, it is detected whether the parent class belongs to a third-party library (i.e., a class whose definition cannot be detected); if so, it is considered that the attribute is inherited from the parent class of the third library and is a legal operation; otherwise, it is determined to be an AttributeError.

[0185] In programming development practice, the common scenarios for generating TypeError errors are: (1) an unallowed binary operation occurs between two object instances; (2) an object other than an integer and a defined index method is used as an index of a list, tuple, or string; (3) a non-callable object is called in the form of a method call; (4) the number of parameters of a method call is incorrect; (5) a type that does not support hashing (list, set, dictionary) is placed into a set or used as a key of a dictionary.

[0186] In programming development practice, the common scenarios for generating TypeError errors are: (1) an unallowed binary operation occurs between two object instances; (2) an object other than an integer and a defined index method is used as an index of a list, tuple, or string; (3) a non-callable object is called in the form of a method call; (4) the number of parameters of a method call is incorrect; (5) a type that does not support hashing (list, set, dictionary) is placed into a set or used as a key of a dictionary.

[0187] After all the rule analysis is completed, all the errors are summarized, and a JSON file is generated. This JSON file can be used again as an intermediate result for other analysis. In the present application, the generated JSON file is used as a data source to fill in an HTML template, and finally the results of the code detection are fed back to the user in the form of HTML.

[0188] In summary, the present application can use a syntax and semantics-based inference method to supplement variable types that cannot be inferred in the prior art.

[0189] Based on the same concept, the present application also discloses a Python code static analysis device, as shown in the accompanying drawings, comprising a syntax tree construction module, an object generation module, a pointing analysis module, a type inference module and a result inference module. Figure 2

[0190] The syntax tree construction module is configured to obtain the source code of a Python project and convert the source code into an abstract syntax tree.

[0191] The object generation module is configured to generate an abstract object, which is used to simulate an object created in the running process of the source code.

[0192] The pointing analysis module is configured to bind the abstract object to a corresponding variable based on the abstract syntax tree and semantic information implied by Python syntax, so as to obtain a pointing analysis result of the Python project.

[0193] The type inference module is configured to obtain a variable to be inferred according to the node type of a node in the abstract syntax tree, and obtain a variable type inference result of the Python project by inferring the type of each variable to be inferred.

[0194] The result inference module is configured to judge whether the Python project violates a code programming rule and touches an object type error based on the pointing analysis result and the variable type inference result, so as to obtain a static analysis result of the Python project.

[0195] In an example, the system interface design of the present application comprises:

[0196] Command line format: python main.py <detect project or file path> [optional parameters]

[0197] Optional parameter description:

[0198] --result <path>, result generation path, default value is. / result

[0199] --circle <num>, the maximum number of loops allowed during execution, default is 3

[0200] - lib, if this parameter is used, the item is considered to exist as a whole as a Python library. This parameter affects the module finding logic of the cross-process analysis. Not used by default.

[0201] If the item path exists:

[0202] moduleA

[0203] - fileA.py

[0204] - fileB.py

[0205] If the item exists as a library, then the fileB.py needs to import the fileA module using the statement import moduleA.fileA; otherwise, it needs to use the statement import fileA.

[0206] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.< / num> < / path> ​

Claims

1. A static analysis method for Python code, characterized in that, The method includes: Obtain the source code of a Python project and convert the source code into an abstract syntax tree; Generate an abstract object, which is used to simulate the object created during the execution of the source code; Based on the abstract syntax tree and the semantic information contained in the Python syntax, the abstract objects are bound to the corresponding variables to obtain the pointing analysis results of the Python project; Based on the node type of the node in the abstract syntax tree, the variable name of the inferred variable is obtained, and by inferring the type of each inferred variable, the variable type inference result of the Python project is obtained; Based on the pointing analysis results and the variable type inference results, it is determined whether the Python project violates code programming rules and touches on object type errors, so as to obtain the static analysis results of the Python project; Specifically, the step of binding abstract objects to corresponding variables based on the semantic information implied by the abstract syntax tree and Python syntax to obtain the pointing analysis results of the Python project includes: Generate pointer analysis assignment operation rules; the pointer analysis assignment operation rules represent the assignment operation rules of assignment statements in the Python project; Based on the aforementioned pointer analysis assignment operation rules, stream-sensitive, context-sensitive, and domain-sensitive pointer analysis is performed on each file in the Python project to obtain the syntax analysis results of the Python project. The step of obtaining the variable name of the inferred variable based on the node type of the node in the abstract syntax tree, and obtaining the variable type inference result of the Python project by inferring the type of each inferred variable, includes: Construct a variable prediction type table; the variable prediction type table includes: a type prediction table for attributes, a type prediction table for subscript syntax, and a type prediction table for binary operations; Initialize the variable type relationship table; Traverse the abstract syntax tree; When the node type is attribute, subscript syntax or binary operator, obtain the variable name of the inferred variable, infer the variable type of the inferred variable according to the variable inference type table, and add the variable type to the variable type relationship table to obtain the type list of the inferred variable on the node. When the node type is a module, class, or function definition, take the intersection of all elements in the type list of each specified variable, and when the intersection element is 1, update the type list of the specified variable using the intersection element. Until the traversal is completed, the variable type prediction result of the Python project is obtained based on the current variable type relationship table.

2. The method as described in claim 1, characterized in that, The abstract objects include: basic types, code region types, built-in object types, and other types; the basic types include: numbers, booleans, strings, tuples, lists, sets, dictionaries, and null values; the code region types include: package definitions, user class definitions, and branch blocks; the built-in object types include: built-in functions and built-in classes; the other types include: instances of classes, unknown objects, multiple objects, methods with bound parameters, and undefined objects.

3. The method as described in claim 1, characterized in that, When performing pointer analysis on the file, if the file references other files and those other files have not yet undergone pointer analysis, the pointer analysis will be performed on the other files first, and then the pointer analysis on the file will be performed.

4. The method as described in claim 3, characterized in that, The rules for pointer analysis assignment operations include: For assignment statements where the right side is a primitive type expression, if the primitive type expression points to the first abstract object, then the variable on the left side also points to that first abstract object. And / or, For an assignment statement where the right side is a variable, if the variable on the right side points to the second abstract object, then the variable on the left side also points to the second abstract object. And / or, For an assignment statement where the right side is a field of a variable, if the variable on the right side points to a third abstract object and the fourth abstract object is a field of the third abstract object, then the variable on the left side points to the fourth abstract object. And / or, For an assignment statement where the right side is an index-accessed container expression, if the variable on the right side points to the fifth abstract object, and the sixth abstract object is the value of the fifth abstract object at the index, then the variable on the left side points to the sixth abstract object. And / or, For an assignment statement where the right side is a binary expression in which the operator op is applied to the first and second expressions, if the first expression points to the seventh abstract object, the second expression points to the eighth abstract object, and the ninth abstract object is the value of the op operation on the seventh and eighth abstract objects, then the variable on the left side points to the ninth abstract object. And / or, For assignment statements where the right side is a conditional statement, the corresponding right-side expression is bound to the left-side variable based on the tenth abstract object pointed to by the conditional expression. And / or, For assignment statements on the right that are ordinary method calls, we get the pointers to all actual arguments, the actual method object being called, bind the formal parameters to the objects that the actual arguments point to, and return the value to the variable on the left. And / or, For assignment statements where the right side is a call to a class constructor, based on the assignment operation rules for assignment statements where the right side is a call to a normal method, the following additional rules are added: initializing the object itself, binding the first formal parameter to the object itself, and binding the object itself to the variable on the left side instead of binding the return value. And / or, For assignment statements where the right side is an instance method call, based on the assignment operation rules for assignment statements where the right side is a class constructor call, the object itself is not initialized, and the object itself is obtained from the pointer of the instance object t, and the return value is bound to the variable on the left.

5. The method as described in claim 1, characterized in that, The code programming rules include: It is forbidden to import all members of a module using the `from xxx import *` statement; And / or, When a docstring is longer than one line, the last three quotation marks should be on their own line. And / or, When slicing a sequence, do not use negative values ​​other than -1 for slicing; And / or, Construct sequences using comprehensions instead of repetitive logical operations; And / or, When a class method does not need to access an instance, use the @staticmethod decorator. And / or, Do not modify the sys.path list in the code; And / or, When using the try...except... structure to protect code, a finally... structure must be used after an exception to ensure the release of the object being operated on; And / or, In scenarios where the number of members and their contents remain unchanged, use a tuple instead of a list; And / or, Use generator comprehension instead of list comprehension; And / or, Use the format method, the "%" operator, and the join method instead of the "+" and "+=" operators to perform string formatting.

6. The method as described in claim 1, characterized in that, The object type errors include: AttributeError, which indicates an error caused by calling a property that does not exist in the object; The determination of whether the Python project has triggered an AttributeError based on the pointer analysis results and the variable type inference results includes: Based on the pointing analysis results and the variable type inference results, the object instance pointed to by the called variable is obtained, and it is detected whether the object instance has a calling attribute; If the access attribute exists, the operation is deemed legitimate. If no attribute is invoked, check whether the object instance is an object of a custom type; If the object is not a custom type, the operation is considered valid. In the case of a custom type object, check whether the parent class of the object instance has the property to be called; If the attribute being called exists in the parent class of the object instance, the operation is deemed legitimate. If the parent class of the object instance does not possess the attribute being called, check whether the parent class of the object instance belongs to a third-party library; Since the parent class of the object instance belongs to a third-party library, the operation is deemed legitimate. The object instance's parent class does not belong to a third-party library, resulting in an AttributeError.

7. The method as described in claim 1, characterized in that, The object type error includes: TypeError error; The TypeError includes: An unallowed binary operation occurs between two object instances; and, Besides integers and definitions, index Objects outside of methods can be used as indices for lists, tuples, or strings. and, Invoke a non-callable object in the form of a method call; and, The number of parameters in the method call is incorrect; and, And to put types that do not support hashing into collections or as keys in dictionaries.

8. A Python code static analysis device, characterized in that, The device includes: The syntax tree building module is used to obtain the source code of a Python project and transform the source code into an abstract syntax tree. An object generation module is used to generate abstract objects, which are used to simulate objects created during the execution of the source code. The pointer analysis module is used to bind abstract objects to corresponding variables based on the semantic information contained in the abstract syntax tree and Python syntax, so as to obtain the pointer analysis results of the Python project; The type inference module is used to obtain the variable name of the variable to be inferred based on the node type of the node in the abstract syntax tree, and to obtain the variable type inference result of the Python project by inferring the type of each variable to be inferred. The result inference module is used to determine whether the Python project violates code programming rules and touches on object type errors based on the pointing analysis results and the variable type inference results, so as to obtain the static analysis results of the Python project; Specifically, the step of binding abstract objects to corresponding variables based on the semantic information implied by the abstract syntax tree and Python syntax to obtain the pointing analysis results of the Python project includes: Generate pointer analysis assignment operation rules; the pointer analysis assignment operation rules represent the assignment operation rules of assignment statements in the Python project; Based on the aforementioned pointer analysis assignment operation rules, stream-sensitive, context-sensitive, and domain-sensitive pointer analysis is performed on each file in the Python project to obtain the syntax analysis results of the Python project. The step of obtaining the variable name of the inferred variable based on the node type of the node in the abstract syntax tree, and obtaining the variable type inference result of the Python project by inferring the type of each inferred variable, includes: Construct a variable prediction type table; the variable prediction type table includes: a type prediction table for attributes, a type prediction table for subscript syntax, and a type prediction table for binary operations; Initialize the variable type relationship table; Traverse the abstract syntax tree; When the node type is attribute, subscript syntax or binary operator, obtain the variable name of the inferred variable, infer the variable type of the inferred variable according to the variable inference type table, and add the variable type to the variable type relationship table to obtain the type list of the inferred variable on the node. When the node type is a module, class, or function definition, take the intersection of all elements in the type list of each specified variable, and when the intersection element is 1, update the type list of the specified variable using the intersection element. Until the traversal is completed, the variable type prediction result of the Python project is obtained based on the current variable type relationship table.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the method as described in any one of claims 1-7.

Images