Code index generation method, device, equipment and computer storage medium

Abstract

The present application discloses a code index generation method, device, equipment and computer storage medium, which relate to the field of code management technology. Aiming at scenarios such as massive enterprise warehouses with complex compilation environments, a full-scale accurate code navigation index generation method combining compiler front-end technology with a lightweight semantic symbol enhancement analysis mechanism is proposed. In this method, by performing syntax analysis and semantic analysis on source code, a definition and reference relationship graph is constructed, and symbol enhancement analysis is performed on files with identifier analysis errors to update the previously generated definition and reference relationship graph to obtain a more complete code index for the target project, which effectively solves the technical problem that a compiled language cannot generate a code navigation index in the case of a compilation error, and improves the recall rate of the generated code navigation index.

Description

Background Art

[0002] The code repository provides corresponding code query functions for the languages ​​it supports, such as definition, reference relationship query, variable declaration jump, function declaration jump, and class declaration jump. These functions rely on the code navigation index to be implemented. The navigation index refers to the metadata of the objects that can be indexed in the code. Therefore, accurate generation of code index is a prerequisite for realizing these functions.

[0003] In the related art, the generation of a navigation index usually involves calling the front-end analysis technology of the compiler of the corresponding language to perform grammatical analysis and semantic analysis on the source code to obtain definitions and reference relationships, and then using a graph to construct a definition and reference relationship graph for the entire project. For example, Java language (an object-oriented programming language) calls the Java compiler (Javacompiler) application programming interface (Application Programming Interface, API) for analysis, and C language or C++ language calls clang (a lightweight compiler for C language, C++, Objective-C language, etc.) for analysis.

[0004] However, the above methods all use the underlying compiler to obtain the definition and reference relationship of the code. However, in the scenario of massive code repositories, the compilation environment of the code is complicated, and it is impossible to simulate a large number of compilation environments. As a result, the way of calling the underlying compiler is subject to many restrictions. For example, the third-party dependency cannot be found, the symbol cannot be found in the conditional compilation, which will lead to the macro not being expanded and the definition being incomplete, and the function in clang is incompatible with the GCC function (GCC is the GNU Compiler Collection, a programming language compiler developed by GNU). The above problems will lead to errors in identifier parsing, and the definition and reference relationship of some codes cannot be obtained, so that the navigation index of the code repository cannot be established, or even if the code index can be established, a large number of erroneous identifiers cannot be parsed, so the code index established is also incomplete. Summary of the invention

[0005] The embodiments of the present application provide a code index generation method, apparatus, device and computer storage medium, which are used to solve the technical problem that a navigation index cannot be generated or is incompletely generated when a compiled language has a compilation error.

[0006] In one aspect, a method for generating a code index is provided, the method comprising:

[0007] Based on the syntax analysis results and semantic analysis results of the source code of the target project, a definition reference relationship graph corresponding to the target project is generated, wherein the definition reference relationship graph is used to indicate: a first reference set corresponding to each symbol definition, each first reference set including a referenced identifier corresponding to a symbol definition;

[0008] Determine at least one code file in the target project where identifier parsing errors occur, and perform definition reference relationship analysis on the abstract syntax tree corresponding to each of the at least one code file to obtain a plurality of reference relationship pairs, each reference relationship pair comprising: jump link information between the referenced identifier and the corresponding symbol definition in each identifier reference node;

[0009] Based on the multiple reference relationship pairs, the definition reference relationship graph is updated, and based on the updated definition reference relationship graph, a code index corresponding to the target project is obtained.

[0010] In one aspect, a code index generating device is provided, the device comprising:

[0011] A relationship graph generating unit is used to generate a definition reference relationship graph corresponding to the target project based on the syntax analysis result and semantic analysis result of the source code of the target project, wherein the definition reference relationship graph is used to indicate: a first reference set corresponding to each symbol definition, each first reference set including a referenced identifier corresponding to a symbol definition;

[0012] The enhanced analysis unit is used to determine at least one code file in the target project where the identifier parsing error occurs, and to perform definition reference relationship analysis on the abstract syntax tree corresponding to each of the at least one code file to obtain a plurality of reference relationship pairs, each of which includes: jump link information between the referenced identifier and the corresponding symbol definition in each identifier reference node;

[0013] The index generating unit is used to update the definition reference relationship graph based on the multiple reference relationship pairs, and obtain the code index corresponding to the target project based on the updated definition reference relationship graph.

[0014] Optionally, the enhanced analysis unit is specifically used to:

[0015] Respectively obtain the abstract syntax tree corresponding to each of the at least one code file;

[0016] Based on the identifier definition nodes included in the obtained at least one abstract syntax tree, obtain symbolic definitions of each identifier included in the at least one code file;

[0017] The plurality of reference relationship pairs are obtained based on a symbol definition corresponding to a reference identifier in an identifier reference node included in the at least one abstract syntax tree.

[0018] Optionally, the enhanced analysis unit is specifically used to:

[0019] For each identifier reference node, perform the following operations respectively:

[0020] For an identifier reference node, obtain the prefix content of the reference delimiter in the corresponding expression statement;

[0021] If the prefix content is a reference identifier, determining the prefix type of the prefix content based on the symbol type of the reference identifier;

[0022] From the identifiers corresponding to the prefix type, an identifier with the same name as the referenced identifier after the reference separator is searched, and a corresponding reference relationship pair is generated based on the referenced identifier and the symbol definition of the searched identifier.

[0023] Optionally, the enhanced analysis unit is specifically used to:

[0024] If the prefix content is an expression statement, iteratively obtain the prefix content of the reference delimiter in the prefix content until the obtained prefix content is a reference identifier;

[0025] Starting from the prefix content obtained last time, iteratively generate reference relationship pairs corresponding to the referenced identifiers corresponding to each prefix content until all reference relationship pairs corresponding to the referenced identifiers in the entire expression statement are generated.

[0026] Optionally, the enhanced analysis unit is specifically used to:

[0027] If the symbol type is a class, searching for an identifier with the same name as the reference identifier from the identifiers corresponding to the class included in each identifier, and determining the symbol name of the identifier found as the prefix type;

[0028] If the symbol type is a function, searching for an identifier with the same name as the reference identifier from the identifiers corresponding to the functions included in the identifiers, and determining the return type of the identifier found as the prefix type;

[0029] If the symbol type is a variable, an identifier with the same name as the reference identifier is searched from the identifiers corresponding to the variables included in the identifiers, and the definition type of the identifier found is determined as the prefix type.

[0030] Optionally, the index generating unit is specifically used to:

[0031] For each symbol definition in the multiple reference relationship pairs, perform the following operations respectively:

[0032] For a symbol definition, obtaining a second reference set corresponding to the symbol definition, wherein the second reference set includes referenced identifiers corresponding to each reference relationship pair of the symbol definition;

[0033] Compare the second reference set with the first reference set corresponding to the one symbol definition in the definition reference relationship graph to obtain a comparison result;

[0034] If the comparison result indicates that the second reference set has a difference set relative to the first reference set, the difference set is merged with the first reference set.

[0035] Optionally, the index generating unit is specifically used to:

[0036] For each definition node and each reference node in the updated definition reference relationship graph, respectively determine a corresponding indexable object and a corresponding index content;

[0037] According to the relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined;

[0038] Based on the obtained indexable objects and corresponding index contents, as well as the association relationship between the indexable objects, the code index is generated, and the code index includes a symbol information table composed of each indexable object and index content, and a definition reference relationship table indicating the association relationship between the indexable objects.

[0039] Optionally, the index generating unit is specifically used to:

[0040] For each definition node in the updated definition reference relationship graph, perform the following operations respectively:

[0041] For a definition node, a depth-first search method is used to determine a reference node from subordinate nodes of the definition node;

[0042] Based on the connection relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined.

[0043] Optionally, the index generating unit is specifically used to:

[0044] In the updated definition reference relationship graph, determining a query result node from an exit node of the one definition node;

[0045] Determining a reference set node from the exit nodes of the query result node;

[0046] A reference node of the one definition node is determined from associated items of the reference set node.

[0047] On the one hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0048] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0049] In one aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above methods.

[0050] In an embodiment of the present application, for at least one code file in the target project in which identifier parsing errors occur, definition reference relationship analysis is performed based on the respective corresponding abstract syntax trees to obtain multiple reference relationship pairs, each reference relationship pair containing jump link information between the referenced identifier and the corresponding symbol definition in each identifier reference node, and then the definition reference relationship graph obtained based on the syntax analysis results and the semantic analysis results is updated through the obtained multiple reference relationship pairs. In this way, when an identifier parsing error occurs, an enhanced analysis can be performed in combination with the abstract syntax trees of these code files to obtain a more complete code navigation index for the target project, thereby solving the problem of being unable to generate a code index or an incomplete index under compilation conditions and improving the recall rate of the generated code index. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0053] Figure 2 A flowchart of a code index generation method provided in an embodiment of the present application;

[0054] Figure 3Some examples of generated definition reference relationship diagrams provided in the embodiments of the present application;

[0055] Figure 4 A schematic diagram of the enhanced analysis process provided in the embodiment of the present application;

[0056] Figure 5 A schematic diagram of a sub-AST corresponding to a code snippet provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the process of citing analysis provided in the embodiment of the present application;

[0058] Figure 7 An example diagram of an application provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of a front-end query display provided in an embodiment of the present application;

[0060] Fig. 9 Another front-end query display schematic diagram provided in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of another front-end query display provided in an embodiment of the present application;

[0062] Fig.11 A schematic diagram of the structure of a code index generating device provided in an embodiment of the present application;

[0063] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0064] Fig.13 A schematic diagram of the structure of another computer device to which the embodiment of the present application is applied. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0066] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0067] Code index: When implementing functions such as definition and reference relationship query, it is implemented by relying on the index of indexable objects in the code file. The indexable objects here are specifically identifiers (symbol) in the code file, such as variables (var), classes (class), constants (const) or functions (func) in the code file. The index specifically refers to the metadata (metadata) of the indexable objects. Metadata is data that describes data, that is, data that describes the indexable objects. As an example, when the identifier is a definition identifier, the index content can specifically be the specific content of the symbol definition of the identifier; when the identifier is a referenced identifier, the index content can specifically be the specific content of the symbol definition of the definition identifier corresponding to the identifier, or the jump link information of the definition identifier, such as the code file and location information.

[0068] Abstract Syntax Tree (AST): or simply Syntax tree, is an abstract representation of the grammatical structure of the source code. It represents the grammatical structure of the programming language in a tree form, and each node on the tree represents a structure in the source code.

[0069] Identifier definition node, identifier reference node: Identifier definition node and identifier reference node are nodes in AST, which are used to represent structures in different source codes. Identifier definition node is used to indicate that the structure is specifically a defined identifier in the code file. Identifier definition node contains the type of the node, the corresponding identifier, and the symbolic definition of the identifier. Similarly, identifier reference node is used to indicate a structure with a reference relationship in the code file. For example, an expression statement "ExcerptAppender appender = queue.acquireAppender()" is a structure with a reference relationship. "queue" is a reference identifier, that is, the caller, and "acquireAppender()" is the referenced identifier of the function call, that is, the callee. Therefore, an identifier definition node can contain the type of the node, the reference relationship between the reference identifier and the referenced identifier, and other contents.

[0070] Symbol definition: Each identifier must and can only be defined in one source code file, but can be referenced or declared in multiple other source code files. When defining an identifier, not only is the existence of the identifier indicated, but storage is also allocated for it.

[0071] Reference delimiter: The reference delimiter can be used to indicate that a reference relationship exists. For example, for the above expression statement "ExcerptAppender appender = queue.acquireAppender()", "." is the reference delimiter, which is used to separate the caller and the callee. Of course, in actual applications, the reference delimiter is not limited to ".". There are other possible reference delimiters, such as "*", " / ", "—>", etc.

[0072] Expression statement: Expression is one of the most common basic grammatical units in program code. In AST, it is a tree structure: the root node is the last executed operator, and the operators closer to the leaf nodes are executed earlier. The bottom leaf nodes are variables or constants. Expression statements can be divided into arithmetic expressions, assignment expressions, comma expressions, relational expressions, logical expressions, etc. according to their types.

[0073] Definition reference relationship diagram: used to express the relationship between various entities in a code file or program project in a graphical way. In the embodiment of the present application, the entities involved are specifically definition identifiers, referenced identifiers, etc.

[0074] The following is a brief introduction to the design concept of the embodiment of the present application:

[0075] In the related technology, the generation of code index usually calls the front-end analysis technology of the compiler of the corresponding language to perform syntax analysis and semantic analysis on the source code to obtain the definition and reference relationship, and then uses a graph to construct the definition and reference relationship graph of the entire project. However, the method of calling the underlying compiler is subject to many restrictions, and there may be problems such as incomplete definition acquisition, which will cause identifier errors and fail to obtain the definition and reference relationship of some codes, thereby causing the navigation index of the code repository to be unable to be correctly established.

[0076] Based on this, an embodiment of the present application provides a code index generation method, in which, for at least one code file in the target project in which identifier parsing errors occur, definition reference relationship analysis is performed based on their respective corresponding abstract syntax trees to obtain multiple reference relationship pairs, each reference relationship pair containing jump link information between the referenced identifier and the corresponding symbol definition in each identifier reference node, and then the definition reference relationship graph obtained based on the grammatical analysis results and the semantic analysis results is updated through the obtained multiple reference relationship pairs. In this way, when an identifier parsing error occurs, an enhanced analysis can be performed in combination with the abstract syntax trees of these code files to obtain a more complete code index for the target project, thereby solving the problem of being unable to generate a code index when an identifier error occurs and improving the recall rate of the generated code index.

[0077] Specifically, for at least one determined code file, the corresponding abstract syntax trees are obtained respectively, so as to obtain the symbolic definition of each identifier contained in the at least one code file based on the identifier definition node contained in the obtained at least one abstract syntax tree, and obtain multiple reference relationship pairs based on the symbolic definition corresponding to the referenced identifier in the identifier reference node contained in the at least one abstract syntax tree, each reference relationship pair includes: jump link information between the referenced identifier and the corresponding symbolic definition in each identifier reference node. Through the above process, the symbolic definition of each identifier and the relationship between each referenced identifier and its application identifier can be accurately obtained from at least one code file, thereby assisting in creating a code index corresponding to at least one code file. It can be seen that this process does not need to rely on the underlying technology of the compiler, and improves the recall rate of the index generated by each identifier in the code file.

[0078] Considering that the query method based on the definition reference relationship graph is a layer-by-layer traversal process with low efficiency, in the embodiment of the present application, the definition reference relationship graph is used to extract the definition and reference relationship from the definition reference relationship graph through a depth-first search algorithm and store them in the database. Then, when querying, the query can be performed in parallel to improve the query efficiency.

[0079] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0080] The solution provided in the embodiment of the present application can be applied to most scenarios where code indexes need to be generated, especially to scenarios such as massive codes and code hosting platforms. Figure 1 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, it can include a terminal device 101, a code warehouse server 102 and a code warehouse database 103.

[0081] The terminal device 101 can be, for example, a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, a smart car device, and a smart wearable device. The terminal device 101 can be installed with a target application, and the target application can be used to display code, query code, etc. The application involved in the embodiment of the present application can be a software client, or a client such as a web page or a small program, and the code warehouse server 102 is a background server corresponding to the software or web page, small program, etc., without limiting the specific type of the client. For example, it can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms, but is not limited to this. The code warehouse database 103 is used to store source code. The code warehouse server 102 and the code warehouse database 103 belong to the same code storage system, which can be implemented by independent and different devices. In some cases, the code warehouse server 102 and the code warehouse database 103 can also be implemented by the same device.

[0082] In the embodiment of the present application, the target application on the terminal device 101 can upload the code file to request the code warehouse server 102 to store the code file, and the code warehouse server 102 can store the code file in the code warehouse database 103. At the same time, the code warehouse server 102 can also generate a code index corresponding to the code file or the program project where the code file is located based on the method of the embodiment of the present application, and store the code index in the code warehouse database 103. Therefore, when viewing the source code through the target application on the terminal device 101, code index-related functions such as definition query, reference relationship jump, etc. can be realized.

[0083] Taking the code warehouse server 102 executing the method steps provided in the embodiment of the present application as an example, the code warehouse server 102 may include one or more processors 1021, a memory 1022, and an I / O interface 1023 for interacting with other devices, etc. Among them, the memory 1022 of the code warehouse server 102 may also store program instructions of the code index generation method provided in the embodiment of the present application, and these program instructions can be used to implement the steps of the code index generation method provided in the embodiment of the present application when executed by the processor 1021 to generate a code index for the target project.

[0084] In the embodiment of the present application, each device may be directly or indirectly connected to each other through one or more networks 104. The network 104 may be a wired network or a wireless network, for example, a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, or other possible networks, which are not limited in the embodiment of the present invention.

[0085] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and code repository servers is unlimited and is not specifically limited in the embodiments of the present application.

[0086] In a possible application scenario, the code-related data in the embodiment of the present application, such as code indexes, can be stored using cloud storage technology. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (or storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0087] In a possible application scenario, in order to reduce communication latency, code warehouse servers 102 may be deployed in various regions, or different code warehouse servers 102 may serve the regions corresponding to each terminal device 101 for load balancing. Multiple code warehouse servers 102 share data through blockchain, which is equivalent to a data sharing system composed of multiple code warehouse servers 102. For example, if the terminal device 101 is located at location a, and its corresponding code warehouse server 102 is the first code warehouse server 102, then the first code warehouse server 102 provides code query and code index related services for the terminal device 101 at location a. If the terminal device 101 is located at location b, and its corresponding code warehouse server 102 is the second code warehouse server 102, then the second code warehouse server 102 provides recommendation services for the terminal device 101 at location b.

[0088] Each code warehouse server 102 in the data sharing system has a node identifier corresponding to the code warehouse server 102. Each code warehouse server 102 in the data sharing system can store the node identifiers of other code warehouse servers 102 in the data sharing system, so that the generated blocks can be broadcast to other code warehouse servers 102 in the data sharing system according to the node identifiers of other code warehouse servers 102. Each code warehouse server 102 can maintain a node identifier list as shown in the following table, and store the code warehouse server 102 name and node identifier in the node identifier list. Among them, the node identifier can be the Internet Protocol (IP) address of the interconnection protocol between networks and any other information that can be used to identify the node.

[0089] The following describes the code index generation method provided by the exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0090] See also Figure 2 As shown, it is a flow chart of the code index generation method provided in the embodiment of the present application, and the specific implementation process of the method is as follows:

[0091] Step 201: Perform syntax analysis and semantic analysis on the source code corresponding to the target project to obtain syntax analysis results and semantic analysis results corresponding to the target project respectively.

[0092] In an embodiment of the present application, compiler front-end analysis technology can be used to call the compiler native function corresponding to the target project to perform syntax and semantic analysis in order to facilitate the construction of definition and reference relationships.

[0093] In an embodiment of the present application, corresponding compiler native functions, such as clang, java compiler and other compiler native functions, can be selected according to the programming language used by the target project to obtain the syntax analysis results and semantic analysis results of the target project source code.

[0094] Step 202: Based on the syntax analysis results and semantic analysis results of the source code of the target project, generate a definition reference relationship graph corresponding to the target project, where the definition reference relationship graph is used to indicate: a first reference set corresponding to each symbol definition, each first reference set including a referenced identifier corresponding to a symbol definition.

[0095] Generally speaking, the process of analyzing the target project source code includes grammatical analysis and semantic analysis, so that the various identifiers contained in the target project, as well as the definition and reference relationship between the various identifiers can be known, and then, based on this, the corresponding definition reference relationship diagram can be constructed.

[0096] See also Figure 3 As shown in the following code, there is a definition of an identifier and two references to the identifier. Figure 2 As shown, it is a definition reference relationship between definition application relationship diagrams. Here, the lsif description is taken as an example. It can be seen that in the definition reference relationship diagram, there is a definition node corresponding to line 2 (line2) in the following code. The definition node describes the location of the identifier and the type of the node and other related information. For example, the tag indicates that type is "definition", which is a definition. At the same time, there are reference nodes corresponding to lines 16 and 19, respectively, which also describe the location of the reference and the type of the node and other related information.

[0097]

[0098] Step 203: Determine at least one code file in the target project where identifier parsing errors occur, and perform definition reference relationship analysis on the abstract syntax trees corresponding to each of the at least one code file to obtain a plurality of reference relationship pairs.

[0099] In the embodiment of the present application, when calling the compiler native function for grammatical and semantic analysis and constructing definition and reference relationship, there may be errors in identifier parsing. These errors may be due to the failure to download the dependency or the lack of environment variables, which leads to the inability to expand the macro, or the failure to analyze the definition in cross-files such as protbuf, which leads to a large number of symbols not being parsed, the symbol cannot be found in the conditional compilation, which leads to the failure to expand the macro, resulting in incomplete definition acquisition, or the incompatibility of the function in clang with the gcc function. In the case of an error in identifier parsing, the navigation index of many code warehouses of the enterprise cannot be established, or a large number of code symbols in the warehouse cannot establish the code navigation index. Therefore, for the above scenarios, the embodiment of the present application adds a symbol enhancement analysis mechanism. In the case of an identifier error, a secondary analysis is performed, the identifier table is updated, and the definition and reference index of the symbol error are generated again, and the symbol definition and reference index generated based on the compilation are combined to form a complete symbol index of the code warehouse, thereby improving the recall rate of the code index, and at the same time, the workload of symbol analysis can be reduced and the generation efficiency of the code index can be improved.

[0100] Specifically, after performing semantic analysis, the code files with identifier errors in the target project can be determined as at least one code file to be analyzed, so that symbol enhancement analysis can be performed on these code files through a subsequent analysis process. The analysis process will be described in detail later, so it will not be elaborated here.

[0101] Step 204: Based on the multiple reference relationship pairs, update the definition reference relationship graph.

[0102] Step 205: Based on the updated definition reference relationship graph, obtain the code index corresponding to the target project.

[0103] In an embodiment of the present application, after obtaining the corresponding code index through the symbol enhancement analysis process, it is necessary to integrate the definition reference relationship diagram obtained by the semantic analysis result and the code index obtained by the symbol enhancement analysis to obtain a more complete result.

[0104] Specifically, the definition reference relationship graph may be updated based on a code index corresponding to at least one code file.

[0105] In specific implementation, the obtained SRT can be analyzed to obtain the referenced identifier set corresponding to each symbol definition, and then compared with the referenced identifier set corresponding to the same symbol definition obtained previously. If the two combined contents are consistent, it indicates that the result obtained by the compiler is not wrong. If they are different, the two sets corresponding to the same symbol definition are merged to obtain a new reference result set as the reference result set corresponding to the final symbol definition.

[0106] Since the process of each symbol definition is similar, here we take one symbol definition A as an example. One symbol definition corresponds to a definition identifier, and in the definition reference relationship graph it corresponds to a definition node. When fusion is performed, the second reference set corresponding to the symbol definition A is obtained from the obtained code index. The second reference set contains the referenced identifiers corresponding to each reference relationship pair where the symbol definition A is located. At the same time, the first reference set corresponding to the symbol definition A is obtained from the definition reference relationship graph, and the first reference set is compared with the second reference set to obtain a comparison result. If the comparison result indicates that the second reference set has a difference set relative to the first reference set, that is, the symbol enhancement analysis is performed to obtain a new definition reference relationship, then the difference set and the first reference set can be merged to obtain the final reference result set of the symbol definition A, and it can be updated to the definition reference relationship graph.

[0107] Then, the code index corresponding to the target project can be obtained based on the updated definition reference relationship graph. In the embodiment of the present application, the obtained definition reference relationship graph can actually be used as a code index for querying the index content in subsequent use. For example, when it is necessary to query the definition identifier corresponding to a referenced identifier, the reference node corresponding to the referenced identifier can be found in the definition reference relationship graph, and then starting from the reference node, the exit node of the reference node is traversed until the definition node of the corresponding definition identifier is found.

[0108] In a possible implementation, the process of step 203 may be performed as follows: Figure 4 The method shown is implemented as follows, Figure 4 A flowchart of the code file enhanced analysis process provided in an embodiment of the present application.

[0109] Step 2031: Obtain the abstract syntax tree corresponding to at least one code file to be analyzed in the target project respectively.

[0110] In an embodiment of the present application, the at least one code file to be analyzed may be one or more files in the target project, for example, one or more specified files, or all files in the target project, or one or more files obtained based on conditional screening, such as a code file with an error in the above-mentioned identifier or at least one code file that failed to compile.

[0111] Among them, the abstract syntax tree is an abstract representation of the syntax structure of the source code, so the abstract syntax tree can be obtained by performing syntax analysis on at least one code file. In actual application, the syntax analysis process can be performed by code parsers corresponding to various programming languages, that is, the code file is input into the corresponding code parser to obtain the AST to be output by the code parser. For example, when the programming language is Java, Java compiler, JDT (Java Development Tools, a full-featured Java integrated development environment plug-in for the Eclipse platform), Javalang, Java Parser and Eclipse Ast Parser can be used to implement syntax analysis. When the programming language is C / C++, CDT (C / C++ Development Tools), pycparser and Joern can be called to implement syntax analysis. When the programming language is python, ast library and tokenize library can be called to implement syntax analysis. Of course, there are also some general syntax analysis tools, such as ANTLR, SonarQube and tree-sitter, which provide APIs for different programming languages. When these tools need to be used for syntax analysis, the corresponding API can be called based on the current programming language.

[0112] In an embodiment of the present application, considering that syntax analysis is an essential step when compiling a code file, if the target project has been compiled, then the AST of each code file has been constructed and generated, and the AST corresponding to each of the above at least one code file can be obtained from the existing AST.

[0113] Here we take the code snippet in the code file as an example to introduce the structure of AST. See the following code snippet, which is a code snippet of the function definition in the line number range of 121 to 126 in the code file:

[0114]

[0115] The sub-AST corresponding to this code snippet can be found in Figure 5As shown, in this sub-AST, it can be seen that its type is "MethodDeclaration", which is specifically a function definition node, and it can be known that the modifier of the function is "private", the function name is "accepMethod", the return value type is "boolean", the function parameter is "IMethodmethod", and the function body is "body". The function body also contains multiple statements, represented as "Statements". The Statements node contains IfStatement statements and ReturnSatement, and the IfStatement statement contains Expression and thenStatement. It can be seen that the obtained AST is a hierarchical tree from the root node to the leaf node. Generally speaking, the root node of the code file is the compilation unit (Compilation Unit), and the nodes of each level contain the semantic information of the code.

[0116] Step 2032: Based on the identifier definition nodes included in the obtained at least one abstract syntax tree, obtain the symbolic definition of each identifier included in the at least one code file.

[0117] In the embodiment of the present application, after obtaining the AST of each code file respectively, each AST can be analyzed to obtain the symbolic definition of each identifier from the AST. It should be noted that since the analysis process of each AST is similar, an AST is used as an example for introduction here.

[0118] In one possible implementation, a traversal method may be used to traverse the nodes in the AST one by one. If the type of the currently traversed node is an identifier definition node, the symbol definition of the corresponding identifier is obtained from the identifier definition node to collect the symbol definition of each identifier in the entire AST.

[0119] In a possible implementation, AST commands can also be used to find qualified nodes from AST. In this step, the identifier definition nodes are specifically screened. For example, the "visit" command can be used to select nodes of the required type. For example, if you want to operate a function definition, you can use "visitFunctionDelaration" to traverse the function definition in AST, or if you want to operate an assignment expression, you can use "visitExpressionStatement" to traverse the assignment expression in AST. Of course, this is just an example of processing a command, and does not limit the actual command used.

[0120] In an embodiment of the present application, in order to facilitate the storage of symbolic definitions of identifiers of all code files, the symbolic definitions of identifiers obtained from the AST of each code file may be integrated and summarized for subsequent use.

[0121] Specifically, in the embodiment of the present application, the definition information of the symbol obtained from the AST may include one or more combinations of namespaces, packages, classes, Field variables, functions, and import statements.

[0122] In a possible implementation, the symbol definition of each identifier may be stored in a table, that is, the symbol definition information of the identifier obtained from all code files is stored in a table, which is called the symbol definition table (SDT) of the target project. The symbol definition table may include one or more combinations of the following fields:

[0123] (1)Symbol name (symbolName): that is, the simple name (simpleName) of the identifier.

[0124] (2) Position: used to describe the position of the identifier in the code file, including the start line (startLine), end line (endLine) and offset (offset).

[0125] (3) Identifier type: The symbol type is an enumeration type that includes packages, classes, methods, variables, etc.

[0126] (4) Namespace of an identifier: used to indicate the code context in which the identifier is located. For example, the namespace of a function in Java is expressed as "package name + class name".

[0127] (5) Identifier description: Descriptive information related to a specific identifier. For example, when the identifier is a function, its description may include the function name, return value type, function parameters, etc. When the identifier is a variable, its description may include whether it is a static variable, whether it is a local variable, etc.

[0128] (6) The code file to which the identifier belongs (filePath): that is, the path of the code file to which the identifier belongs.

[0129] Step 2033: Based on the symbolic definition corresponding to the referenced identifier in the identifier reference node contained in at least one abstract syntax tree, multiple reference relationship pairs are obtained, each reference relationship pair comprising: jump link information between the referenced identifier and the corresponding symbolic definition in each identifier reference node.

[0130] In the embodiment of the present application, considering that an identifier can only be defined once in a code file, it may still be defined in other code files. Therefore, in order to prevent the occurrence of duplicate names in the project, it is necessary to accurately find the definition of each referenced identifier. In order to obtain the definition and reference relationship, it is also necessary to analyze the reference relationship in the code file, and the analysis process can also be based on AST. Therefore, the above-mentioned method for obtaining AST can also be used to obtain the corresponding AST, or the AST obtained by the above steps can also be used. Similarly, since the analysis process of each AST is similar, an AST is introduced here as an example.

[0131] Specifically, for each AST, extract the identifier reference node in the AST and determine the referenced identifier it contains. Each identifier reference node contains a reference identifier and a referenced identifier that have a reference relationship. Then analyze the referer of each referenced identifier, that is, analyze the reference identifier, infer the type of the reference identifier, and obtain the definition of the referenced identifier. See the following code snippet example, where "acquireAppender()" is a referenced identifier of a function call.

[0132]

[0133]

[0134] In a possible implementation, the following may be used: Figure 6 The process of citation analysis is Figure 6 This is a flowchart of analyzing the referrer for an identifier reference node. Since the analysis process of each identifier reference node is similar, an identifier reference node is specifically used as an example for description.

[0135] Step 20331: For the current identifier reference node, obtain the prefix content of the reference delimiter in the corresponding expression statement.

[0136] Specifically, if the expression statement contains only one application delimiter, the prefix content before the reference delimiter is directly obtained; if the expression statement contains only multiple application delimiters, the prefix content before the last reference delimiter is obtained.

[0137] In the embodiment of the present application, the reference delimiters used in different programming languages ​​may be different, so the determination can be made based on the reference delimiters used in the actual programming language. For example, the reference delimiter may be symbols such as “.”, “—>” or “ / ”.

[0138] Taking the above code snippet as an example, in “ExcerptAppender appender = queue.acquireAppender()”, “.” is a reference delimiter, “acquireAppender()” is a referenced identifier, “queue” is a reference identifier, and the prefix content before “.” is “queue”.

[0139] Step 20332: Determine the type of the prefix content, that is, determine whether it is an expression.

[0140] Step 20333: If the judgment result of step 2032 indicates that the prefix content is a reference identifier, the prefix type of the prefix content is determined based on the symbol type of the reference identifier.

[0141] In the embodiment of the present application, when the symbol type of the reference identifier is different, the prefix type should be different accordingly, so it is necessary to judge the symbol type of the reference identifier. In practical application, it may include but is not limited to the following situations:

[0142] (1) If the symbol type is a class, then search for an identifier with the same name as the referenced identifier from the identifiers corresponding to the class included in each identifier, and determine the symbol name of the identifier found as the prefix type.

[0143] Specifically, when the identifiers are aggregated into an SDT, when the symbol type is a class, the name of the class is searched from the constructed SDT, and if found, the symbol name of the class is used as the prefix type corresponding to the prefix content.

[0144] (2) If the symbol type is a function, then search for an identifier with the same name as the referenced identifier from the identifiers corresponding to the functions included in each identifier, and determine the return type of the identifier found as the prefix type.

[0145] Specifically, when the identifiers are aggregated into an SDT, when the symbol type is a function, the name of the function is searched from the constructed SDT, and if the name is found, the return type of the function is used as the prefix type corresponding to the prefix content.

[0146] (3) If the symbol type is a variable, then from the identifiers corresponding to the variables included in each identifier, search for an identifier with the same name as the referenced identifier, and determine the definition type of the identifier found as the prefix type.

[0147] Specifically, when the identifiers are aggregated into an SDT, when the symbol type is a variable, the name of the variable is searched from the constructed SDT, and if the name is found, the definition type of the variable is used as the prefix type corresponding to the prefix content.

[0148] Step 20334: From the identifiers corresponding to the prefix type, search for an identifier with the same name as the referenced identifier after the reference delimiter, and generate a corresponding reference relationship pair based on the referenced identifier and the symbol definition of the found identifier.

[0149] Specifically, the name of the current referenced identifier in the prefix type is searched in the SDT. If it can be found, the referenced identifier after the reference delimiter is used as the reference of the symbol, and the identifier found in the SDT is used as the definition to form a reference relationship pair. The composition structure of the pair can be <symbol reference, definition>.

[0150] Among them, the symbol reference part can include the referenced identifier and the position of the referenced identifier, and the definition part can include the identifier obtained by the above search and the corresponding position. The position can generally include information such as the starting line number, the ending line number, and the offset. In this way, according to the pair, the position of a referenced identifier and its corresponding definition identifier can be known, and the position of a definition identifier and its corresponding referenced identifier can also be known, thereby facilitating the rapid jump between the two.

[0151] Taking "ExcerptAppender appender = queue.acquireAppender()" in the above code snippet as an example, in order to accurately find the definition of the acquireAppender() function and prevent duplicate names in the project, it is necessary to analyze the reference queue. queue is a variable, so it is necessary to find queue from the identifier belonging to the variable in the SDT. If it can be found, the type of the queue variable ExcerptAppender is used as the prefix type, and the function with the symbol name acquireAppender in the ExcerptAppender class is searched. If it can be found, it is used as the definition of the acquireAppender() function.

[0152] Step 20335: If the prefix content is an expression statement, obtain the prefix content of the previous reference delimiter and return to execute step 2032.

[0153] Specifically, when the prefix content is an expression statement, the prefix content of the reference delimiter in the prefix content is obtained in an iterative manner until the obtained prefix content is a reference identifier. This process is a recursive calling process.

[0154] For example, if the expression statement is "ABCD", then when performing the referrer analysis, it will start from the last ".", that is, obtain the prefix content "ABC", and if it is still an expression, continue to obtain the prefix content "AB" of the previous "." until the prefix content "A" of the first "." is obtained.

[0155] When generating a pair, starting from the last obtained prefix content, iteratively generates reference relationship pairs corresponding to the referenced identifiers corresponding to each prefix content until all reference relationship pairs corresponding to the referenced identifiers in the entire expression statement are generated.

[0156] Taking the above "ABCD" as an example, "A" will be analyzed first. If its corresponding prefix type contains "B", a pair (B, definition of B) will be formed. Then "B" will be analyzed. If its corresponding prefix type contains "C", a pair (C, definition of C) will be formed. The same method is used to generate a pair (D, definition of D).

[0157] For each identifier reference node, the above method can be used to obtain the corresponding pair until the pair <symbol reference, definition> of the entire target project is collected to construct the symbol reference table (SRT, Symbol Reference Table) of the target project.

[0158] In an embodiment of the present application, multiple indexable objects can be extracted based on the acquired SDT and SRT, each indexable object can be a definition identifier, a reference identifier or a referenced identifier, and the index content of each indexable object can be set according to its type. For example, when the indexable object is a definition identifier, its index content can be its symbol definition and the position of the corresponding referenced identifier. When the indexable object is a referenced identifier, its index content can be the position of the corresponding definition identifier.

[0159] In an embodiment of the present application, considering that the efficiency of the process of traversing the definition reference relationship graph for query is limited, the front-end display may not be timely. Therefore, in order to improve the real-time performance of the front-end definition reference relationship query, in an embodiment of the present application, after obtaining the updated definition reference relationship graph, the definition reference relationship graph can be traversed in advance and the definition reference relationship can be stored. When needed, it can be conveniently queried to improve query efficiency.

[0160] Specifically, for each definition node and each reference node in the updated definition reference relationship graph, the corresponding indexable object and the corresponding index content are determined respectively, wherein each indexable object is a definition identifier, a reference identifier or a referenced identifier, and the index content is the relevant information of the symbol, and then, according to each reference node, and the connection relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined, so as to generate a code index based on the obtained indexable objects and the corresponding index content, as well as the association relationship between the indexable objects, and the code index includes a symbol information table consisting of each indexable object and the index content, and a definition reference relationship table indicating the association relationship between the indexable objects.

[0161] In the embodiment of the present application, a depth first search (DFS) algorithm may be used to analyze the definition reference relationship graph finally generated to obtain all referenced identifiers of each definition identifier.

[0162] Specifically, for any definition node in the updated definition reference relationship graph, a depth-first search method is used to determine the reference node from the subordinate nodes of the definition node, thereby determining the association relationship between the corresponding indexable objects based on the symbol definition corresponding to the referenced identifier included in each reference node. For example, if the reference node B can be found according to the definition node A, there is an association relationship between the definition identifier corresponding to the definition node A and the referenced identifier corresponding to the reference node B.

[0163] In one possible implementation, a reference node is determined from the subordinate nodes of the definition node, specifically including determining a query result node from the exit node of the definition node in an updated definition reference relationship graph, and determining a reference collection node from the exit node of the query result node, and then determining a reference node of a definition node from the associated items of the reference collection node.

[0164] by Figure 3 Take the definition reference relationship diagram shown in the figure as an example. When searching for symbol references in the definition reference relationship diagram using depth-first search, see Figure 3 The path shown in bold in the figure is as follows: for each Range node whose type is defined, find its next outgoing edge. If the target node is a resultSet node, get the item outgoing edge whose target node is referenceResult. Then find all the target nodes whose outgoing edges are item as the reference nodes corresponding to the symbol.

[0165] In order to save storage space, the definition reference relationship pairs obtained above can be stored using ES (Elastic Search). Specifically, the following data structure can be used to simplify the code index. That is, in actual application, the information of each identifier and the definition reference relationship can be stored in a table. That is, the code index includes a symbol information table consisting of each indexable object and index content, and a definition reference relationship table indicating the definition relationship or reference relationship between indexable objects.

[0166] Specifically, the symbol information table may include one or a combination of the following information:

[0167] (1) Symbol ID: a unique primary key representing an identifier in the symbol information table.

[0168] (2)Symbol Name (symbolName)

[0169] (3) Symbol Category, which can include categories such as defined identifiers and referenced identifiers.

[0170] (4) Identifier type (symbolType), which can include function, variable, class, interface, package and other types.

[0171] (5) The context to which the symbol belongs (symbolContext), for example, the identifier is in a certain package or a certain function of a certain class.

[0172] (6) Position, such as the starting line (startLine) of the symbol, the ending line (endLine) of the symbol, and the offset (offset) of the symbol.

[0173] (7) The file path (filePath) of the symbol.

[0174] (8) The project ID (projectID) to which the symbol belongs.

[0175] Of course, other information besides the above information may also be included, and the embodiments of the present application are not limited to this.

[0176] Specifically, the definition reference relationship table may include one or a combination of the following information:

[0177] (1) CommitID, which is used to identify each commit ID and represents a unique primary key. In an actual enterprise code repository scenario, all code repositories of the enterprise need to be stored, and each repository may contain multiple branches and multiple commits.

[0178] (2)branchName, that is, the branch name.

[0179] (3)relatedSymbolIDA, the ID of the first symbol.

[0180] (4) relatedSymbolIDB, the symbol ID that is associated with the first symbol. The association here mainly defines the reference relationship.

[0181] The definition reference relationship table establishes the association between the definition identifier and the referenced identifier in the symbol information table. When the front end needs to query the association between the definition identifier and the referenced identifier, it can be queried through the above two tables. In addition, by introducing the commitID field to distinguish symbols in different commits, the same symbol information in different commits can be reused, thereby reducing the size of the symbol information table in ES.

[0182] See also Figure 7 As shown, it is an application example diagram of an embodiment of the present application. Specifically, after the front end submits the code, the code warehouse server at the back end generates or updates the code index for it, that is, it includes the above-mentioned two core tables, the symbol information table and the definition reference information table. In this way, when the front end needs to query the definition and reference relationship, for example, when the user selects a certain identifier when viewing the code, or operates a certain query function button, it triggers the query of the definition reference relationship to the code warehouse server. The code warehouse server searches the above-mentioned two core tables, queries the definition and reference relationship of the identifier, and returns it to the front end for display, for example Figure 7 The location of the definition and the specific content of the definition are displayed.

[0183] The following shows examples of code indexes constructed using the method of the embodiments of the present application when problems occur in several scenarios.

[0184] (1) The missing link variable causes the macro to fail to expand, resulting in a compilation error

[0185] See the following code snippet of macro definition. When the WIN32 environment variable is not found, the compiler will only parse the symbols in the else part, while the symbols in the if part will be lost. Therefore, when the related art method calls the macro grn_getenv(), it can only find the macro defined in the else part, and will lose the macro defined in the if part. That is, it can only find the macro definition of line 56, which makes it impossible to establish the relevant code index for the identifier in the if part. Figure 8 As shown, it is a front-end query display schematic diagram provided by an embodiment of the present application. When the method of the embodiment of the present application is adopted, the macro grn_getenv() defined in two places can be found. That is, when the program code macro grn_getenv() is selected, the location of its definition can be viewed, such as Figure 8As shown, there are two definitions, one is on line 44 of the code file "storage / mroonga / vendor / groonga / include / groonga / portability.h", which is "#define grn getenv(name,dest,dest size)do{char*dest=(dest)", and the other is on line 56 of the code file, which is "#define grn getenv(name,dest,dest size)do{const char*env_value=getenv((name));" In this way, you can clearly perceive the location of the definition, and selecting one of the definitions can also achieve quick jump.

[0186] Code link: storage / mroonga / vendor / groonga / include / groonga / portability.h

[0187]

[0188]

[0189] (2) The definitions in cross-files such as protbuf are not analyzed, resulting in a large number of identifiers not being resolved, leading to compilation errors

[0190] Refer to the following code snippets 1 and 2. The ExportTraceServiceRequest definition of ExportTraceServiceRequest*Req in code snippet 2 is located in the cross-file code snippet 1. Because the clang compiler of this file does not analyze the cross-file definitions such as protbuf, the ExportTraceServiceRequest in the following code snippet 2 is not parsed, which leads to a parsing error of the rpc_span_fill_pb_request function in the following code snippet 2. Then, there are problems with the functions that reference rpc_span_fill_pb_request. In other words, the definition cannot be found at the reference location of rpc_span_fill_pb_request using the relevant technology.

[0191] Code snippet 1 (protbuf identifier definition):

[0192]

[0193] Code snippet 2 (including identifier references to protbuf):

[0194] 17rpc_span_fill_pb_request(const std::string&service_name.

[0195] 18 const std::unordered map<std::string,std::string> & attributes,

[0196] 19 ExportTraceServiceRequest*req)

[0197] See also Fig. 9 As shown, it is another front-end query display schematic diagram provided by the embodiment of the present application. When the method of the embodiment of the present application is adopted, the definition of the rpc_span_fill_pb_request function can be found at the reference position of rpc_span_fill_pb_request, such as Fig. 9 As shown, its definition is in the file srpc / src / module / rpc_filter_span.cc. It can be seen that the method of the embodiment of the present application can accurately find the definition of the function call.

[0198] (3) The identifier cannot be found because the third-party dependency is not downloaded

[0199] The following is a code snippet of the application of the third-party package. The file is org.eclipse.jdt.codemining / src / org / eclipse / jdt / experimental / ui / javaeditor / codemining / methods / JavaMethodParameterCodeMining.java. The code is as follows:

[0200]

[0201] Because the download of the third-party package for this project may fail, such as due to the company's intranet proxy, network problems, or the package version is low and has been removed, the symbol CompilationUnit in the file cannot be found, and the compilation will report an error, causing the reference and definition of the variable cu declared by CompilationUnit to report an error. However, after using the method provided in the embodiment of the application for symbol enhancement analysis, the definition and reference of cu can be found, see Fig.10As shown, it is another front-end query display schematic diagram provided by the embodiment of the present application. When the method of the embodiment of the present application is used, the definition of cu can be found in line 77 of the above code file, namely "Compilationllnit cu ICodeMiningProvider provider, boolean showName, boolean showType," It can be seen that the method of the embodiment of the present application can accurately find the definition of the identifier in the tripartite package.

[0202] In summary, the embodiment of the present application proposes a method for generating a full-scale accurate code navigation index that combines compiler front-end technology with a lightweight semantic symbol enhancement analysis mechanism for scenarios such as complex compilation environments such as massive warehouses of enterprises. In this method, the compiler front-end analysis technology is used for the source code, and the compiler native function is called for syntax and semantic analysis. Then, the definition and reference relationship graph is constructed, and the symbol enhancement analysis is performed on the file with the identifier error, and the definition and reference relationship graph generated before is updated. Finally, the definition and reference relationship are extracted from the graph and stored in ES for front-end query. It can effectively solve the problem that the symbols of compiled languages ​​(C / C++, Java, Go, etc.) cannot generate code navigation indexes in the case of compilation errors. By introducing a symbol enhancement mechanism, it is ensured that the massive code of the enterprise can establish a symbol navigation index, thereby improving the recall rate. At present, this method has been used on the actual code hosting platform, and an enterprise 5W+ code navigation index has been established, supporting compiled languages ​​such as C / C++, Java, and Go.

[0203] See also Fig.11 Based on the same inventive concept, the embodiment of the present application further provides a code index generating device 110, which includes:

[0204] The relationship graph generating unit 1101 is used to generate a definition reference relationship graph corresponding to the target project based on the syntax analysis result and semantic analysis result of the source code of the target project, wherein the definition reference relationship graph is used to indicate: a first reference set corresponding to each symbol definition, each first reference set including a referenced identifier corresponding to a symbol definition;

[0205] The enhanced analysis unit 1102 is used to determine at least one code file in the target project where the identifier parsing error occurs, and perform definition reference relationship analysis on the abstract syntax tree corresponding to each of the at least one code file to obtain a plurality of reference relationship pairs, each of which includes: jump link information between the referenced identifier and the corresponding symbol definition in each identifier reference node;

[0206] The index generating unit 1103 is used to update the definition reference relationship graph based on multiple reference relationship pairs, and obtain the code index corresponding to the target project based on the updated definition reference relationship graph.

[0207] Optionally, the enhanced analysis unit 1102 is specifically configured to:

[0208] Obtaining the abstract syntax tree corresponding to at least one code file respectively;

[0209] Based on the identifier definition nodes included in the obtained at least one abstract syntax tree, obtain symbolic definitions of each identifier included in at least one code file;

[0210] Based on the symbol definition corresponding to the reference identifier in the identifier reference node contained in at least one abstract syntax tree, a plurality of reference relationship pairs are obtained.

[0211] Optionally, the enhanced analysis unit 1102 is specifically configured to:

[0212] For each identifier reference node, perform the following operations respectively:

[0213] For an identifier reference node, obtain the prefix content of the reference delimiter in the corresponding expression statement;

[0214] If the prefix content is a reference identifier, the prefix type of the prefix content is determined based on the symbol type of the reference identifier;

[0215] From the identifiers corresponding to the prefix type, find an identifier with the same name as the referenced identifier after the reference delimiter, and generate a corresponding reference relationship pair based on the referenced identifier and the symbol definition of the found identifier.

[0216] Optionally, the enhanced analysis unit 1102 is specifically configured to:

[0217] If the prefix content is an expression statement, iteratively obtain the prefix content of the reference delimiter in the prefix content until the obtained prefix content is a reference identifier;

[0218] Starting from the prefix content obtained last time, iteratively generate reference relationship pairs corresponding to the referenced identifiers corresponding to each prefix content until all reference relationship pairs corresponding to the referenced identifiers in the entire expression statement are generated.

[0219] Optionally, the enhanced analysis unit 1102 is specifically configured to:

[0220] If the symbol type is a class, then search for an identifier with the same name as the referenced identifier from the identifiers corresponding to the class included in each identifier, and determine the symbol name of the identifier found as the prefix type;

[0221] If the symbol type is a function, then search for an identifier with the same name as the referenced identifier from the identifiers corresponding to the functions included in each identifier, and determine the return type of the identifier found as the prefix type;

[0222] If the symbol type is a variable, an identifier with the same name as the referenced identifier is searched from the identifiers corresponding to the variables included in each identifier, and the definition type of the found identifier is determined as the prefix type.

[0223] Optionally, the index generating unit 1103 is specifically configured to:

[0224] For each symbol definition in multiple reference relationship pairs, perform the following operations respectively:

[0225] For a symbol definition, a second reference set corresponding to the symbol definition is obtained, wherein the second reference set includes referenced identifiers corresponding to each reference relationship pair of the symbol definition;

[0226] Compare the second reference set with the first reference set corresponding to a symbol definition in the definition reference relationship graph to obtain a comparison result;

[0227] If the comparison result indicates that the second reference set has a difference set relative to the first reference set, the difference set is merged with the first reference set.

[0228] Optionally, the index generating unit 1103 is specifically configured to:

[0229] For each definition node and each reference node in the updated definition reference relationship graph, respectively determine a corresponding indexable object and a corresponding index content;

[0230] According to the relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined;

[0231] Based on the obtained indexable objects and corresponding index contents, as well as the association relationship between the indexable objects, a code index is generated, which includes a symbol information table consisting of each indexable object and index content, and a definition reference relationship table indicating the association relationship between the indexable objects.

[0232] Optionally, the index generating unit 1103 is specifically configured to:

[0233] For each definition node in the updated definition reference relationship graph, perform the following operations respectively:

[0234] For a definition node, a depth-first search method is used to determine the reference node from the subordinate nodes of a definition node;

[0235] Based on the connection relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined.

[0236] Optionally, the index generating unit 1103 is specifically configured to:

[0237] In the updated definition reference relationship graph, a query result node is determined from an exit node of a definition node;

[0238] Determine the reference set node from the exit node of the query result node;

[0239] From the associated items of the reference collection node, determine a reference node that defines the node.

[0240] Through the above device, the compiler's front-end analysis of the source code can be performed, and then the definition and reference relationship graph can be constructed, and the symbol enhancement analysis can be performed on the files with symbol errors, and the previously generated definition and reference relationship graph can be updated. Finally, the definition and reference relationship are extracted from the graph and stored in ES for front-end query. This can effectively solve the problem that symbols in compiled languages ​​​​(C / C++, Java, Go, etc.) cannot generate code navigation indexes in the case of compilation errors. By introducing the symbol enhancement mechanism, it ensures that the company's massive code can establish symbol navigation indexes, thereby improving the recall rate.

[0241] For the convenience of description, the above parts are divided into various unit modules (or modules) according to their functions and described separately. Of course, when implementing this application, the functions of each unit (or module) can be implemented in the same or multiple software or hardware.

[0242] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0243] The device can be used to execute the methods shown in the various embodiments of the present application. Therefore, for the functions that can be implemented by the various functional modules of the device, reference can be made to the description of the aforementioned embodiments and no further details will be given.

[0244] See also Fig.12 Based on the same technical concept, the present application also provides a computer device. In one embodiment, the computer device can be Figure 1 The code repository server shown in the figure is a computer device such as Fig.12 As shown, it includes a memory 1201 , a communication module 1203 and one or more processors 1202 .

[0245] The memory 1201 is used to store computer programs executed by the processor 1202. The memory 1201 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0246] The memory 1201 may be a volatile memory, such as a random-access memory (RAM); the memory 1201 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1201 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1201 may be a combination of the above memories.

[0247] The processor 1202 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1202 is used to implement the above code index generation method when calling the computer program stored in the memory 1201 .

[0248] The communication module 1203 is used to communicate with terminal devices and other servers.

[0249] The specific connection medium between the memory 1201, the communication module 1203 and the processor 1202 is not limited in the embodiment of the present application. Fig.12 In the embodiment, the memory 1201 and the processor 1202 are connected via a bus 1204. The bus 1204 is connected to the processor 1202 via a bus 1204. Fig.12 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1204 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.12 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0250] The memory 1201 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the code index generation method of the embodiment of the present application. The processor 1202 is used to execute the code index generation method of each of the above embodiments.

[0251] In another embodiment, the computer device may also be other computer devices, such as Figure 1 In this embodiment, the structure of the computer device can be as follows: Fig.13 As shown, it includes: a communication component 1310, a memory 1320, a display unit 1330, a camera 1340, a sensor 1350, an audio circuit 1360, a Bluetooth module 1370, a processor 1380 and other components.

[0252] The communication component 1310 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The computer device can help the user to send and receive information through the WiFi module.

[0253] The memory 1320 can be used to store software programs and data. The processor 1380 executes various functions and data processing of the terminal device by running the software programs or data stored in the memory 1320. The memory 1320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The memory 1320 stores an operating system that enables the terminal device to run. In the present application, the memory 1320 can store an operating system and various application programs, and can also store code for executing the code index generation method of the embodiment of the present application.

[0254] The display unit 1330 can also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device. Specifically, the display unit 1330 may include a display screen 1332 disposed on the front of the terminal device. The display screen 1332 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 1330 may be used to display various code-related display pages in the embodiments of the present application.

[0255] The display unit 1330 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device. Specifically, the display unit 1330 may include a touch screen 1331 arranged on the front of the terminal device, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0256] The touch screen 1331 can be covered on the display screen 1332, or the touch screen 1331 and the display screen 1332 can be integrated to realize the input and output functions of the terminal device, and the integrated touch screen can be referred to as a touch display screen. In this application, the display unit 1330 can display the application and the corresponding operation steps.

[0257] The camera 1340 can be used to capture still images, and users can post comments on the images taken by the camera 1340 through the application. The camera 1340 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the processor 1380 to convert it into a digital image signal.

[0258] The terminal device may further include at least one sensor 1350, such as an acceleration sensor 1351, a distance sensor 1352, a fingerprint sensor 1353, and a temperature sensor 1354. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0259] The audio circuit 1360, the speaker 1361, and the microphone 1362 can provide an audio interface between the user and the terminal device. The audio circuit 1360 can transmit the electrical signal converted from the received audio data to the speaker 1361, which is converted into a sound signal for output. The terminal device can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1362 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1360 and converted into audio data, and then the audio data is output to the communication component 1310 to be sent to, for example, another terminal device, or the audio data is output to the memory 1320 for further processing.

[0260] The Bluetooth module 1370 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable computer device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1370 to exchange data.

[0261] The processor 1380 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1320 and calling data stored in the memory 1320. In some embodiments, the processor 1380 may include one or more processing units; the processor 1380 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1380. In the present application, the processor 1380 can run the operating system, application programs, user interface display and touch response, as well as the code index generation method of the embodiment of the present application. In addition, the processor 1380 is coupled to the display unit 1330.

[0262] In some possible implementations, various aspects of the code index generation method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the code index generation method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the steps of each embodiment.

[0263] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0264] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited thereto. In the present application, the readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with a command execution system, device or device.

[0265] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0266] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0267] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user equipment, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0268] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0269] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0270] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0271] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0272] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A code index generation method, characterized in that: The method comprises: Based on the syntax analysis results and semantic analysis results of the source code of the target project, a definition reference relationship graph corresponding to the target project is generated, wherein the definition reference relationship graph is used to indicate: a first reference set corresponding to each symbol definition, each first reference set including a referenced identifier corresponding to a symbol definition; Determine at least one code file in the target project where identifier parsing errors occur, and perform definition reference relationship analysis on the abstract syntax tree corresponding to each of the at least one code file to obtain a plurality of reference relationship pairs, each reference relationship pair comprising: jump link information between the referenced identifier and the corresponding symbol definition in each identifier reference node; Based on the multiple reference relationship pairs, the definition reference relationship graph is updated, and based on the updated definition reference relationship graph, a code index corresponding to the target project is obtained.

2. The method according to claim 1, characterized in that Performing definition reference relationship analysis on the abstract syntax tree corresponding to each of the at least one code file to obtain a plurality of reference relationship pairs, including: Respectively obtain the abstract syntax tree corresponding to each of the at least one code file; Based on the identifier definition nodes included in the obtained at least one abstract syntax tree, obtain symbolic definitions of each identifier included in the at least one code file; The plurality of reference relationship pairs are obtained based on a symbol definition corresponding to a reference identifier in an identifier reference node included in the at least one abstract syntax tree.

3. The method according to claim 2, characterized in that The method of obtaining the plurality of reference relationship pairs based on the symbol definition corresponding to the reference identifier in the identifier reference node included in the at least one abstract syntax tree comprises: For each identifier reference node, perform the following operations respectively: For an identifier reference node, obtain the prefix content of the reference delimiter in the corresponding expression statement; If the prefix content is a reference identifier, determining the prefix type of the prefix content based on the symbol type of the reference identifier; From the identifiers corresponding to the prefix type, an identifier with the same name as the referenced identifier after the reference separator is searched, and a corresponding reference relationship pair is generated based on the referenced identifier and the symbol definition of the searched identifier.

4. The method according to claim 3, characterized in that After obtaining the prefix content of the reference delimiter in the corresponding expression statement for an identifier reference node, the method further includes: If the prefix content is an expression statement, iteratively obtain the prefix content of the reference delimiter in the prefix content until the obtained prefix content is a reference identifier; Starting from the prefix content obtained last time, iteratively generate reference relationship pairs corresponding to the referenced identifiers corresponding to each prefix content until all reference relationship pairs corresponding to the referenced identifiers in the entire expression statement are generated.

5. The method according to claim 3, characterized in that Determining the prefix type of the prefix content based on the symbol type of the reference identifier includes: If the symbol type is a class, searching for an identifier with the same name as the reference identifier from the identifiers corresponding to the class included in each identifier, and determining the symbol name of the identifier found as the prefix type; If the symbol type is a function, searching for an identifier with the same name as the reference identifier from the identifiers corresponding to the functions included in the identifiers, and determining the return type of the identifier found as the prefix type; If the symbol type is a variable, an identifier with the same name as the reference identifier is searched from the identifiers corresponding to the variables included in the identifiers, and the definition type of the identifier found is determined as the prefix type.

6. The method according to any one of claims 1 to 5, characterized in that: Based on the multiple reference relationship pairs, updating the definition reference relationship graph includes: For each symbol definition in the multiple reference relationship pairs, perform the following operations respectively: For a symbol definition, obtaining a second reference set corresponding to the symbol definition, wherein the second reference set includes: referenced identifiers corresponding to each reference relationship pair of the symbol definition; Compare the second reference set with the first reference set corresponding to the one symbol definition in the definition reference relationship graph to obtain a comparison result; If the comparison result indicates that the second reference set has a difference set relative to the first reference set, the difference set is merged with the first reference set.

7. The method according to any one of claims 1 to 5, characterized in that: Based on the updated definition reference relationship graph, a code index corresponding to the target project is obtained, including: For each definition node and each reference node in the updated definition reference relationship graph, respectively determine a corresponding indexable object and a corresponding index content; According to the relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined; Based on the obtained indexable objects and corresponding index contents, as well as the association relationship between the indexable objects, the code index is generated, and the code index includes a symbol information table composed of each indexable object and index content, and a definition reference relationship table indicating the association relationship between the indexable objects.

8. The method according to claim 7, characterized in that According to the relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined, including: For each definition node in the updated definition reference relationship graph, perform the following operations respectively: For a definition node, a depth-first search method is used to determine a reference node from subordinate nodes of the definition node; Based on the connection relationship between the definition node and the reference node, the association relationship between the corresponding indexable objects is determined.

9. The method according to claim 7, characterized in that Using a depth-first search method, determining a reference node from subordinate nodes of the one definition node includes: In the updated definition reference relationship graph, determining a query result node from an exit node of the one definition node; Determining a reference set node from the exit nodes of the query result node; A reference node of the one definition node is determined from associated items of the reference set node.

10. A code index generating device, characterized in that: The device comprises: A relationship graph generating unit is used to generate a definition reference relationship graph corresponding to the target project based on the syntax analysis result and semantic analysis result of the source code of the target project, wherein the definition reference relationship graph is used to indicate: a first reference set corresponding to each symbol definition, each first reference set including a referenced identifier corresponding to a symbol definition; The enhanced analysis unit is used to determine at least one code file in the target project where the identifier parsing error occurs, and to perform definition reference relationship analysis on the abstract syntax tree corresponding to each of the at least one code file to obtain a plurality of reference relationship pairs, each of which includes: jump link information between the referenced identifier and the corresponding symbol definition in each identifier reference node; The index generating unit is used to update the definition reference relationship graph based on the multiple reference relationship pairs, and obtain the code index corresponding to the target project based on the updated definition reference relationship graph.

11. The device according to claim 10, characterized in that The enhanced analysis unit is specifically used for: Respectively obtain the abstract syntax tree corresponding to each of the at least one code file; Based on the identifier definition nodes included in the obtained at least one abstract syntax tree, obtain symbolic definitions of each identifier included in the at least one code file; The plurality of reference relationship pairs are obtained based on a symbol definition corresponding to a reference identifier in an identifier reference node included in the at least one abstract syntax tree.

12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images