Code information retrieval method and device, equipment and medium

Abstract

The invention discloses a code information retrieval method and device, equipment and a medium. The method comprises the following steps: vectorizing target retrieval information input by a user to obtain a corresponding target retrieval vector; and inputting the target retrieval vector into a code vector library to output corresponding target code information, the target code information comprises at least one of the following: a target code corresponding to a target vector with the highest similarity with the target retrieval vector or greater than a target threshold in a code vector library, target directory information corresponding to the target code, and target syntax tree information corresponding to the target code; the code vector library consists of a directory sub-library for storing directory vectors after directory information vectorization in a code warehouse, a code file sub-library for storing code vectors after code file vectorization in the code warehouse, and a syntax tree sub-library for storing syntax tree vectors after syntax tree information vectorization corresponding to code files; vectors related to the index code warehouse are stored through the code vector library, and the code information retrieval efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a code information retrieval method, device, equipment and medium. Background Art

[0002] As a kind of knowledge asset, the construction of code base and the application based on code base are becoming more and more important in order to give full play to the role of existing code assets. In the process of software development, developers need to spend a lot of time to retrieve code from the code repository for code reuse or as a reference.

[0003] At present, there is an urgent need for an efficient, general and widely used code information retrieval method. Summary of the invention

[0004] The embodiment of the present application provides a code information retrieval method, device, equipment and medium, which improves the retrieval efficiency and versatility of code information by storing and indexing vectors related to the code warehouse through a code vector library. The above technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a code information retrieval method, the method comprising:

[0006] Obtaining target search information input by a user; the target search information includes a target search text and / or a target search code;

[0007] Vectorizing the target retrieval information to obtain a target retrieval vector corresponding to the target retrieval information;

[0008] Input the target search vector into a code vector library, and output target code information corresponding to the target search vector; the target code information includes at least one of the following: a target code corresponding to the target vector in the code vector library, target directory information corresponding to the target code, and target syntax tree information corresponding to the target code; the target vector is used to represent a vector in the code vector library with the highest similarity to the target search vector or a vector greater than a target threshold;

[0009] Among them, the above-mentioned code vector library is composed of a directory sub-library, a code file sub-library and a syntax tree sub-library; the above-mentioned directory sub-library is used to store directory vectors after the directory information in the code warehouse is vectorized, the above-mentioned code file sub-library is used to store code vectors after the code files in the above-mentioned code warehouse are vectorized, and the above-mentioned syntax tree sub-library is used to store syntax tree vectors after the syntax tree information corresponding to the above-mentioned code files is vectorized.

[0010] In a possible implementation, before inputting the target search vector into a code vector library and outputting the target code information corresponding to the target search vector, the method further includes:

[0011] Vectorize the directory information of the code repository to obtain a directory vector corresponding to the directory information, and store the directory vector in the directory sub-library;

[0012] Vectorize the code files in the code repository to obtain code vectors corresponding to the code files, and store the code vectors in the code file sub-repository;

[0013] Vectorize the syntax tree information corresponding to the code file in the code repository to obtain a syntax tree vector corresponding to the syntax tree information, and store the syntax tree vector in the syntax tree sub-library;

[0014] The directory sub-library, the code file sub-library and the syntax tree sub-library are combined to obtain a code vector library corresponding to the code warehouse.

[0015] In a possible implementation, the directory information includes the name and path of the code file;

[0016] The above vectorizes the directory information of the above code warehouse to obtain the directory vector corresponding to the above directory information, including:

[0017] The name and path of each code file in the above code warehouse are vectorized to obtain a directory vector of directory information corresponding to each code file in the above code warehouse.

[0018] In a possible implementation, vectorizing the code files in the code repository to obtain code vectors corresponding to the code files includes:

[0019] Parse the code files in the above code repository to obtain the syntax tree corresponding to the above code files;

[0020] Based on the structure of the syntax tree, the code in the code file is segmented according to the target length to obtain at least one target code segment corresponding to the code file; the code length of the target code segment is less than or equal to the target length;

[0021] The at least one target code fragment is vectorized to obtain at least one target code fragment vector corresponding to the code file.

[0022] In a possible implementation, the syntax tree information includes the node type of the syntax tree corresponding to the code file, the number of code lines contained in the node, and the first line of code corresponding to the node;

[0023] The above-mentioned vectorization of the syntax tree information corresponding to the above-mentioned code file in the above-mentioned code repository to obtain the syntax tree vector corresponding to the above-mentioned syntax tree information includes:

[0024] Performing a deep traversal on the syntax tree corresponding to the above code file, and segmenting the above syntax tree according to the target length to obtain at least one target syntax tree segment;

[0025] The target syntax tree segment information corresponding to the at least one target syntax tree segment is vectorized to obtain at least one target syntax tree segment vector corresponding to the syntax tree information; the length of the target syntax tree segment information is less than or equal to the target length.

[0026] In a possible implementation, vectorizing the directory information of the code repository to obtain a directory vector corresponding to the directory information includes:

[0027] Input the directory information of the above code warehouse into the code vectorization model, and output the directory vector corresponding to the above directory information;

[0028] The code files in the above code warehouse are vectorized to obtain code vectors corresponding to the above code files, including:

[0029] Input the code files in the above code repository into the above code vectorization model, and output the code vectors corresponding to the above code files;

[0030] The above-mentioned vectorization of the syntax tree information corresponding to the above-mentioned code file in the above-mentioned code repository to obtain the syntax tree vector corresponding to the above-mentioned syntax tree information includes:

[0031] Input the syntax tree information corresponding to the above code file in the above code repository into the above code vectorization model, and output the syntax tree vector corresponding to the above syntax tree information;

[0032] The above-mentioned target search information is vectorized to obtain a target search vector corresponding to the above-mentioned target search information, including:

[0033] The target retrieval information is input into the code vectorization model, and a target retrieval vector corresponding to the target retrieval information is output.

[0034] In a possible implementation, the code vector library includes a Milvus vector database, which is used to analyze the correlation between two vectors to obtain the similarity between the two vectors.

[0035] In a second aspect, an embodiment of the present application provides a code information retrieval device, the code information retrieval device comprising:

[0036] An acquisition module is used to acquire target search information input by a user; the target search information includes a target search text and / or a target search code;

[0037] A first vectorization module, used to vectorize the target search information to obtain a target search vector corresponding to the target search information;

[0038] A code information retrieval module, used to input the target retrieval vector into a code vector library, and output target code information corresponding to the target retrieval vector; the target code information includes at least one of the following: a target code corresponding to the target vector in the code vector library, target directory information corresponding to the target code, and target syntax tree information corresponding to the target code; the target vector is used to represent a vector in the code vector library with the highest similarity to the target retrieval vector or a vector greater than a target threshold;

[0039] Among them, the above-mentioned code vector library is composed of a directory sub-library, a code file sub-library and a syntax tree sub-library; the above-mentioned directory sub-library is used to store directory vectors after the directory information in the code warehouse is vectorized, the above-mentioned code file sub-library is used to store code vectors after the code files in the above-mentioned code warehouse are vectorized, and the above-mentioned syntax tree sub-library is used to store syntax tree vectors after the syntax tree information corresponding to the above-mentioned code files is vectorized.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0041] The processor is connected to the memory;

[0042] The above-mentioned memory is used to store executable program code;

[0043] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method provided by the first aspect of the embodiment of this specification or any possible implementation of the first aspect.

[0044] In a fourth aspect, an embodiment of the present specification provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method provided by the first aspect of the embodiment of the present specification or any possible implementation of the first aspect.

[0045] In one or more embodiments of the present application, target retrieval information input by a user is obtained, the target retrieval information including a target retrieval text and / or a target retrieval code; the target retrieval information is vectorized to obtain a target retrieval vector corresponding to the target retrieval information; the target retrieval vector is input into a code vector library, and target code information corresponding to the target retrieval vector is output, the target code information including at least one of the following: a target code corresponding to the target vector in the code vector library, a target directory information corresponding to the target code, and a target syntax tree information corresponding to the target code; the target vector is used to represent a vector in the code vector library with the highest similarity to the target retrieval vector or a vector greater than a target threshold, and the code vector library is composed of a directory sub-library, a code file sub-library, and a syntax tree sub-library. The directory sub-library is used to store the directory vectors after the directory information in the code warehouse is vectorized, the code file sub-library is used to store the code vectors after the code files in the code warehouse are vectorized, and the syntax tree sub-library is used to store the syntax tree vectors after the syntax tree information corresponding to the code files is vectorized, so as to utilize the embedding technology to convert the code and related information in the code warehouse into vectors, and store and index these vectors with the help of the code vector library, so as to facilitate developers to efficiently find the desired code and related information in the code vector library, improve the efficiency of code information retrieval, so as to assist them in improving software development efficiency and enable AI to have a broad expansion space in the downstream applications of the code vector library, such as code retrieval, code recommendation, code analysis, code tracing, code classification, etc., so as to facilitate AI to be better applied in the code field.

[0046] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of the architecture of a code information retrieval system provided for an exemplary embodiment of the present application;

[0049] Figure 2 A flowchart of a code information retrieval method provided for an exemplary embodiment of the present application;

[0050] Figure 3A schematic diagram of a code vector library construction process provided for an exemplary embodiment of the present application;

[0051] Figure 4 A schematic diagram of a code vector library construction process provided for an exemplary embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of a code information retrieval device provided for an exemplary embodiment of the present application;

[0053] Figure 6 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0055] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0056] Please refer to the following Figure 1 , which is a schematic diagram of the architecture of a code information retrieval system provided by an exemplary embodiment of this specification. Figure 1 As shown, the code information retrieval system may include: a terminal 110 and a server 120. Among them:

[0057] The terminal 110 can interact with the server 120 through the network to receive messages from the server 120 or send messages to the server 120. The terminal 110 can be hardware or software. When the terminal 120 is hardware, it can be various electronic devices, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers and desktop computers. When the terminal 110 is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0058] In the embodiments of the present specification, the terminal 110 may be a user terminal corresponding to a user such as a developer who needs to perform code information retrieval. When a user wants to find the complete code or source or similar code or syntax tree corresponding to a certain code segment, or wants to find code information that can realize a certain function, the corresponding target retrieval information may be entered into the code information retrieval software on the terminal 110, but is not limited to, such as but not limited to including a target retrieval text and / or a target retrieval code related to the code. After receiving the target retrieval information entered by the user, the terminal 110 may first vectorize the target retrieval information to obtain a target retrieval vector corresponding to the target retrieval information; then, the target retrieval vector is input into the code vector library, and the target code information corresponding to the target retrieval vector is output, the target code information including at least one of the following: the target code corresponding to the target vector in the code vector library, the target directory information corresponding to the target code, and the target syntax tree information corresponding to the target code. The target vector is used to represent the vector in the code vector library with the highest similarity to the target retrieval vector or greater than the target threshold. The above-mentioned code vector library consists of a directory sub-library, a code file sub-library and a syntax tree sub-library. The directory sub-library is used to store directory vectors after vectorization of directory information in the code warehouse, the code file sub-library is used to store code vectors after vectorization of code files in the code warehouse, and the syntax tree sub-library is used to store syntax tree vectors after vectorization of syntax tree information corresponding to code files.

[0059] Optionally, after the terminal 110 user inputs the target search information, the target search information may be directly sent to the server 120 through the network, so that the server 120 vectorizes the target search information, searches based on the code vector library, and returns the corresponding target code information.

[0060] The server 120 may be a server that provides various code information retrieval services. It should be noted that the server 120 may be hardware or software. When the server 120 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 120 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module, which is not specifically limited here. The server 120 may be, but is not limited to, a hardware server, a virtual server, a cloud server, etc.

[0061] Specifically, after receiving the target search information sent by the terminal 110 , the server 120 may also perform corresponding code information retrieval through the code information retrieval method provided in the embodiment of the present application and return the retrieved corresponding target code information to the terminal 110 .

[0062] It can be understood that the code information retrieval method provided in the embodiment of the present application can be executed by the terminal 110 alone, by the server 120 alone, or by the terminal 110 and the server 120 together, and the embodiment of the present application is not limited to this.

[0063] The network may be a medium providing a communication link between the terminal 110 and the server 120, or may be the Internet including network devices and transmission media, but is not limited thereto. The transmission media may be a wired link, such as but not limited to coaxial cable, optical fiber, and digital subscriber line (DSL), or a wireless link, such as but not limited to wireless fidelity (WIFI), Hypertext Transfer Protocol (HTTP), Bluetooth, and mobile device network, etc.

[0064] Understandably, Figure 1 The number of terminals 110 and servers 120 in the code information retrieval system shown is only an example. In a specific implementation, the code information retrieval system may include any number of terminals 110 and servers 120.

[0065] Next, combine Figure 1 , taking the server 120 performing code information retrieval as an example, the code information retrieval method provided by an exemplary embodiment of the present application is introduced. For details, please refer to Figure 2 , which exemplarily shows a flow chart of a code information retrieval method provided in an embodiment of the present application. Figure 2 As shown, the code information retrieval method includes the following steps:

[0066] S201, obtaining target search information input by a user, where the target search information includes a target search text and / or a target search code.

[0067] Specifically, when the user wants to retrieve code information, the server 120 can obtain the target search information input by the user and sent by the terminal 110 through the network. The above-mentioned target search information may include, but is not limited to, target search text and / or target search code, and the above-mentioned target search text may include, but is not limited to, function description text and / or attribute description text corresponding to the code information that the user wants to retrieve, that is, when the user needs to find a code that can realize certain functions or has certain attributes or its related information, the corresponding search can be performed by entering the corresponding function description text and / or attribute description text, i.e., the target search information. The above-mentioned target search code may include, but is not limited to, the code or code fragment that the user currently wants to retrieve, that is, the user can retrieve a code similar to the target search code or trace the source of the target search code by entering the target search code that the user currently wants to retrieve, i.e., the target search information.

[0068] S202, vectorizing the target search information to obtain a target search vector corresponding to the target search information.

[0069] Specifically, after obtaining the target search information input by the user, the target search information may be vectorized to obtain a corresponding target search vector, so that efficient code information search can be performed in the code vector library based on the target search vector later.

[0070] Optionally, in order to realize the vectorization of target retrieval information more efficiently and accurately, after obtaining the target retrieval information input by the user, the target retrieval information can be input into a trained code vectorization model, and the target retrieval vector corresponding to the target retrieval information can be output. The above-mentioned code vectorization model may include but is not limited to a sentence-transformers / all-mpnet-base-v2 model or a longcoder-base model.

[0071] It can be understood that the above code vectorization model can be selected from models such as sentence-transformers / all-mpnet-base-v2 model or longcoder-base model according to actual vectorization requirements or code information retrieval requirements, and the embodiments of the present application are not limited to this. After testing, the accuracy of vectorization using the above sentence-transformers / all-mpnet-base-v2 model is lower than the accuracy of vectorization using the longcoder-base model, and the vectorization speed of the above sentence-transformers / all-mpnet-base-v2 model is greater than the vectorization speed of the longcoder-base model.

[0072] S203, input the target retrieval vector into the code vector library, and output the target code information corresponding to the target retrieval vector. The code vector library consists of a directory sub-library, a code file sub-library and a syntax tree sub-library. The directory sub-library is used to store directory vectors after the directory information in the code warehouse is vectorized, the code file sub-library is used to store code vectors after the code files in the code warehouse are vectorized, and the syntax tree sub-library is used to store syntax tree vectors after the syntax tree information corresponding to the code files is vectorized.

[0073] Specifically, the target search vector can be input into the code vector library, the similarity between the target search vector and each vector stored in the code vector library is calculated, one or more target vectors in the code vector library with the highest similarity to the target search vector or greater than the target threshold are found, and the corresponding target code information is output based on the one or more target vectors. The above-mentioned target vector may include but is not limited to at least one of the following: a target directory vector in the directory sub-library with the highest similarity to the target search vector or greater than the target threshold, a target code vector in the code file sub-library with the highest similarity to the target search vector or greater than the target threshold, and a target syntax tree vector in the syntax tree sub-library with the highest similarity to the target search vector or greater than the target threshold. The above-mentioned target code information includes at least one of the following: a target code corresponding to the target vector in the code vector library, a target directory information corresponding to the target code, and a target syntax tree information corresponding to the target code. When the above-mentioned target vector is a target directory vector or a target syntax tree vector, the target code corresponding to the above-mentioned target vector may include but is not limited to the target code corresponding to the target directory information corresponding to the target directory vector in the code vector library or the target code corresponding to the target syntax tree information corresponding to the target syntax tree vector in the code vector library.

[0074] Optionally, the above-mentioned code vector library may include, but is not limited to, a Milvus vector database, which is used to analyze the correlation between two vectors and obtain the similarity between the two vectors, thereby providing developers or users with a more efficient and convenient code data processing method, giving full play to the role of existing code repositories, and speeding up the user's code information retrieval speed.

[0075] The embodiment of the present application converts the code and related information in the code warehouse into vectors by using embedding technology, and stores and indexes these vectors with the help of the code vector library, so that developers can efficiently find the desired code and related information (target code information) in the code vector library according to the vectorized target retrieval information they want to retrieve, thereby improving the efficiency of code information retrieval to assist them in improving software development efficiency. At the same time, it can also enable AI to have a wide range of expansion space in the downstream application of the code vector library, such as code retrieval, code recommendation, code analysis, code tracing, code classification, etc., so as to facilitate the better application of AI in the field of code.

[0076] With the development and growth of AI, especially the application of large models, vector databases have gradually emerged in everyone's eyes. Unlike traditional relational databases that mainly process structured data, vector databases can accelerate the retrieval of unstructured data. The underlying design processes embedded vectors converted from unstructured data, adapting to the explosive growth of unstructured data in the Internet era, such as short videos, grass-seeking pictures and texts, IoT sensor data, protein structures, and so on.

[0077] Based on this, in order to improve the efficiency of code information retrieval and make full use of the code assets in the code repository, before performing code information retrieval, the embodiment of the present application also needs to first build a corresponding code vector library based on the code repository, and use embedding technology to convert the code and related information in the code repository into vectors. With the help of a vector database, these vectors are stored and indexed, which facilitates AI to be better applied in the code field, and provides an important foundation and knowledge source for the application of code AI in code retrieval, code recommendation, code analysis, code traceability, code classification, etc.

[0078] Specifically, the construction process of the above code vector library is as follows: Figure 3 As shown, that is, in the above S203, before the target search vector is input into the code vector library and the target code information corresponding to the target search vector is output, the above code information search method may further include:

[0079] S301, vectorize the directory information of the code repository to obtain a directory vector corresponding to the directory information, and store the directory vector in a directory sub-library.

[0080] Optionally, the code repository may be a code repository pre-stored in the server, or a code repository sent by other terminals or other servers received through the network. The above code repository is used to manage and store the source code of the software (application project). Figure 4 As shown, after obtaining the code repository to be vectorized, the directory information of the code repository can be input into the code vectorization model to vectorize it, and the directory vector corresponding to the directory information is output.

[0081] Optionally, the above directory information may include, but is not limited to, the name and path of each code file in the code warehouse. The above-mentioned vectorizing the directory information of the code warehouse to obtain the directory vector corresponding to the directory information may include: traversing the code warehouse, obtaining the names and paths of all code files in the code warehouse, and vectorizing the directory information with the path and the directory information containing only the file name, that is, vectorizing the name and path of each code file in the code warehouse, obtaining the directory vector of the directory information corresponding to each code file in the code warehouse and storing it in the directory sub-library.

[0082] It can be understood that if the directory information is larger than the target length, it can be considered that the directory information is too long, and the directory information will be divided according to the target length first, and the directory information will be divided into multiple directory sub-information that is less than or equal to the target length. The above target length can be but is not limited to the length of the minimum language unit that can be recognized and processed by the code vectorization model, and can also be set according to actual vectorization requirements, so as to avoid the problem of directory information being too long resulting in its vectorization storage failure or a long time consumption, thereby improving the success rate and efficiency of directory information vectorization.

[0083] S302, vectorize the code files in the code repository to obtain code vectors corresponding to the code files, and store the code vectors in the code file sub-repository.

[0084] Optionally, after obtaining the code repository to be vectorized, the code files in the code repository may be input into the code vectorization model, and the code vectors corresponding to the code files may be directly output and stored in the code file sub-repository.

[0085] Alternatively, if Figure 4As shown, the above S302, vectorizing the code files in the code warehouse, and obtaining the code vectors corresponding to the code files may include: first parsing the code files in the code warehouse to obtain the syntax tree corresponding to the code files, for example but not limited to, for each code file in the code warehouse, obtaining the code content in the code file, using tree-sitter parsing to generate and extract the corresponding abstract syntax tree; then, based on the structure of the syntax tree, segmenting the code in the code file according to the target length to obtain at least one target code fragment corresponding to the code file, for example but not limited to, deeply traversing the syntax tree corresponding to the code file, when the traversed code length reaches the target length, the target code fragment can be segmented based on the target length and the structure of the syntax tree, and so on, until the syntax tree is deeply traversed to obtain at least one target code fragment corresponding to the code file, the at least one target code fragment is continuous and the code in each target code fragment is complete, for example, they all belong to the code corresponding to the same syntax tree node. The code length of the above target code fragment is less than or equal to the target length. After obtaining at least one target code fragment corresponding to the code file, the above at least one target code fragment can be vectorized to obtain at least one target code fragment vector corresponding to the code file. The code vector corresponding to the above code file may include the above at least one target code fragment vector or be obtained by merging the above at least one target code fragment vector.

[0086] It can be understood that in order to reduce the time required for vectorization when a large number of target code fragments are too short, that is, the code files are too fragmented, and improve the efficiency of code file vectorization, when there are two consecutive target code fragments and the sum of their code lengths is less than or equal to the target length in at least one target code fragment, the above two consecutive target code fragments and the sum of their code lengths is less than or equal to the target length can be merged before vectorization.

[0087] S303, vectorizing the syntax tree information corresponding to the code file in the code repository to obtain a syntax tree vector corresponding to the syntax tree information, and storing the syntax tree vector in a syntax tree sub-library.

[0088] Optionally, after obtaining the code repository to be vectorized, the syntax tree information corresponding to the code files in the code repository may be input into the code vectorization model, and the syntax tree vector corresponding to the syntax tree information may be output and stored in the syntax tree sub-library.

[0089] Optionally, the syntax tree information includes the node type of the syntax tree corresponding to the code file, the number of code lines contained in the node, and the first line of code corresponding to the node. Figure 4As shown, the above S303, vectorizing the syntax tree information corresponding to the code file in the code repository to obtain the syntax tree vector corresponding to the syntax tree information may include: firstly deeply traversing the syntax tree corresponding to the code file, and segmenting the syntax tree according to the target length to obtain at least one target syntax tree segment; then, vectorizing the target syntax tree segment information corresponding to the at least one target syntax tree segment to obtain at least one target syntax tree segment vector corresponding to the syntax tree information, wherein the length of the target syntax tree segment information is less than or equal to the target length. That is, the syntax tree vector corresponding to the syntax tree information may include but is not limited to the at least one target syntax tree segment vector or be obtained by merging the at least one target syntax tree segment vector.

[0090] In the embodiment of the present application, the syntax tree information that is vectorized and stored not only extracts the type, structure, number of lines of code, etc. of the syntax tree node, but also vectorizes the first line of the code (the first line of content) as the attribute of the syntax tree node and stores it in the code vector library together, so that the vectors in the code vector library can more richly represent the meaning information represented by some codes, etc., so as to improve the versatility and practicality of subsequent code information retrieval. At the same time, when performing code segmentation and syntax tree segmentation, language differences can be ignored, and code fragments and syntax tree fragments can be segmented according to the depth of the syntax tree and the length of the code content, so that the construction of the code vector library has versatility.

[0091] S304, combining the directory sub-library, the code file sub-library and the syntax tree sub-library to obtain a code vector library corresponding to the code repository.

[0092] Specifically, Figure 4 As shown, after constructing the directory sub-library, code file sub-library and syntax tree sub-library corresponding to the code warehouse, the above directory sub-library, code file sub-library and syntax tree sub-library can be directly combined to obtain the code vector library corresponding to the code warehouse, so as to construct a code vector library including various vectorized vectors such as directory information, code files and syntax tree information in the code warehouse, so that the vectors in the code vector library can fully and comprehensively cover various code information resources in the code warehouse, so that the code vector library corresponding to the code warehouse can be applied to downstream tasks such as code retrieval, code recommendation, code analysis, code tracing, code classification, etc., to provide the required information for more downstream applications and improve the application scope of code information retrieval.

[0093] Optionally, the code vector library can be constructed based on one or more code warehouses, but is not limited to it, that is, the directory sub-library in the code vector library can store the vectorized directory information (directory vector) in one or more code warehouses, the code file sub-library in the code vector library can store the vectorized code files (code vectors) in one or more code warehouses, and the syntax tree sub-library in the code vector library can store the vectorized syntax tree information (syntax tree vector) in one or more code warehouses. At the same time, in order to facilitate retrieval, in addition to storing the corresponding vectors, the directory sub-library, code file sub-library and syntax tree sub-library also store the code warehouse identifier corresponding to each vector, the number of contents corresponding to such vectors in the code warehouse corresponding to the code warehouse identifier, the project label of the code warehouse corresponding to the vector (for example, types such as front-end, back-end, algorithm, etc.), the specific content corresponding to the vector, etc.

[0094] Please refer to the following Figure 5 , which is a schematic diagram of the structure of a code information retrieval device provided by an exemplary embodiment of the present application. Figure 5 As shown, the code information retrieval device 500 includes:

[0095] The acquisition module 510 is used to acquire the target search information input by the user; the target search information includes the target search text and / or the target search code;

[0096] A first vectorization module 520, configured to vectorize the target search information to obtain a target search vector corresponding to the target search information;

[0097] The code information retrieval module 530 is used to input the target retrieval vector into the code vector library and output the target code information corresponding to the target retrieval vector; the target code information includes at least one of the following: the target code corresponding to the target vector in the code vector library, the target directory information corresponding to the target code, and the target syntax tree information corresponding to the target code; the target vector is used to represent the vector in the code vector library with the highest similarity to the target retrieval vector or a vector greater than a target threshold;

[0098] Among them, the above-mentioned code vector library is composed of a directory sub-library, a code file sub-library and a syntax tree sub-library; the above-mentioned directory sub-library is used to store directory vectors after the directory information in the code warehouse is vectorized, the above-mentioned code file sub-library is used to store code vectors after the code files in the above-mentioned code warehouse are vectorized, and the above-mentioned syntax tree sub-library is used to store syntax tree vectors after the syntax tree information corresponding to the above-mentioned code files is vectorized.

[0099] In a possible implementation, the code information retrieval device 500 further includes:

[0100] A second vectorization module is used to vectorize the directory information of the code repository to obtain a directory vector corresponding to the directory information, and store the directory vector in the directory sub-library;

[0101] A third vectorization module is used to vectorize the code files in the code repository to obtain code vectors corresponding to the code files, and store the code vectors in the code file sub-library;

[0102] A fourth vectorization module, used to vectorize the syntax tree information corresponding to the code file in the code repository, obtain a syntax tree vector corresponding to the syntax tree information, and store the syntax tree vector in the syntax tree sub-library;

[0103] The combination module is used to combine the directory sub-library, the code file sub-library and the syntax tree sub-library to obtain a code vector library corresponding to the code warehouse.

[0104] In a possible implementation, the directory information includes the name and path of the code file;

[0105] The second vectorization module is specifically used to: vectorize the name and path of each code file in the code warehouse to obtain a directory vector of directory information corresponding to each code file in the code warehouse.

[0106] In a possible implementation, the third vectorization module includes:

[0107] A code file parsing unit, used to parse the code files in the above code repository to obtain a syntax tree corresponding to the above code files;

[0108] A code slicing unit, configured to segment the code in the code file according to a target length based on the structure of the syntax tree to obtain at least one target code segment corresponding to the code file; the code length of the target code segment is less than or equal to the target length;

[0109] The first vectorization unit is used to vectorize the at least one target code fragment to obtain at least one target code fragment vector corresponding to the code file.

[0110] In a possible implementation, the syntax tree information includes the node type of the syntax tree corresponding to the code file, the number of code lines contained in the node, and the first line of code corresponding to the node;

[0111] The fourth vectorization module includes:

[0112] A syntax tree segmentation unit, used for deeply traversing the syntax tree corresponding to the above code file, and segmenting the above syntax tree according to a target length to obtain at least one target syntax tree segment;

[0113] The second vectorization unit is used to vectorize the target syntax tree segment information corresponding to the above-mentioned at least one target syntax tree segment to obtain at least one target syntax tree segment vector corresponding to the above-mentioned syntax tree information; the length of the above-mentioned target syntax tree segment information is less than or equal to the above-mentioned target length.

[0114] In a possible implementation, the second vectorization module is specifically used to: input the directory information of the code repository into a code vectorization model, and output a directory vector corresponding to the directory information;

[0115] The third vectorization module is specifically used to: input the code file in the code repository into the code vectorization model, and output the code vector corresponding to the code file;

[0116] The fourth vectorization module is specifically used to: input the syntax tree information corresponding to the code file in the code repository into the code vectorization model, and output the syntax tree vector corresponding to the syntax tree information;

[0117] The first vectorization module 520 is specifically used to: input the target search information into the code vectorization model, and output a target search vector corresponding to the target search information.

[0118] In a possible implementation, the code vector library includes a Milvus vector database, which is used to analyze the correlation between two vectors to obtain the similarity between the two vectors.

[0119] The division of each module in the above-mentioned code information retrieval device is only for illustration. In other embodiments, the code information retrieval device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned code information retrieval device. The implementation of each module in the code information retrieval device provided in the embodiment of this specification can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, all or part of the steps of the code information retrieval method described in the embodiment of this specification are implemented.

[0120] See next Figure 6 , which is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of this specification. Figure 6As shown, the electronic device 600 may include: at least one processor 610 , at least one communication bus 620 , a user interface 630 , at least one network interface 640 , and a memory 650 .

[0121] The communication bus 620 may be used to realize the connection and communication among the above-mentioned components.

[0122] The user interface 630 may include a display screen (Display) and a camera (Camera), and the optional user interface 630 may also include a standard wired interface and a wireless interface.

[0123] The network interface 640 may optionally include a Bluetooth module, a Near Field Communication (NFC) module, a Wireless Fidelity (Wi-Fi) module, and the like.

[0124] Among them, the processor 610 may include one or more processing cores. The processor 610 uses various interfaces and lines to connect various parts within the entire electronic device 600, and executes various functions and processes data of the routing electronic device 600 by running or executing instructions, programs, code sets or instruction sets stored in the memory 650, and calling data stored in the memory 650. Optionally, the processor 610 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 610 can integrate one or a combination of a processor (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 610, and it can be implemented separately through a chip.

[0125] The memory 650 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 650 includes a non-transitory computer-readable medium. The memory 650 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 650 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as an acquisition function, a vectorization function, a code information retrieval function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 650 may optionally be at least one storage device located away from the aforementioned processor 610. As Figure 6 As shown, the memory 650 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.

[0126] In some possible embodiments, the electronic device 600 may be the aforementioned Figure 5 In the code information retrieval device 500 shown, the processor 610 can be used to call the program instructions stored in the memory 650 and specifically perform the following operations:

[0127] Obtain target retrieval information input by a user; the target retrieval information includes a target retrieval text and / or a target retrieval code; vectorize the target retrieval information to obtain a target retrieval vector corresponding to the target retrieval information; input the target retrieval vector into a code vector library, and output target code information corresponding to the target retrieval vector; the target code information includes at least one of the following: a target code corresponding to the target vector in the code vector library, a target directory information corresponding to the target code, and a target syntax tree information corresponding to the target code; the target vector is used to represent a vector in the code vector library that has the highest similarity with the target retrieval vector or has a similarity greater than a target threshold.

[0128] Among them, the above-mentioned code vector library is composed of a directory sub-library, a code file sub-library and a syntax tree sub-library; the above-mentioned directory sub-library is used to store directory vectors after the directory information in the code warehouse is vectorized, the above-mentioned code file sub-library is used to store code vectors after the code files in the above-mentioned code warehouse are vectorized, and the above-mentioned syntax tree sub-library is used to store syntax tree vectors after the syntax tree information corresponding to the above-mentioned code files is vectorized.

[0129] In some possible embodiments, before the processor 610 executes the input of the target retrieval vector into the code vector library and outputs the target code information corresponding to the target retrieval vector, it is also used to execute: vectorizing the directory information of the code warehouse to obtain the directory vector corresponding to the directory information, and storing the directory vector in the directory sub-library; vectorizing the code files in the code warehouse to obtain the code vectors corresponding to the code files, and storing the code vectors in the code file sub-library; vectorizing the syntax tree information corresponding to the code files in the code warehouse to obtain the syntax tree vectors corresponding to the syntax tree information, and storing the syntax tree vectors in the syntax tree sub-library; combining the directory sub-library, the code file sub-library and the syntax tree sub-library to obtain the code vector library corresponding to the code warehouse.

[0130] In some possible embodiments, the directory information includes the name and path of the code file;

[0131] When the processor 610 performs the vectorization of the directory information of the code repository to obtain the directory vector corresponding to the directory information, it is specifically used to execute:

[0132] The name and path of each code file in the above code warehouse are vectorized to obtain a directory vector of directory information corresponding to each code file in the above code warehouse.

[0133] In some possible embodiments, when the processor 610 performs the vectorization of the code file in the code repository to obtain the code vector corresponding to the code file, it is specifically configured to perform:

[0134] The code files in the above code repository are parsed to obtain a syntax tree corresponding to the above code files; based on the structure of the above syntax tree, the codes in the above code files are segmented according to the target length to obtain at least one target code fragment corresponding to the above code file; the code length of the above target code fragment is less than or equal to the above target length; the above at least one target code fragment is vectorized to obtain at least one target code fragment vector corresponding to the above code file.

[0135] In some possible embodiments, the syntax tree information includes the node type of the syntax tree corresponding to the code file, the number of code lines contained in the node, and the first line of code corresponding to the node;

[0136] When the processor 610 executes the vectorization of the syntax tree information corresponding to the code file in the code repository to obtain the syntax tree vector corresponding to the syntax tree information, it is specifically used to execute: deeply traverse the syntax tree corresponding to the code file, and split the syntax tree according to the target length to obtain at least one target syntax tree segment; vectorize the target syntax tree segment information corresponding to the at least one target syntax tree segment to obtain at least one target syntax tree segment vector corresponding to the syntax tree information; the length of the target syntax tree segment information is less than or equal to the target length.

[0137] In some possible embodiments, when the processor 610 performs the vectorization of the directory information of the code repository to obtain the directory vector corresponding to the directory information, it is specifically used to perform:

[0138] The directory information of the above code warehouse is input into the code vectorization model, and the directory vector corresponding to the above directory information is output.

[0139] When the processor 610 executes the vectorization of the code files in the code repository to obtain the code vectors corresponding to the code files, it is specifically used to execute: inputting the code files in the code repository into the code vectorization model, and outputting the code vectors corresponding to the code files.

[0140] When the processor 610 performs the vectorization of the syntax tree information corresponding to the code file in the code repository to obtain the syntax tree vector corresponding to the syntax tree information, it is specifically used to execute:

[0141] The syntax tree information corresponding to the above code file in the above code repository is input into the above code vectorization model, and the syntax tree vector corresponding to the above syntax tree information is output.

[0142] When the processor 610 executes the vectorization of the target retrieval information to obtain the target retrieval vector corresponding to the target retrieval information, it is specifically used to execute: input the target retrieval information into the code vectorization model, and output the target retrieval vector corresponding to the target retrieval information.

[0143] In some possible embodiments, the code vector library includes a Milvus vector database, which is used to analyze the correlation between two vectors to obtain the similarity between the two vectors.

[0144] The embodiment of the present application also provides a computer storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods. If the components of the above code information retrieval device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above storage medium.

[0145] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the above process or function according to the embodiment of the present application is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer-readable storage medium or transmitted by the above computer-readable storage medium. The above computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The above computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The above available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes. In the absence of conflict, the technical features in this embodiment and the implementation scheme can be combined arbitrarily.

[0147] The above-mentioned embodiments are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the protection scope determined by the claims of the present application.

Claims

1. A code information retrieval method, It is characterized in that The method comprises: Acquire target search information input by a user; the target search information includes a target search text and / or a target search code; Vectorizing the target search information to obtain a target search vector corresponding to the target search information; Input the target search vector into a code vector library, and output the target code information corresponding to the target search vector; the target code information includes at least one of the following: a target code corresponding to the target vector in the code vector library, a target directory information corresponding to the target code, and a target syntax tree information corresponding to the target code; the target vector is used to represent the vector in the code vector library with the highest similarity to the target search vector or a vector greater than a target threshold; Among them, the code vector library is composed of a directory sub-library, a code file sub-library and a syntax tree sub-library; the directory sub-library is used to store directory vectors after the directory information in the code warehouse is vectorized, the code file sub-library is used to store code vectors after the code files in the code warehouse are vectorized, and the syntax tree sub-library is used to store syntax tree vectors after the syntax tree information corresponding to the code files is vectorized.

2. The method according to claim 1, It is characterized in that Before inputting the target search vector into a code vector library and outputting target code information corresponding to the target search vector, the method further includes: Vectorizing the directory information of the code repository to obtain a directory vector corresponding to the directory information, and storing the directory vector in the directory sub-library; Vectorizing the code files in the code repository to obtain code vectors corresponding to the code files, and storing the code vectors in the code file sub-repository; Vectorizing the syntax tree information corresponding to the code file in the code repository to obtain a syntax tree vector corresponding to the syntax tree information, and storing the syntax tree vector in the syntax tree sub-library; The directory sub-library, the code file sub-library and the syntax tree sub-library are combined to obtain a code vector library corresponding to the code warehouse.

3. The method according to claim 2, It is characterized in that The directory information includes the name and path of the code file; The vectorizing the directory information of the code repository to obtain a directory vector corresponding to the directory information includes: The name and path of each code file in the code warehouse are vectorized to obtain a directory vector of directory information corresponding to each code file in the code warehouse.

4. The method according to claim 2, It is characterized in that The vectorizing the code file in the code repository to obtain a code vector corresponding to the code file includes: Parsing the code files in the code repository to obtain syntax trees corresponding to the code files; Based on the structure of the syntax tree, the code in the code file is segmented according to the target length to obtain at least one target code segment corresponding to the code file; the code length of the target code segment is less than or equal to the target length; The at least one target code segment is vectorized to obtain at least one target code segment vector corresponding to the code file.

5. The method according to claim 2, It is characterized in that The syntax tree information includes the node type of the syntax tree corresponding to the code file, the number of code lines contained in the node, and the first line of code corresponding to the node; The vectorizing the syntax tree information corresponding to the code file in the code repository to obtain a syntax tree vector corresponding to the syntax tree information includes: Performing a deep traversal on the syntax tree corresponding to the code file, and segmenting the syntax tree according to a target length to obtain at least one target syntax tree segment; The target syntax tree segment information corresponding to the at least one target syntax tree segment is vectorized to obtain at least one target syntax tree segment vector corresponding to the syntax tree information; the length of the target syntax tree segment information is less than or equal to the target length.

6. The method according to any one of claims 2 to 4, It is characterized in that The vectorizing the directory information of the code repository to obtain a directory vector corresponding to the directory information includes: Inputting directory information of the code repository into a code vectorization model, and outputting a directory vector corresponding to the directory information; The vectorizing the code file in the code repository to obtain a code vector corresponding to the code file includes: Inputting the code files in the code repository into the code vectorization model, and outputting the code vectors corresponding to the code files; The vectorizing the syntax tree information corresponding to the code file in the code repository to obtain a syntax tree vector corresponding to the syntax tree information includes: Inputting syntax tree information corresponding to the code file in the code repository into the code vectorization model, and outputting a syntax tree vector corresponding to the syntax tree information; The vectorizing the target search information to obtain a target search vector corresponding to the target search information includes: The target retrieval information is input into the code vectorization model, and a target retrieval vector corresponding to the target retrieval information is output.

7. The method according to claim 1, It is characterized in that The code vector library includes a Milvus vector database, which is used to analyze the correlation between two vectors and obtain the similarity between the two vectors.

8. A code information retrieval device, It is characterized in that The code information retrieval device comprises: An acquisition module, used for acquiring target search information input by a user; the target search information includes a target search text and / or a target search code; A first vectorization module, used for vectorizing the target search information to obtain a target search vector corresponding to the target search information; a code information retrieval module, configured to input the target retrieval vector into a code vector library, and output target code information corresponding to the target retrieval vector; the target code information includes at least one of the following: a target code corresponding to the target vector in the code vector library, target directory information corresponding to the target code, and target syntax tree information corresponding to the target code; the target vector is used to represent a vector in the code vector library with the highest similarity to the target retrieval vector or a vector greater than a target threshold; Among them, the code vector library is composed of a directory sub-library, a code file sub-library and a syntax tree sub-library; the directory sub-library is used to store directory vectors after the directory information in the code warehouse is vectorized, the code file sub-library is used to store code vectors after the code files in the code warehouse are vectorized, and the syntax tree sub-library is used to store syntax tree vectors after the syntax tree information corresponding to the code files is vectorized.

9. An electronic device, It is characterized in that include: processor and memory; wherein, The processor is connected to the memory; the memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method steps according to any one of claims 1 to 7.

10. A computer storage medium, It is characterized in that The computer storage medium stores a plurality of instructions, which are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 7.

Images