Method and device for performing context retrieval based on code repository

Abstract

The embodiment of the invention provides a method for conducting context retrieval based on a code warehouse. The method comprises the steps that a first question text is received; rewriting the first question text, performing keyword retrieval based on the obtained rewritten text to obtain a plurality of first warehouse files matched with the rewritten text, and classifying the first warehouse files into a first set; performing target reasoning by utilizing a pre-trained target model to obtain a plurality of supplementary warehouse files related to the rewritten text, and classifying the supplementary warehouse files into a first set; screening the first set to retain the warehouse files meeting a first standard, the first standard being defined according to the summary relevancy of the warehouse files and the rewritten text; any warehouse file in the first set is divided into a plurality of context fragments, target context fragments meeting a second standard are selected, the second standard is defined according to the content relevancy of the context fragments and the first problem text, and the target context fragments are used for downstream task processing related to the first problem text.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of large language models and retrieval enhancement technology, and in particular, to a method and device for contextual retrieval based on a code repository. Background Art

[0002] With the continuous development of artificial intelligence technology, large language models are widely used in various software engineering projects, including code problem solving, program automatic repair, code automatic completion, software engineering intelligent agent development and many other sinking scene applications. The development of these scene applications usually relies on retrieval-augmented generation (RAG) technology, which expands the knowledge scope of large language models by using external knowledge bases.

[0003] However, as the complexity of software engineering projects continues to increase, the reference and calling relationships between the functional modules contained in the source code files become intricate. Traditional retrieval enhancement generation technology finds it difficult to understand the code structure in modern software projects, and thus inevitably produces errors when identifying and locating code contexts, retrieving code contexts that are not highly correlated with the problem query, which in turn affects the accuracy of downstream code processing tasks.

[0004] Therefore, we hope to have a solution that can improve the ability of large language models to understand code repositories containing source code of many software engineering projects through technical means, so as to accurately locate the code context related to the problem query and improve the accuracy of retrieval. Summary of the invention

[0005] One or more embodiments of this specification describe a method and device for contextual retrieval based on a code repository to enhance the large language model's ability to understand repository files in a code repository and improve the large language model's ability to locate context fragments, thereby better completing repository-level retrieval enhancement tasks.

[0006] According to a first aspect, a method for context retrieval based on a code repository is provided, wherein the code repository includes a plurality of repository files, and the method comprises:

[0007] A first question text is received.

[0008] The first question text is rewritten, and a keyword search is performed based on the obtained rewritten text to obtain a number of first warehouse files matching the rewritten text and place them into a first set.

[0009] According to the file name of the warehouse file, target reasoning is performed using a pre-trained target model to obtain several supplementary warehouse files related to the rewritten text and classify them into the first set.

[0010] The first set is screened to retain repository files that meet a first criterion, the first criterion being defined based on a summary relevance of the repository files to the rewritten text.

[0011] Divide any warehouse file in the first set into several context segments, and select target context segments that meet a second criterion, where the second criterion is defined based on the content relevance of the context segment to the first question text. The target context segment is used for downstream task processing related to the first question text.

[0012] According to an implementation, the rewritten text includes a plurality of first identifiers and a first description representing a code entity; the target reasoning includes:

[0013] The first identifiers and / or first descriptions are input into a first model so that the first model can infer a plurality of supplementary warehouse files having the code entity definitions based on the file names of the warehouse files.

[0014] In one scenario of the above implementation, the first model is pre-trained based on a target task, where the target task is used to predict the naming convention of a repository file having code entity definitions.

[0015] In one scenario of the above implementation, the code entity includes at least one of the following: a module, a package, a class, a function, and a variable.

[0016] According to one implementation, the target reasoning includes:

[0017] A subtree search is performed in a pre-constructed repository tree to determine a first subtree containing existing repository files in the first set, wherein the nodes of the repository tree contain the file names of repository files in the code repository, and the edges of the repository tree represent directory relationships between repository files.

[0018] The first subtree is input into the second model so that it can infer and obtain the supplementary warehouse file related to the rewritten text in the first subtree.

[0019] In one scenario of the above implementation, the subtree search includes:

[0020] A first node is determined, which is the most recent common ancestor of the existing repository files in the first set in the repository tree.

[0021] Taking the first node as the root, the first subtree is extracted in the warehouse tree.

[0022] According to an implementation manner, screening the first set includes:

[0023] For any second repository file in the first set, summary information is determined based on the content of the second repository file.

[0024] A first vector similarity between the summary information and the rewritten text is calculated.

[0025] If the first vector similarity is less than a first similarity threshold preset in the first standard, the second warehouse file is removed from the first set.

[0026] In one scenario of the above implementation, the second warehouse file is a code file, and the determining summary information includes:

[0027] An abstract syntax tree corresponding to the second repository file is obtained, wherein the abstract syntax tree includes a definition node representing a code entity in the second repository file, and the definition node includes a definition header and a definition body of the corresponding code entity.

[0028] For any definition node in the abstract syntax tree, determine its code entity summary, which includes the definition header of the node and the definition body that satisfies a first constraint; the first constraint is used to limit the number of tokens contained in the text to be less than a preset first quantity threshold.

[0029] The code entity summaries corresponding to the respective definition nodes in the abstract syntax tree are concatenated to obtain summary information of the second warehouse file.

[0030] In one scenario of the above implementation, the second warehouse file is a non-code text file, and the determining summary information includes:

[0031] The content of the second warehouse file is segmented based on a first rule to obtain a plurality of text blocks, wherein the first rule includes a first text paragraph number as a unit segmentation length.

[0032] For any text block of the second warehouse file, extract its text block summary.

[0033] The text block summaries corresponding to the plurality of text blocks are concatenated to obtain summary information of the second warehouse file.

[0034] According to one implementation, if the repository file is a code file, the context fragment is a definition of a code entity in the repository file.

[0035] If the warehouse file is a non-code text file, the context fragment is a text block obtained by performing text segmentation based on the length of the first text line number preset in the second rule.

[0036] In one scenario of the above implementation, the second rule also includes that there is a second number of overlapping text lines between adjacent text blocks.

[0037] According to one implementation, selecting the target context segment that meets the second criterion includes:

[0038] For any context segment, the second vector similarity between its content and the first question text is calculated.

[0039] If the second vector similarity is higher than a second similarity threshold preset in the second standard, it is determined as a target context segment.

[0040] In one scenario of the above implementation, the method further includes: sorting the obtained target context segments based on the corresponding second vector similarities.

[0041] According to a second aspect, a device for performing context retrieval based on a code repository is provided, wherein the code repository includes a plurality of repository files, and the device includes:

[0042] The receiving module is configured to receive a first question text.

[0043] The retrieval module is configured to rewrite the first question text, perform keyword search based on the obtained rewritten text, obtain several first warehouse files matching the rewritten text, and classify them into a first set.

[0044] The reasoning module is configured to perform target reasoning using a pre-trained target model according to the file name of the warehouse file, obtain several supplementary warehouse files related to the rewritten text, and classify them into the first set.

[0045] The first screening module is configured to screen the first set to retain warehouse files that meet a first criterion, wherein the first criterion is defined according to a summary relevance between the warehouse file and the rewritten text.

[0046] The second screening module is configured to divide any warehouse file in the first set into a number of context segments, and select a target context segment that meets a second standard, wherein the second standard is defined based on the content relevance of the context segment to the first question text, and the target context segment is used for downstream task processing related to the first question text.

[0047] According to a third aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0048] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0049] In summary, the above method and device disclosed in the embodiments of this specification can be used to analyze the code warehouse based on the question text. During the analysis process, a quick search based on the file name is first performed, and then a similarity analysis based on the content is performed. This can not only improve the analysis speed of the large language model on the code warehouse, but also improve the accuracy of the large language model's understanding of the code structure, and gradually and deeply search the code warehouse, so that relevant context fragments can be accurately retrieved for the question text. The embodiments of this specification can improve the understanding and retrieval capabilities of the large language model on the code warehouse, and help the model better complete the code warehouse level context retrieval task. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic diagram of a method framework for context retrieval based on a code repository disclosed in this specification;

[0052] Figure 2 A flowchart of a method for context retrieval based on a code repository provided according to an embodiment of this specification;

[0053] Figure 3 A schematic diagram of a warehouse tree provided for an embodiment of this specification;

[0054] Figure 4A A schematic diagram of an exemplary subtree construction provided according to an embodiment of this specification;

[0055] Figure 4B A schematic diagram of an exemplary subtree search provided according to an embodiment of this specification;

[0056] Figure 5 A schematic diagram of an exemplary abstract syntax tree provided according to an embodiment of this specification;

[0057] Figure 6 A schematic diagram of extracting a code entity summary according to an embodiment of this specification;

[0058] Figure 7A schematic diagram of a device for performing context retrieval based on a code repository according to an embodiment of the present specification. DETAILED DESCRIPTION

[0059] The solution provided by the embodiments of this specification is described below in conjunction with the accompanying drawings.

[0060] In one or more embodiments of this specification, the Python programming language will be used as an example to describe the process and defects of the large language model performing relevant context retrieval for the question text, and combined with the Python programming example, the technical solution that can solve the defects is explained. It should be noted that although Python is used as an example for description in some embodiments of this specification, it does not represent a limitation on the application scenarios and technical tools. The technical concepts embodied in the various embodiments of this specification can be applied to various code projects / code repositories constructed by large language models that support code context retrieval and the programming languages ​​they support.

[0061] As mentioned above, in order to improve the efficiency of developers in the R&D process, large language models (LLMs) have been widely used to solve various coding tasks in software engineering projects. Thanks to the massive amount of code text training, LLMs have shown certain potential in code understanding, searching, analysis, and generation.

[0062] Among many code tasks, there are many problem queries described in natural language (NL). Currently, most LLMs (or pre-trained large code models) generally adopt the NL-to-Code processing paradigm, which lacks the ability to integrate additional code context. However, in many complex software engineering projects, directly processing problem queries without additional code context information will face many challenges. For example, when faced with an unfamiliar code base or when there is a lack of reference information for related code in the problem query, the reasoning effect of LLM will be greatly reduced. In addition, based on the parameter knowledge learned during pre-training, it will also become very difficult to solve problem queries in specific software engineering. For example, LLM cannot follow up on the real-time updates of the public code base, cannot grasp the code features in the private code base, and so on.

[0063] Retrieval enhancement generation technology RAG can help LLM retrieve and integrate relevant documents (context) required in the reasoning process, thereby effectively improving the processing ability of question queries. Specifically, when LLM needs to answer a question query, RAG can help LLM to first retrieve context fragments related to the question query from a huge external knowledge base and pass them to LLM; then, LLM can use these context fragments to guide its downstream reasoning tasks for question queries, so as to effectively answer the question query and improve the quality and accuracy of the prediction.

[0064] In the field of software engineering, code-related external knowledge bases usually refer to code repositories that store source files. Context snippets can be code snippets, documents, or metadata related to problem-solving queries, such as "path / to / test.py:10-20". Code repositories contain a large number of code files (e.g., py files, cpp files, java files, html files, etc.) and non-code text files (e.g., txt files, md files, resource files, etc.) related to specific software engineering projects. There must be a certain semantic gap between these source files written in different languages ​​and problem queries described in natural languages. Therefore, in the context retrieval process of RAG, it is a very challenging task to accurately understand the true intention behind the problem query text and convert it into the retrieval operation of the corresponding context snippet in the code repository.

[0065] In some related technologies, contextual retrieval is regarded as a standard keyword search problem and is implemented through search engines. These search engines are usually built on top of a pre-stored vector database, which uses distributed vectors to capture and represent the features of context fragments, and matches and retrieves based on the vector similarity between the keywords in the question query text and the context fragments. However, this method overly relies on the vector's ability to express the features of a single context fragment. When faced with code repository source files with complex associations, it cannot understand the structural associations between repository files or code entities, and it is often difficult to obtain accurate retrieval results, affecting the efficiency and accuracy of contextual retrieval.

[0066] In other related technologies, the Reason and Act (ReAct) paradigm is adopted to perform contextual retrieval with the help of the powerful semantic understanding ability of the large language model. During the retrieval process, the large language model is guided to generate an iterative retrieval plan, and the reasoning trajectory and actions related to the question query are generated in an interwoven manner, so that the large language model can perform dynamic reasoning, create and adjust the execution plan of the action (reasoning → action), and interact with the external knowledge base to incorporate relevant information into the reasoning (action → reasoning), and gradually execute to approach the best retrieval result. When necessary, these methods will also integrate knowledge graph technology to model the code repository and assist retrieval through graph search. However, this technology relies more on the planning ability of the large language model for the retrieval plan. In some scenarios, such as processing highly abstract or ambiguous question queries, retrieval bias is prone to occur, resulting in the large language model being unable to correctly understand the deep semantics of the question query or the code repository, making the retrieval process difficult to control and unable to guarantee the retrieval accuracy of the context fragment, which has an adverse impact on the subsequent code task processing. Moreover, the application of knowledge graphs will further reduce the generalization of this method.

[0067] It can be seen that there are various defects in the relevant technologies in dealing with the context retrieval task at the code repository level. Although the relevant context can be quickly located by using a keyword search engine for retrieval, its accuracy is low; and although more accurate retrieval results can be obtained by retrieval through dynamic reasoning, the retrieval process is complicated, difficult to control, and has poor generalization. The inventor found through research that although the accuracy of keyword retrieval is not ideal, the context results returned are often closer to the target context required to solve the problem query, that is, keyword retrieval can capture some real results and contexts close to the real results. Based on this discovery, with the help of the powerful semantic understanding ability of the large language model, on the basis of keyword retrieval, the context obtained by the retrieval can be further supplemented with retrieval to make up for the context that may be missed in the keyword retrieval, and then these contexts can be carefully screened and sorted, so that the context fragments most relevant to the problem-solving query can be more accurately determined, thereby improving the overall retrieval effect.

[0068] In summary, in order to solve the above technical problems, the inventors propose a method for contextual retrieval based on code repositories, which can enhance the ability of the large language model to understand the repository files in the code repository and improve the ability of the large language model to locate the code context, thereby better completing the repository-level retrieval enhancement task. Specifically, Figure 1 The present invention discloses a framework diagram of a method for context retrieval based on a code repository in an embodiment. Referring to the accompanying drawings, the method can be generally divided into two stages: a file retrieval stage and a context fragment retrieval stage.

[0069] In the file retrieval stage, the question query text is rewritten to eliminate redundant natural language expressions and extract core keywords and key descriptions. Subsequent keyword retrieval operations can be performed based on the rewritten text, and warehouse files matching the rewritten text are retrieved in the code warehouse, and these selected warehouse files are included in the retrieval result set (i.e., the first set). As mentioned above, the context results obtained by relying solely on keyword search engines are not accurate. Therefore, LLM can be used to perform supplementary searches on the code entities involved in the rewritten text based on the file names of the warehouse files in the code warehouse, and the selected warehouse files are added to the first set, thereby enriching the retrieval result set. Next, the summary information of each retrieved warehouse file in the first set can be used to calculate the summary relevance with the rewritten text to screen out warehouse files that are not sufficiently related to the rewritten text, thereby effectively reducing the result errors that may be introduced in the early retrieval process.

[0070] In the context retrieval stage, the contents of each repository file retrieved from the first set are divided into several context fragments. The content relevance between each context fragment and the question query text is analyzed, and those context fragments whose relevance meets certain standards are screened out. These screened context fragments can be determined as target contexts to complete the context retrieval task for the code repository.

[0071] Following the above technical ideas, Figure 2 In FIG. 1 , a flowchart of a method for context retrieval based on a code repository according to an embodiment of the present specification is shown. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 In one embodiment, the code repository includes a plurality of repository files, and the method includes at least the following steps. S201: Receive a first question text. S203: Rewrite the first question text, and perform keyword retrieval based on the obtained rewritten text to obtain a plurality of first repository files matching the rewritten text, and classify them into a first set. S205: Perform target reasoning based on the file name of the repository file using a pre-trained target model to obtain a plurality of supplementary repository files related to the rewritten text, and classify them into the first set. S207: Filter the first set to retain repository files that meet a first standard, and the first standard is defined according to the summary relevance of the repository file to the rewritten text. S209: Divide any repository file in the first set into a plurality of context fragments, and select a target context fragment that meets a second standard, and the second standard is defined according to the content relevance of the context fragment to the first question text, and the target context fragment is used for downstream task processing related to the first question text.

[0072] The specific implementation methods of the above steps will be described in detail below with reference to the accompanying drawings.

[0073] The code repository may be a general code repository consisting of multiple project source files, or a code repository containing a specific software project, or a code repository used by a private project, and this specification does not make specific limitations on this.

[0074] In a code repository, the files contained are collectively referred to as repository files. These files are of various types, including but not limited to: code files that define code entities and record code logic, and annotated text files written in natural language. The code entity includes at least one of the following: module, package, class, function, variable.

[0075] Code files can be program codes written in any programming language, such as C++, Java, Python, etc. In some software projects that apply mixed programming methods, code files can also contain program codes written in multiple programming languages. Non-code text files are usually recorded in natural language, such as txt files for descriptions, md files written in markup languages, resource files that record application configuration information, etc.

[0076] In the embodiments of this specification, Python code will be used as an example for explanation. However, it should be understood that one or more embodiments in this specification are intended to provide a method for context retrieval based on a code repository, and are not limited to a specific programming language. The method provided by the embodiments of the present invention can be applied to any scenario related to the technical concept provided by the embodiments of the present invention.

[0077] In step S201, a first question text is received.

[0078] The first problem text may be a natural language description of various problems related to the source code in the code repository, such as: code location, error repair, adding new functions, updating dependencies, etc. Depending on the problem, the first problem text may include a problem description corresponding to the problem. For example, when code location is required, the first problem text may include a functional description related to the code entity to be located; when error repair is required, the first problem text may provide error information and error output related to program running errors; when code for adding new functions needs to be generated, the first problem text may include a functional description of the new function.

[0079] The first question text may also include text content in various forms. For example, the description of the problem may be given in the form of natural language, and the example of the program code snippet may be given in the target programming language. In subsequent steps, the first question text containing text content in various formats may be processed by means of the semantic understanding ability of the large language model. The embodiments of this specification do not specifically limit the text format of the first question text.

[0080] Next, in step S203, the first question text is rewritten, and a keyword search is performed based on the obtained rewritten text to obtain a number of first warehouse files matching the rewritten text and place them into a first set.

[0081] The first question text given by the user does not have a fixed writing format or paradigm, so it can be presented in a variety of different description forms. Usually, the first question text will have problems such as too brief description, ambiguous expression or lack of necessary background information. These problems may make it difficult for LLM to accurately capture the deep semantics of the first question text during the context retrieval process. In order to bridge the problem of sparse semantics of the first question text, in this step, the first question text can be appropriately rewritten to enrich or refine the semantic content of the text, making it easier for LLM to understand and process.

[0082] In some embodiments, a specific technical method may be used to perform the rewriting task. For example, Query2Doc, HyDE and other technologies may be used to semantically expand and refine the first question text. These technologies can effectively convert the original question text into a more detailed and clear version, thereby improving the accuracy of subsequent LLM understanding.

[0083] In other embodiments, a more concise and direct method than Query2Doc, HyDE and other technologies can be used to perform the rewriting task. Specifically, the large language model LLM can be used to reason and summarize the first question text. During the reasoning process, the LLM is guided to focus on the core of the question text and ignore non-critical information to ensure that the rewritten text can accurately reflect the core semantics of the question. In this embodiment, complex text processing steps can be avoided, and the semantic understanding ability of the LLM can be directly used to rewrite the first question text to simplify the process.

[0084] After the rewriting is completed, the rewritten text can be used to perform keyword search in the code warehouse. Through the search, the first warehouse files that match the rewritten text can be found, and these files are included in the first set as search results.

[0085] In some embodiments, the keyword search can be a text word frequency search method such as BM25, or other search methods that do not involve vector similarity comparison. Vector similarity is not used for search and matching here because in the contextual search scenario of the code repository, problem queries usually involve keywords representing specific code entities. These keywords are often some kind of naming and code given by developers in the repository file. They have strong encoding characteristics and contain very little natural semantic information. In this context, the use of a search method based on vector similarity comparison may not only introduce unnecessary semantic ambiguity, thereby affecting the accuracy of the search, but may also increase the computational burden and reduce the search efficiency. Therefore, in order to avoid these potential problems, a more direct and efficient word frequency search method can be used in keyword search.

[0086] After text rewriting and keyword retrieval, a first set is obtained, which includes several warehouse files related to the rewritten text (the first warehouse files). In the next step S205: according to the file name of the warehouse file, target reasoning is performed using the pre-trained target model to obtain several supplementary warehouse files related to the rewritten text and classify them into the first set.

[0087] As mentioned above, although keyword search can quickly obtain context related to the question query, its accuracy is low. This means that the first set may contain some first warehouse files that are not highly relevant to the question query, and may also miss some first warehouse files that are highly relevant to the question query. In other words, the search results of the first set may be neither comprehensive nor accurate enough, and further searches are needed to complete and optimize them.

[0088] In this step, the pre-trained target model can be used to reason about each code entity mentioned in the rewritten text. The reasoning process involves analyzing the file names of the repository files in the code repository, with the goal of finding the repository files that contain the definitions of these code entities. The repository files (the supplementary repository files) retrieved by this reasoning method will be integrated into the first set containing the previous search results, thereby improving the overall coverage of the search.

[0089] According to an implementation, based on the identifier of the code entity, the warehouse file that may contain its definition can be inferred. In this implementation, the rewritten text contains several first identifiers and first descriptions representing the code entity. In the process of performing the target reasoning, the several first identifiers and / or first descriptions are input into the first model so that it can infer several supplementary warehouse files with the definition of the code entity based on the file name of the warehouse file. Specifically, the identifier refers to a string used to uniquely identify a specific code entity. It is not difficult to understand that, corresponding to the example of the code entity above, the first identifier may include but is not limited to the following: module name, package name, class name, function name, variable name. In addition, the rewritten text may also contain a first description presented in natural language, which corresponds to a specific code entity. For example, the first description "the date of instance A" may correspond to the code entity identifier "A.Date", and the first description "the separability of the matrix" may correspond to the code entity identifier "separate". In this implementation, the semantic understanding ability of the first model can be used to parse the first description in the rewritten text and derive the first identifier of the corresponding code entity. These identifiers, together with the identifiers directly contained in the rewritten text body, are inferred by the first model to identify warehouse files that may contain corresponding code entity definitions. These files will be determined as the supplementary warehouse files and integrated into the first set previously established.

[0090] In a specific example, the first model can be a large language model LLM. The first model is pre-trained based on a target task, and the target task is used to predict the naming conventions of warehouse files with code entity definitions. Specifically, the first model is pre-trained on a large amount of general data and code data, so that it can deeply understand and master various common code entities, naming conventions of warehouse files, and the habits followed by developers when naming. Through pre-training, the first model can not only recognize standard programming terms and grammatical keywords, but also understand the unique file naming patterns in different programming languages, or variants of these naming patterns. When processing actual retrieval tasks, the first model can capture the association between the file name and the code entity, and infer the file name of the warehouse file containing the code entity definition.

[0091] According to another implementation, the file structure relationship between the warehouse files can be considered, and the adjacent files of each warehouse file in the previous search result set (the first set) can be inferred to supplement the first set. In this implementation, a subtree search can be performed in the pre-constructed warehouse tree to determine the first subtree containing the warehouse files in the first set, the nodes of the warehouse tree contain the file names of the warehouse files in the code warehouse, and the edges of the warehouse tree represent the directory relationship between the warehouse files; then the first subtree is input into the second model, so that it can be inferred to obtain the supplementary warehouse files related to the rewritten text in the first subtree. Specifically, according to the introduction in the previous text, it can be understood that the content covered by the software engineering / code warehouse in reality is usually not limited to a single function body or a single source code file. The source code of the application is usually constructed as a code warehouse containing multiple interdependent files, modules, and libraries. Therefore, when the second model performs target reasoning, it is also necessary to fully understand the relationship between the warehouse files in the code warehouse. The warehouse files with associated relationships may also have a certain degree of correlation in their content. As mentioned above, although the result set (the first set) obtained through the keyword search in the aforementioned steps is not ideal in terms of accuracy, the results returned are often closer to the real target required to solve the problem query. Therefore, based on the previous search results, target reasoning can be performed on each warehouse file that has a file association with it, in order to retrieve the missed related warehouse files.

[0092] Next, the warehouse tree involved in the implementation is briefly introduced, and then the generation and target reasoning process of the first subtree is introduced in detail.

[0093] In this implementation, a complex code repository can be abstracted into a repository tree based on the file directory relationship, which can assist the second model in understanding the structure and semantic relationship between repository files. The file directory relationship represents the organization of repository files, such as the directory hierarchy between files, or the program architecture described by various code elements (such as modules, classes, functions, etc.) written in a programming language (such as init.py in a Python program). Figure 3An exemplary warehouse tree is shown. In the code warehouse, several warehouse files are organized according to a certain directory structure. As shown in the figure, the astropy folder contains both warehouse files, such as setup.py, and subfolders, such as modeling, and so on. Warehouse files are distributed in different hierarchical directories; it can be known that in some programming languages ​​(such as Python), source code files organized in folders constitute a code module, and in some programming languages ​​(such as C++), source code organized in folders represents a close logical call relationship between them. This code warehouse with a directory structure can be constructed as a file tree (shown on the right side of the figure) according to the directory hierarchy of each warehouse file, which is the warehouse tree. In the warehouse tree, the edges between each node represent the structural information of the code warehouse, that is, the directory hierarchy relationship between warehouse files.

[0094] It is understandable that the above content briefly introduces the repository tree by taking a code repository containing source files of a single project as an example. As mentioned above, in actual application scenarios, the code repository may contain source files of multiple projects. In this scenario, there may be multiple corresponding repository trees, which is not limited in the embodiments of this specification.

[0095] Up to now, the first collection contains previously retrieved warehouse files related to the rewritten text. In order to find the neighboring warehouse files of these files, a subtree can be constructed around the first collection, that is, a subtree is extracted from the warehouse tree, and the subtree can cover the existing warehouse files in the first collection.

[0096] In a specific scenario, the first node may be determined first, which is the most recent common ancestor of the existing repository files in the first set in the repository tree; and then the first subtree may be extracted from the repository tree with the first node as the root. Figure 4A The construction of an exemplary subtree is shown. The first set includes the warehouse files (two files shown in the figure) that are pre-retrieved and related to the rewritten text. In the warehouse tree (shown in the tree on the left side of the figure), the nearest common ancestor node of these files is found, which is the "modeling" node in the example, that is, the first node. With this node as the root, a subtree is extracted, such as the tree on the right side of the figure, which is the first subtree. It can be seen that the first subtree includes neighboring nodes of each warehouse file node in the first set, and these neighboring nodes have a relatively close code structure relationship with the previously retrieved files.

[0097] Based on the obtained first subtree, the second model is used to perform target reasoning. As mentioned above, the target reasoning is performed based on the file name of the warehouse file, in order to retrieve warehouse files related to the rewritten text in the first subtree but missed in the previous search, and add them to the first set as supplementary warehouse files. Figure 4B In the subtree search example shown, the second model searches in the first subtree and finds that the neighboring warehouse file "test_compound.py" is also a warehouse file related to the rewritten text. Therefore, the file can be added to the first set.

[0098] The above two implementations, one uses the first model to infer the repository file containing the code entity definition in the rewritten text, and the other uses the second model to infer the neighboring repository files related to the rewritten text in the repository tree, both of which can supplement the search result set (i.e., the first set). In some embodiments, the methods outlined in the above two implementations can also be used simultaneously to supplement the first set, performing both reasoning based on file name and reasoning based on file directory relationship.

[0099] It should be noted that the various implementations of the supplementary retrieval mentioned above can also be implemented using the strategy of static program analysis. However, given that static program analysis relies on strict matching rules and is strongly bound to a specific programming language, it may result in a lower hit rate for supplementary retrieval and weaken generalization. Therefore, in the above implementation, a strategy of reasoning based on file names using a large language model is adopted. However, this does not mean that there is a limitation on the retrieval strategy, and any strategy that can effectively retrieve warehouse files containing code entity definitions can be equally applicable to the above implementations, and the embodiments of this specification will no longer list them in detail one by one.

[0100] After the above steps are processed, by performing a supplementary search, many warehouse files related to the rewritten text can be included in the first set, which makes up for the shortcomings of the previous keyword search in terms of coverage. However, as mentioned above, the error problem in the keyword search process still needs to be further resolved. In the aforementioned steps, the reasoning and retrieval performed are based on the warehouse file name or file directory structure, without in-depth analysis of the specific content of the warehouse file. Next, the previous search results can be further analyzed based on the file content to identify and filter out warehouse files with a low correlation with the rewritten text, thereby further improving the accuracy of the search results.

[0101] In step S207: the first set is screened to retain warehouse files that meet a first criterion, where the first criterion is defined based on a summary relevance of the warehouse files to the rewritten text.

[0102] In this step, a large language model LLM can be used to perform relevance analysis on the warehouse file. In specific practice, especially in some large software engineering projects, the content contained in the warehouse file is usually large (some code files can even reach tens of thousands of lines). When calculating the relevance of the warehouse file content, LLM usually decomposes the file content into tokens for semantic analysis, and the size of the file content will directly affect the number of tokens. Tokens are the basic units of text processing by LLM, usually representing a word, punctuation mark, and other text elements. The larger the file content, the more tokens it contains, which will cause the amount of information that LLM needs to process during semantic analysis to increase significantly, reducing the analysis efficiency. Therefore, considering the analysis efficiency of LLM, in this step, the summary relevance with the rewritten text can be calculated based on the summary information of the warehouse file to ensure fast and accurate semantic analysis.

[0103] According to one implementation, for any second warehouse file in the first set, summary information is determined based on its content; a first vector similarity between the summary information and the rewritten text is calculated; if the first vector similarity is less than a first similarity threshold preset in the first standard, the second warehouse file is removed from the first set.

[0104] In this implementation, summary information can be extracted based on the specific content of the second warehouse file. This involves an in-depth understanding of the file content, which may include: identifying key function descriptions, extracting main code entities, extracting annotations, etc. The extraction of summary information aims to capture the core features of the warehouse file and lay the foundation for the subsequent relevance calculation. Based on the summary information, the first vector similarity between it and the rewritten text can be calculated. The calculation of vector similarity can be implemented by a variety of algorithms, such as cosine similarity, Euclidean distance, etc., and the correlation between the summary information and the rewritten text (the summary correlation) is determined by judging the distance between the summary information and the rewritten text in the vector space. Then, the calculated first vector similarity is compared with a preset threshold (the first similarity threshold), and for those warehouse files whose first vector similarity is less than the first similarity threshold, they are removed from the first set. The first similarity threshold is pre-set when defining the first standard. It represents the minimum similarity requirement for the warehouse file to be sufficiently relevant to the rewritten text. If the calculated first vector similarity is lower than this threshold, it indicates that the warehouse file is not highly relevant to the rewritten text. In subsequent steps, the correct context fragment will not be generated, and it may even have a negative impact on the retrieval of the context fragment.

[0105] From the above introduction, it can be seen that after the summary relevance screening, the warehouse files retained in the first set are all closely related to the rewritten text, and these files will serve as the basis for extracting context fragments in the subsequent steps. It is not difficult to understand that the more accurate the calculation of the summary relevance, the higher the correlation between the warehouse files in the retained first set and the rewritten text, and thus, the more accurate the context fragments extracted from them will be. In other words, in this step, how to effectively extract summary information that can accurately reflect the core content of the warehouse file has a decisive influence on the subsequent summary relevance calculation. Therefore, the method of extracting summary information is particularly critical.

[0106] In a specific scenario, the second warehouse file is a code file, and its summary information can be extracted through the following steps:

[0107] An abstract syntax tree corresponding to the second warehouse file is obtained, wherein the abstract syntax tree contains definition nodes representing code entities in the second warehouse file, and the definition nodes include a definition header and a definition body of the corresponding code entity. For any definition node in the abstract syntax tree, its code entity summary is determined, which includes the definition header of the node and a definition body that satisfies a first constraint. The first constraint is used to restrict that the number of tokens contained in the text is less than a preset first quantity threshold. The code entity summaries corresponding to the respective definition nodes in the abstract syntax tree are concatenated to obtain summary information of the second warehouse file.

[0108] Before elaborating on the above steps, let's first briefly introduce the Abstract Syntax Tree (AST). An AST is a data structure that displays various code entities and code relationships in a code file in a tree structure, where each node represents a code entity in the source code, and the connections between nodes reflect the syntax hierarchy and code relationships between code entities, such as function calls, variable references, and so on. Figure 5 An exemplary abstract syntax tree is shown. Referring to the attached figure, in a syntax tree, definition nodes of code entities can be found (in the attached figure, functions and classes are used as examples, and other types of code entities can be applied by analogy). Different from other types of nodes (such as expression nodes representing arithmetic and logical operations), definition nodes are nodes representing declarations of code entities, and have definition headers and definition bodies of corresponding code entities.

[0109] By traversing the abstract syntax tree, you can quickly locate each code entity contained in the repository file through the definition nodes in it. Based on the code entity definition content, extract its code entity summary. Figure 6An exemplary code entity summary extraction schematic is shown. Referring to the attached figure, taking the function code entity as an example, it can be divided into two cases for processing. One case is shown on the left side of the attached figure. When the number of words contained in the definition body of the code entity is too large (exceeding the preset first number threshold), the definition body is discarded and only the definition head is retained as the summary of the code entity. Another case is shown on the right side of the attached figure. The definition body of the code entity is relatively short, and the number of words contained does not exceed the preset first number threshold. At this time, it can be retained, and the complete definition (including the definition head and the definition body) is used as the summary of the code entity. It can be seen that in order to extract the summary of each code entity, in this scenario, a first constraint with a preset first number threshold is defined. This constraint is to ensure the conciseness of the extracted code entity summary, and the number of words in the summary is limited by the constraint. In this way, the core information of each code entity can be captured, and the redundancy of the summary caused by too long text can be avoided. After obtaining the summary of each code entity in the warehouse file, finally, these code entity summaries are spliced ​​together to form the summary information of the warehouse file (i.e., the second warehouse file). The summary information can present the core content of the repository file from a macro perspective. It not only includes an overview of each important code entity in the file, but also maintains the compactness and readability of the information due to the word number screening, providing a basis for efficient processing for subsequent analysis and comparison.

[0110] In another specific scenario, the second warehouse file is a non-code text file, and its summary information can be extracted by the following steps:

[0111] The content of the second warehouse file is segmented based on a first rule to obtain a plurality of text blocks, wherein the first rule includes a number of first text paragraphs as a unit segmentation length. For any text block of the second warehouse file, a text block summary is extracted. The text block summaries corresponding to the plurality of text blocks are concatenated to obtain summary information of the second warehouse file.

[0112] In this scenario, the content of the second warehouse file is segmented based on natural paragraphs, mainly considering the natural structural rules of text files for organizing information. Specifically, the number of natural paragraphs contained in each text block is fixed and is determined by the preset number of the first text paragraphs. For example, if the number of first text paragraphs is set to 3, then each text block will contain 3 consecutive natural paragraphs. In a text file, a natural paragraph is usually a complete sentence or sentence group with independent meaning, which revolves around a central idea or theme and is the basic unit of text structure. Segmenting text based on natural paragraphs can ensure that each text block is semantically coherent and complete.

[0113] Next, for each text block obtained by text segmentation, a summary extraction operation is performed. The first sentence of each text block can be extracted as the summary of the text block. The first sentence is usually the sentence that best summarizes the content of the text block, so it can effectively represent the theme and main points of the entire text block. Finally, the text block summaries corresponding to each text block of the second warehouse file are spliced. The summary information of the warehouse file is formed. The summary information extracted in this way not only retains the main structure and key information of the file content, but also greatly reduces the volume of the summary information in the form of a summary, which is convenient for rapid analysis in subsequent steps.

[0114] In the steps described above, the search results in the first set are first supplemented, and then further screened through profile analysis. After this series of operations, the warehouse files retained in the first set are closely related to the rewritten text. In this process, the core benchmark of the analysis is the rewritten text, which may only contain some key information fragments in the first question text. In order to ensure accurate comparison with the complete semantics, in the next stage, the context fragments can be extracted and sorted according to the semantic association between these files and the first question text, so as to obtain the final search results.

[0115] In step S209, any warehouse file in the first set is divided into several context segments, and a target context segment that meets a second criterion is selected. The second criterion is defined based on the content relevance between the context segment and the first question text. The target context segment is used for downstream task processing related to the first question text.

[0116] In this step, the contents of each warehouse file in the first set are divided into multiple context segments. These context segments can be divided according to the content structure of the file, or according to semantic coherence, and there is no specific limitation on this, but it is necessary to ensure that each context segment is logically self-contained. For example, if the warehouse file is a code file, the context segment can be the definition content of each code entity in the warehouse file; if the warehouse file is a non-code text file, the context segment can be each text block obtained by text segmentation with the length divided by the first number of text lines preset in the second rule.

[0117] When the warehouse file for context segmentation is a code file, the definition nodes can be traversed according to the method of parsing the abstract syntax tree mentioned in the aforementioned steps to obtain the definition content of each code entity of the warehouse file; or the code entity definition content of the warehouse file that has been parsed in the previous steps can be directly used; this embodiment of the present specification does not elaborate on this.

[0118] When the warehouse file to be divided into context fragments is a non-code text type, a text segmentation method based on specific rules can be used to process its content. Specifically, the file content can be divided into multiple independent text blocks according to a preset number of lines, and each text block is the context fragment. For example, the second rule defines that the number of first text lines is 5, then the first 5 lines of text in the warehouse file will constitute the first text block, the next 5 lines will constitute the second text block, and so on, until the end of the file.

[0119] In order to enhance the continuity and relevance of information between text blocks, in one example, the second rule may also include: between adjacent text blocks, there is an overlapping second number of text lines. That is to say, when the content of the warehouse file is divided into text blocks, each text block, in addition to containing text lines defined by the first number of text lines, will also share text lines defined by the second number of text lines with the previous text block. For example, the second rule defines that the first number of text lines is 10 and the second number of text lines is 2, then the first text block will contain text content from lines 1 to 10, and the second text block will start from line 9 and contain text content from lines 9 to 18, so that lines 9 and 10 will become overlapping lines between the two text blocks.

[0120] Next, these context fragments are screened by applying the second standard. The second standard is formulated based on the content relevance between the context fragment and the first question text. That is, the semantic relevance between each context fragment and the first question text is calculated, and the second vector similarity between its content and the first question text is calculated. If the second vector similarity is higher than the second similarity threshold preset in the second standard, it is determined as the target context fragment. The second similarity threshold defines the minimum similarity level that needs to be achieved between the context fragment and the first question text in order to be considered relevant. That is, if the second vector similarity of a context fragment exceeds this threshold, then it can be considered that the context fragment is highly relevant to the first question text in terms of content and can be confirmed as the target context.

[0121] In a specific scenario, the obtained target context segments can be sorted based on the similarity of the second vectors corresponding to each other. The purpose of sorting is to put the target context segment most relevant to the first question text at the top of the list so that it can be given priority in subsequent analysis or processing.

[0122] Finally, the selected target context fragments will be used for downstream tasks related to the first question text, including but not limited to: code location, error repair, new function addition, dependency update, etc.

[0123] According to one or more embodiments, a method for contextual retrieval based on a code repository is described in detail above. By adopting the above method provided in the embodiments of this specification, the analysis speed of the large language model on the code repository can be improved by first performing a quick search based on the file name, and then performing a similarity analysis based on the content as the analysis subject. At the same time, the accuracy of the large language model's understanding of the code structure can be improved, and the code repository can be retrieved step by step and in depth, so as to help the model better complete the code repository level contextual retrieval task, improve the retrieval accuracy, and correctly select the target context fragment for the question text.

[0124] In this specification, the word "first" in terms such as the first question text, the first warehouse file, etc., and the corresponding "second", "third" (if any), etc. in the text are merely for the convenience of distinction and description and do not have any limiting meaning.

[0125] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] Figure 7 Schematic diagram of a device for context retrieval based on a code repository according to an embodiment of this specification. The device 700 is deployed in a computing device, which can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 2 The code repository includes several repository files, and the device 700 includes:

[0127] The receiving module 701 is configured to receive a first question text.

[0128] The search module 702 is configured to rewrite the first question text, perform keyword search based on the obtained rewritten text, obtain several first warehouse files matching the rewritten text, and place them into a first set.

[0129] The reasoning module 703 is configured to perform target reasoning using a pre-trained target model according to the file name of the warehouse file, obtain several supplementary warehouse files related to the rewritten text, and classify them into the first set.

[0130] The first screening module 704 is configured to screen the first set to retain warehouse files that meet a first criterion, where the first criterion is defined based on a summary relevance between the warehouse files and the rewritten text.

[0131] The second screening module 705 is configured to divide any warehouse file in the first set into a number of context segments, and select a target context segment that meets a second criterion, wherein the second criterion is defined based on the content relevance of the context segment to the first question text, and the target context segment is used for downstream task processing related to the first question text.

[0132] According to another embodiment of the present specification, there is also provided a computer program product, including a computer program / instruction, which implements the aforementioned combination when executed by a processor. Figure 2 The steps of the method.

[0133] According to another embodiment, the present specification further provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the aforementioned combination is implemented. Figure 2 The steps of the method.

[0134] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0135] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for contextual retrieval based on a code repository, wherein the code repository includes a plurality of repository files, the method comprising: receiving a first question text; Rewrite the first question text, perform keyword search based on the obtained rewritten text, obtain a number of first warehouse files matching the rewritten text, and put them into a first set; According to the file name of the warehouse file, target reasoning is performed using a pre-trained target model to obtain a number of supplementary warehouse files related to the rewritten text and classify them into the first set; screening the first set to retain warehouse files that meet a first criterion, the first criterion being defined based on a summary relevance of the warehouse files to the rewritten text; Divide any warehouse file in the first set into several context segments, and select target context segments that meet a second criterion, where the second criterion is defined based on the content relevance of the context segment to the first question text. The target context segment is used for downstream task processing related to the first question text.

2. The method according to claim 1, wherein: The rewritten text includes a plurality of first identifiers and a first description representing a code entity; the target reasoning includes: The first identifiers and / or first descriptions are input into a first model so that the first model can infer a plurality of supplementary warehouse files having the code entity definitions based on the file names of the warehouse files.

3. The method according to claim 2, wherein: The first model is pre-trained based on a target task, where the target task is used to predict naming conventions for repository files having code entity definitions.

4. The method according to claim 2, wherein: The code entity includes at least one of the following: module, package, class, function, variable.

5. The method according to claim 1, wherein: The target reasoning includes: Perform a subtree search in a pre-built repository tree to determine a first subtree containing existing repository files in the first set, wherein the nodes of the repository tree contain file names of repository files in the code repository, and the edges of the repository tree represent directory relationships between repository files; The first subtree is input into the second model so that it can infer and obtain the supplementary warehouse file related to the rewritten text in the first subtree.

6. The method according to claim 5, wherein: The subtree search includes: Determining a first node that is the most recent common ancestor of the existing repository files in the first set in the repository tree; Taking the first node as the root, the first subtree is extracted in the warehouse tree.

7. The method according to claim 1, wherein: The screening of the first set includes: For any second warehouse file in the first set, determine summary information based on the content of the second warehouse file; Calculating a first vector similarity between the summary information and the rewritten text; If the first vector similarity is less than a first similarity threshold preset in the first standard, the second warehouse file is removed from the first set.

8. The method according to claim 7, wherein: The second warehouse file is a code file, and the determination summary information includes: Acquire an abstract syntax tree corresponding to the second repository file, wherein the abstract syntax tree includes a definition node representing a code entity in the second repository file, and the definition node includes a definition header and a definition body of the corresponding code entity; For any definition node in the abstract syntax tree, determine its code entity summary, which includes the definition header of the node and the definition body that satisfies a first constraint; the first constraint is used to restrict that the number of tokens contained in the text is less than a preset first number threshold; The code entity summaries corresponding to the respective definition nodes in the abstract syntax tree are concatenated to obtain summary information of the second warehouse file.

9. The method according to claim 7, wherein: The second warehouse file is a non-code text file, and the determination summary information includes: Performing text segmentation on the content of the second warehouse file based on a first rule to obtain a plurality of text blocks, wherein the first rule includes a number of first text paragraphs as a unit segmentation length; For any text block of the second warehouse file, extract the text block summary; The text block summaries corresponding to the plurality of text blocks are concatenated to obtain summary information of the second warehouse file.

10. The method according to claim 1, wherein: If the repository file is a code file, the context fragment is the definition of the code entity in the repository file; If the warehouse file is a non-code text file, the context fragment is a text block obtained by performing text segmentation based on the length of the first text line number preset in the second rule.

11. The method according to claim 10, wherein: The second rule also includes that there is a second number of overlapping text lines between adjacent text blocks.

12. The method according to claim 1, wherein: The step of selecting the target context segment that meets the second criterion includes: For any context segment, calculate the second vector similarity between its content and the first question text; If the second vector similarity is higher than a second similarity threshold preset in the second standard, it is determined as a target context segment.

13. The method according to claim 12, further comprising: The obtained target context segments are sorted based on the corresponding second vector similarities.

14. A device for performing context retrieval based on a code repository, wherein the code repository includes a plurality of repository files, the device comprising: A receiving module is configured to receive a first question text; A search module is configured to rewrite the first question text, perform keyword search based on the obtained rewritten text, obtain a number of first warehouse files matching the rewritten text, and classify them into a first set; an inference module configured to perform target inference using a pre-trained target model according to the file name of the warehouse file, obtain a plurality of supplementary warehouse files related to the rewritten text, and classify them into the first set; A first screening module configured to screen the first set to retain warehouse files that meet a first criterion, wherein the first criterion is defined according to a summary relevance of the warehouse files to the rewritten text; The second screening module is configured to divide any warehouse file in the first set into a number of context segments, and select a target context segment that meets a second standard, wherein the second standard is defined based on the content relevance of the context segment to the first question text, and the target context segment is used for downstream task processing related to the first question text.

15. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

16. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 13 is implemented.

Images