Example code search for enhanced model hints

Abstract

A user query to information about data of the codebase is answered by the large language model given a prompt that includes examples of code segments from the codebase that are similar to the user query. Code segments from a code library are associated with metadata including both natural language text and source code. Searching of examples of code segments from a code library is based on embedding of the code segments and associated metadata being highly similar to embedding of user queries and contexts.

Description

Background Technology

[0001] Large Language Models (LLMs) are machine learning models trained on massive datasets of text and / or source code, resulting in models containing billions of parameters. LLMs are used to perform a variety of tasks, such as natural language processing, text generation, machine translation, and source code generation. LLMs are based on deep learning neural networks, such as neural transformer models with attention.

[0002] Large language models are typically given user prompts, which include text in the form of questions, instructions, short paragraphs, and / or source code instructing the model to perform a task and / or the expected response. However, large language models perform poorly when the task involves code elements from a codebase that the model has not yet seen during training, producing vague and / or useless responses. Summary of the Invention

[0003] This summary is provided to introduce some concepts in a simplified form, which will be further described below in the detailed description. This summary is not intended to identify the principal or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] User queries about the codebase are answered by a large language model that was not trained on the codebase's source code. The large language model is given a prompt that includes the user query and context, which includes the codebase's identifier and similar examples from the codebase. These examples come from the codebase's segment table. The segment table includes segments from each file in the codebase and their associated metadata. A segment can include the file in the codebase, the file's class, and the file's method.

[0005] The code snippet table is indexed by embeddings for each combination of code snippets and their associated metadata. The codebase is searched using embeddings of the user query and context to find one or more code snippets and their metadata that are closest to the user query. The code snippet with the embedding closest to the user query and context is used as an example in the hints.

[0006] These and other features and advantages will become apparent from reading the detailed description below and consulting the associated drawings. It should be understood that the foregoing general description and the following detailed description are merely illustrative and do not limit the claimed aspects. Attached Figure Description

[0007] Figure 1 An exemplary system for generating a codebase segment table is illustrated.

[0008] Figure 2 An exemplary system for generating prompts for a large language model is illustrated, wherein the prompts are enhanced by examples.

[0009] Figure 3 It is an exemplary dialogue and model response with a large language model, which uses user prompts including queries, context, and examples.

[0010] Figure 4 The flowchart illustrates an exemplary method that searches for examples to enhance model hints and generates model hints.

[0011] Figure 5 This is a flowchart illustrating an exemplary method for generating a codebase segment table.

[0012] Figure 6 This is a flowchart illustrating an exemplary method for searching highly similar examples in a codebase segment table.

[0013] Figure 7 This is a schematic diagram illustrating a first exemplary operating environment.

[0014] Figure 8 This is a block diagram illustrating a second exemplary operating environment. Detailed Implementation

[0015] Overview

[0016] The cues provided to the large language model regarding data about the codebase include source code segments from the codebase and associated metadata, which are highly similar to the intent of the cues. The intent of the cues is the task the model wants to perform, which takes the form of a query and context. The query is a question, instruction, or short paragraph describing the task, and the context is the identifier of the codebase to which the query refers (e.g., location, URI, file path, etc.).

[0017] Code segments of files in the codebase are stored along with their corresponding metadata in a codebase segment table. These code segments are indexed by embeddings, which represent the code segment and its associated metadata. Query and context embeddings are used as search indexes to find highly similar embeddings from the codebase segment table. The top k embeddings closest to the query and context embeddings point to code segments and metadata from the codebase segment table, and are included as examples in hints for the large language model.

[0018] Large language models, trained on terabytes of source code and natural language, possess powerful reasoning capabilities. On one hand, they are pre-trained on vast amounts of source code and natural language text from a variety of publicly available sources. Using examples with high similarity to the query and context in hints allows large language models (code snippets from files not seen during training) to make more accurate inferences using information predicted during training. Hints are beneficial when the codebase is not publicly accessible and is unlikely to have been seen by the model during training.

[0019] Furthermore, including examples from a codebase allows the model to predict responses that include code elements from the codebase (such as method names, application programming interfaces (APIs), objects, variable names, etc.) and code references (such as local file paths). Including code elements and code references from the codebase makes the response more specific to the codebase the user's query points to.

[0020] Now let’s turn our attention to a more detailed description of the systems, devices, and methods used to provide hints for searching examples to enhance large language models.

[0021] system

[0022] Figure 1 The illustration depicts an exemplary system 100 for generating a codebase segment table. System 100 includes a codebase 102, a segment generator 104, a metadata generator 106, an encoder 108, a codebase segment table 110, and a large language model 120. Codebase 102 is a collection of related files for a source code component, program, or system, including files used for compiling, testing, maintaining, configuring, and / or building the source code component, program, or system, such as (but not limited to) script files, readme files, license information files, etc. Codebase 102 can be hosted on a version control system (i.e., a source code repository on GitHub / Git), a local repository, a cloud service, etc.

[0023] Code snippet generator 104 extracts code snippets from each file 112 of code repository 102. Code snippets are defined by syntax hierarchy 114. In one aspect, syntax hierarchy 114 defines code snippets as including entire files, entire classes, and single methods. Metadata generator 106 extracts metadata associated with code snippets 122. Metadata includes filename, local file path, Git commit branch or hash value, starting line number of the code snippet, class and namespace information for class methods and objects in the namespace, natural language code summary of the code snippet, and predictive query. The natural language code summary and predictive query 118 of the code snippet are generated by a large language model 120 given code snippet 116.

[0024] Encoder 108 takes an input sequence of tokens consisting of code segments and their metadata, and produces a fixed-length vector representation (referred to as an embedding or encoding). The embedding captures the semantic relationships between tokens and groups similar inputs closely together in the embedding space. In one respect, encoder 108 is an attention-only encoder-only neural transformer model. Encoder 108 generates an embedding for each code segment and its associated metadata, which is then used as an embedding index 124. Each code segment and its associated metadata 128 are stored in a code segment table 110. An embedding tree index 126 is used to access the corresponding code segment and its metadata.

[0025] Embedding tree index 126 is based on embeddings of code segments and their metadata and is used to perform fast searches of the codebase segment table. The codebase files are segmented into semantic code segments (such as methods, classes, and class methods) by a concrete syntax tree. These code segments are then annotated with metadata (such as file paths, namespaces, generated summaries, generated keywords, starting line numbers, and perhaps class names for class methods). These annotated code segments are then embedded, and the embeddings are fed into an Approximate Nearest Neighbor (ANN) tree data structure, which enables faster sorting of embedding queries. The ANN data structure or embedding tree index 126 retrieves the embeddings of the query / context and returns a sorted list of indexes representing code segments / metadata, with the most relevant code segments / metadata used as examples provided in the hints.

[0026] Figure 2 An exemplary system 200 for generating suggestions for a large language model is illustrated, the suggestions including one or more examples extracted from a codebase segment table. The system includes a code query engine 202 having an encoder 204, a search engine 206, a codebase segment table 208, a suggestion generator 210, and a large language model 212. The code query engine 202 receives queries and context 214 from an application or user interface. The code query engine 202 generates suggestions 216 for the large language model 212 to generate a response 218.

[0027] Encoder 204 transforms the query and context 214 into search index embeddings 220. Search engine 206 uses the search index embeddings 220 and the embedding tree index 224 of the code segment table 208 to find the top k highly similar embeddings. The metadata and code segments 226 associated with the top k highly similar embeddings are extracted and used as examples 222A to 222K.

[0028] Hint generator 210 generates hint 216. Hint generator 210 uses examples 222A to 222K, user query, context, and examples to form hint 216, which is then passed to the large language model 212. The large language model 212 returns a response 218 given hint 216.

[0029] On one hand, the large language model212 is an attention-based neural transformer model. Attention-based neural transformer models are a different type of machine learning model. Machine learning involves the use and development of computer systems capable of learning or adapting by analyzing patterns in data and deriving inferences from them using algorithms and statistical models, without following explicit instructions. Machine learning uses different types of statistical methods to learn from data and predict future decisions. Traditional machine learning includes classification models, data mining, Bayesian networks, Markov models, clustering, and visual data mapping.

[0030] Deep learning differs from traditional machine learning because it learns and interprets features and relationships between features through many hidden layers of neural networks and multiple stages of data processing. Deep learning embodies neural networks, unlike traditional machine learning techniques that do not use them. Neural transformer models are a type of deep learning that utilizes attention mechanisms. Attention instructs a neural network to focus on a subset of features or labels in an input sequence, thereby learning different representations from different positions of the labels in the input sequence. Neural transformer models handle the dependencies between their inputs and outputs through attention without using recurrent neural networks (RNNs) (e.g., Long Short-Term Memory (LSTM) networks) and convolutional neural networks (CNNs).

[0031] Attention-based neural transformer models have various configurations. On one hand, large language models are configured as attention-based encoder-decoder neural transformer models with a series of stacked encoder blocks coupled to a series of stacked decoder blocks. On the other hand, large language models consist only of stacked decoder blocks. Furthermore, large language models can be trained to perform different tasks and can be configured with different model sizes.

[0032] On one hand, large language models are pre-trained on natural language text and source code. Training large language models requires massive amounts of training data and computational resources, making it impossible for some developers to create their own models. Large language models consist of hundreds of millions of parameters (e.g., weights, embeddings) and are trained on terabytes of data. Examples of large language models include attention-based conversational pre-trained generative neural transformer models (i.e., the chat GPT model) provided by OpenAI, Codex CoPilot, and others.

[0033] Figure 3 The illustration depicts an exemplary prompt 300 to a large language model and a response 302 from the model. In one aspect, the large language model resides on a remote server and receives prompts via a network at an endpoint of the server hosting the large language model. The user and the large language model can engage in a dialogue within a single network session, consisting of several prompts and responses.

[0034] like Figure 3 As shown, the prompts include a user query and a context 304, as well as a single example 306. It should be noted that, although... Figure 3 Only a single example is shown, but the techniques described in this article are not limited to a single example and may include multiple examples.

[0035] The user query returned a 304 error due to the question "How do I build the BPE encoder?", and the 304 context included a URL to a location on GitHub (which contains the MinGPT codebase). https: / / Github.com / Karpathy / MinGPT Example 306 consists of a code segment (i.e., a Python method, decode) 316 and its metadata. The metadata includes the following items: the filename of the code segment 308 (mingpt / bpe.py), the class definition of the code segment 310, the class BPETOKENIZER that includes the code segment 312, the docstring of the decode method 314, the code digest 318, and a predictive developer query for the code segment 320.

[0036] The model's response 302 includes specific references found in the codebase. For example, the model's response includes the function name of the get_encoder() method in the file "mingpt / bpe.py" in the codebase, a detailed description of the get_encoder() method including the files and objects used by the method (e.g., encoder.json, vocab.bpe, the encoder object), an example of the method, and a description of the method's output.

[0037] Because examples are included in the prompts, the model's responses contain specific code elements from the codebase, even if the model was not trained on code from that codebase. Including examples in the prompts allows the model to produce more useful responses because it contains code elements from a specific codebase.

[0038] method

[0039] Attention now turns to the description of various exemplary methods utilizing the systems and devices disclosed herein. Operations for each aspect can also be described with reference to various exemplary methods. It is understood that, unless otherwise indicated, the representative methods do not necessarily have to be performed in the order presented or in any particular order. Furthermore, the various activities described with respect to the methods can be performed serially or in parallel, or in any combination of serial and parallel operations. In one or more aspects, the method illustrates operation for the systems and devices disclosed herein.

[0040] Figure 4 An exemplary method 400 of the system is illustrated. Initially, a codebase segment table is generated (box 402). [Go to...] Figure 5 The diagram illustrates an exemplary method 500 for generating a codebase segment table. The segment generator scans each file in the codebase (box 502). In addition to script files (i.e., markdown files) and documentation files, the codebase also contains source code files. Each file is categorized according to its type, such as source code files written in a programming language, script files written in a specific scripting language, or natural language text files.

[0041] Each file in the codebase is segmented into one or more segments (Box 504). Files containing natural language text or unsupported programming languages ​​are segmented into rolling windows. Rolling windows are calculated through a process where the file is first tokenized using a tokenizer, then segmented into segments with the same number of consecutive tokens, resulting in overlapping sets of segments due to offsets of the first segment's tokens. Script files containing code with syntax in unsupported programming languages ​​are segmented into segments containing headers or subheaders.

[0042] For source code files, the file is parsed into a concrete syntax tree (PPD). Each part of the PPD is extracted to form a code segment. The syntax hierarchy defines the parts of the PPD, which are included in the code segments. The file is represented by overlapping code segments defined by the syntax hierarchy. In one aspect, the syntax hierarchy identifies code segments as complete files, complete classes, and single class methods (collectively referred to as Box 504).

[0043] For each code segment (box 506), a code summary and a predicted query are generated for the code segment (box 508). The code summary is a natural language description of the code segment. The predicted query is a query that may result in a response that produces the code segment. Both the code summary and the predicted query are generated using a large language model given a code segment. The large language model is given a prompt that includes instructions to predict the code summary of the code segment and one or more predicted queries. The code segment and the predicted query become part of the metadata (collectively referred to as box 508).

[0044] Next, additional metadata is extracted from the file containing the code segment to annotate the code segment (Box 510). On one hand, the additional metadata extracted from the file includes the filename of the file containing the code segment, the local file path of the file containing the code segment, the Git commit branch or hash of the location of the file containing the code segment, and the starting line number of the code segment in the file containing the code segment. For class methods and objects within a namespace, class and namespace information is extracted and included in the metadata of the code segment. Class and namespace information includes the class name, the namespace name, and the parent class of the class.

[0045] The encoder then uses the metadata and code snippets to generate embeddings (box 512). The embeddings are used as indexes to search for metadata and code snippets in the codebase snippet table. The indexes, metadata, and code snippets are stored in the codebase snippet table (box 514).

[0046] Back Figure 4 Once the codebase segment table is created (box 402), it will be deployed in the inference system (box 404). The inference system is any software application that uses code from the codebase, such as (but not limited to) an integrated development environment, source code editor, source code repository, code search engine, or any browser-based service that involves viewing source code projects.

[0047] The inference system receives the user query and context (box 406). The search engine searches the codebase segment table for examples similar to the user query and context (box 408). [Go to...] Figure 6 It illustrates an exemplary method 600 for searching a codebase segment table for examples similar to user queries and contexts.

[0048] Given a user query and context, an encoder is used to generate embeddings (box 602). The search engine uses the embeddings of the user query and context to find the closest embedding from the codebase segment table. On one hand, the search uses an approximate nearest neighbor search algorithm on the embedding tree index to find the embedding most similar to the user query / context embedding in the codebase segment table. Metadata and code segments associated with the embedding most similar to the embedding of the user query and context are selected as an example (box 604).

[0049] Due to the syntactic hierarchy (including entire files and subsets of files), overlapping code may exist in examples. The search engine eliminates code segments already included in another example to utilize unique code segments (box 606). For example, if an example includes an entire file, and another example contains a method from that file, the example including the entire file is retained, while the example containing the method is eliminated. A separate graph data structure is maintained that can look up whether any given code segment / metadata index overlaps, and whether one contains the other. The unique example, including the code segment and its corresponding metadata, is then used in hints from the large language model (box 608).

[0050] Back Figure 4 Once an example is obtained (box 408), the suggestion generator constructs a suggestion that includes the query, context, and one or more examples (box 410). The suggestion is applied to the large language model (box 412). The large language model returns a response (box 414), which is output to or returned to the application or user interface that requested the query (416).

[0051] Figure 7 An exemplary inference system 700 utilizing a code query engine 710 is illustrated. For example... Figure 7 As shown, the inference system is a software development tool, such as an integrated development environment (IDE) 702. The IDE 702 includes a user interface 704, a source code editor 706, and a code query engine 710. The code query engine 710 can be a plugin, an add-on, or a component that incorporates query features into the IDE. The code query engine 710 includes an encoder 712, a search engine 716, a codebase segment table 718, a hint generator 722, and a large language model 726.

[0052] Users of IDE 702 can utilize code query engine 710 to find answers to questions related to a specific codebase. Code query engine 710 receives user queries and context 708 from user interface 704 and generates hints 726 from a large language model 726. Encoder 712 generates a search index embedding 714 given the user query and context, which is used by search engine 716 to find examples 720A to 720K from codebase segment table 718. Hint generator 722 then generates hints 726, which include the user query, context, and one or more examples. Large language model 728 generates a response given the hints, which is then returned to the user interface.

[0053] The techniques described in this article can be used in any chatbot environment that requires the conversational agent to have the ability to view the entire codebase. This includes not only code searching, but also any relevant tasks within the library context, including (but not limited to) test generation, example generation, method generation, code review, bug fixing, and code optimization.

[0054] Exemplary operating environment

[0055] Attention now turns to the discussion of exemplary operating environments. Figure 8 An exemplary operating environment 800 is illustrated, in which a user computing device 802 is communicatively coupled to one or more computing devices 804 hosting a large language model via a network 806.

[0056] Computing devices 802 and 804 can be any type of electronic device, such as (but not limited to) mobile devices, personal digital assistants, mobile computing devices, smartphones, cellular phones, portable computers, servers, server arrays or server farms, web servers, network servers, blade servers, internet servers, workstations, microcomputers, mainframe computers, supercomputers, network devices, web devices, distributed computing systems, multiprocessor systems, or combinations thereof. The operating environment 800 can be configured in a network environment, a distributed environment, a multiprocessor environment, or in an independent computing device with access to remote or local storage devices.

[0057] Computing devices 802 and 804 may include one or more processors 808 and 830, one or more communication interfaces 810 and 832, one or more storage devices 812 and 836, one or more input / output devices 814 and 834, and one or more memory devices 816 and 838. Processors 808 and 830 may be any commercially available or custom-made processor and may include dual-microprocessor and multiprocessor architectures. Communication interfaces 810 and 832 facilitate wired or wireless communication between computing devices 802 and 804 and other devices. Storage devices 812 and 836 may be computer-readable media that do not contain propagating signals, such as modulated data signals transmitted via a carrier wave. Examples of storage devices 812 and 836 include (but are not limited to) RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD), or other optical storage devices, cassette tape, disc tape, and disk storage devices, all of which do not contain propagating signals, such as modulated data signals transmitted via a carrier wave. The computing devices 802 and 804 may contain multiple storage devices 812 and 836. The input / output devices 814 and 834 may include keyboards, mice, electronic pens, voice input devices, touch input devices, displays, speakers, printers, and any combination thereof.

[0058] The memory device or memory 816, 838 can be any non-transitory computer-readable storage medium that can store executable programs, applications, and data. The computer-readable storage medium does not involve propagated signals, such as modulated data signals transmitted via a carrier wave. It can be any type of non-transitory memory device (e.g., random access memory, read-only memory, etc.), magnetic storage device, volatile storage device, non-volatile storage device, optical storage device, DVD, CD, floppy disk drive, etc., which does not involve propagated signals, such as modulated data signals transmitted via a carrier wave. Memory devices 816, 838 may also include one or more external storage devices or remotely located storage devices that do not involve propagated signals, such as modulated data signals transmitted via a carrier wave.

[0059] Memory devices 816 and 838 may contain instructions, components, and data. A component is a software program that performs a specific function and is also referred to as a module, program, component, and / or application. Memory device 816 may include an operating system 818, an encoder 820, a search engine 822, a prompt generator 823, a code segment table 824, a code library 825, an integrated development environment 826, a user interface 828, and other applications and data 829. Memory device 838 may include an operating system 840, a large language model 842, and other applications and data 844.

[0060] Computing devices 802 and 804 can be communicatively coupled via network 806. Network 806 can be configured as an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the public switched telephone network (PSTN), a conventional telephone service (POTS) network, a wireless network, a WiFi® network, or any other type of network or combination of networks.

[0061] Network 806 can employ a variety of wired and / or wireless communication protocols and / or technologies. Various versions of different communication protocols and / or technologies that the network can employ may include (but are not limited to) Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access 2000 (CDMA-2000), High-Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), Universal Mobile Telecommunications System (UMTS), Evolved Data Optimized (Ev-DO), Global Microwave Access Interconnection (WiMax), Time Division Multiple Access (TDMA), Orthogonal Frequency Division Multiplexing (OFDM), Ultra Wideband (UWB), Wireless Application Protocol (WAP), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), any part of the Open Systems Interconnection (OSI) model protocol, Session Initiation Protocol / Real-Time Transport Protocol (SIP / RTP), Short Message Service (SMS), Multimedia Messaging Service (MMS), or any other communication protocol and / or technology.

[0062] Technical effects and improvements

[0063] The disclosed subject matter relates to reducing user input on computing devices. A technical feature associated with solving this problem is including code snippets from a codebase that are similar to the query in the prompts of a large language model. The achieved technical effect is to reduce the number of prompts the user needs to make to obtain a useful and relevant response, thereby improving the user interface of the computing device.

[0064] The technique described in this paper is an improvement over solutions that fail to include semantically similar examples from the codebases involved in the hints provided by the large language model. The use of examples improves the accuracy of the response, which includes code elements and references from the codebase.

[0065] The search for code snippets similar to the query encompasses the metadata associated with those snippets. In addition to the source code, the metadata includes a natural language code summary. Including the code summary improves the model's predictions because, in the example for the prompt, not only natural language text but also the source code is used, rather than just the source code. The summary and keywords enhance the connection between the user query (which is natural language) and the code (which is not natural language) in the embedding space. Furthermore, file paths and other metadata allow the model to see the location of each code snippet within the codebase, thus enabling specific referencing of it in both the code and the natural language response.

[0066] Those skilled in the art will understand that the techniques disclosed herein are inherently digital. Operations such as searching for examples from codebase segments, generating suggestions, and applying large language models to generate answers to queries are inherently digital. Human thought cannot directly interface with a CPU or network interface card, or other processor, or with RAM or other digital storage devices to read or write the necessary data and perform the necessary operations disclosed herein.

[0067] The implementation is also considered to be scalable, capable of operating within strict timing constraints in production environments (e.g., integrated development environments), and capable of operating in test labs for production environments, rather than being a pure thought experiment.

[0068] in conclusion

[0069] A system is disclosed, including a processor and a memory. The memory stores a program configured to be executed by the processor. The program includes instructions that, when executed by the processor, perform actions including: receiving a query and context from a user interface, wherein the query relates to data in a codebase, wherein the context identifies the codebase; searching for examples from code segments in the codebase that are similar to the query and context, wherein the search is based on embeddings that are similar to embeddings associated with code segments in the codebase and metadata associated with the code segments; generating a hint for a large language model, wherein the hint includes the query, the context of the query, and examples; receiving a response from the large language model given the hint; and displaying the response in the user interface.

[0070] In one aspect, the metadata of a code segment includes a natural language summary of the code segment. In another aspect, the metadata of a code segment includes the filename associated with the code segment. In another aspect, the metadata of a code segment includes the class definition associated with the code segment, the class associated with the code segment, and the methods associated with the code segment. In another aspect, the metadata includes a query associated with the code segment. In another aspect, a code segment is a file, method, or class of a codebase. In another aspect, the large language model includes a neural transformer model with attention.

[0071] A computer-implemented method is disclosed, comprising: obtaining a query and context from a user interface, wherein the context identifies a codebase, and wherein the query includes a question related to the codebase; accessing a codebase segment table comprising multiple code segments, wherein code segments among the multiple code segments are accessed through embeddings of the code segments and associated metadata; searching the codebase segment table for code segments similar to the query, wherein the search is based on the embeddings of the query and context being highly similar to the embeddings of the code segments from the codebase segment table and associated metadata; obtaining an answer to the query from a large language model with a given prompt, wherein the prompt includes the query, context, and similar code segments and associated metadata; and returning the answer to the user interface.

[0072] In one aspect, a code segment includes a file, class, or method from a codebase. In another aspect, metadata includes a code summary of the code segment. In another aspect, metadata includes the filename of the code segment and the local file path of the file containing the code segment. In another aspect, metadata includes a predictive query for the code segment generated by the large language model. In another aspect, the metadata of a code segment includes the class definition associated with the code segment, the class associated with the code segment, and the method associated with the code segment. In another aspect, the large language model is a neural transformer model with attention. In another aspect, the answer includes at least one code element from the codebase or a reference to a file in the codebase.

[0073] A hardware storage device is disclosed having computer-executable instructions stored thereon, these instructions being structured to be executed by a processor of a computing device, thereby causing the computing device to perform actions, including: obtaining a query related to data in a codebase from a user interface, wherein the query includes an identifier of the codebase; accessing a large language model to generate a response to the query, wherein the large language model is given a prompt including the query, the identifier of the codebase, and an example from the codebase, wherein the example includes a code segment from a file in the codebase and associated metadata, wherein the example includes an embedding based on the code segment and the associated metadata, the embedding being similar to the embedding of the query and the identifier of the codebase; and returning the response to the user interface.

[0074] On one hand, the hardware device has computer-executable instructions that are structured to be executed by the processor of the computing device, thereby enabling the computing device to perform actions, including: storing code segments of files in a codebase in a codebase segment table, the code segments being associated with metadata and indexed by embeddings of the code segments and metadata; and searching the codebase segment table using the embeddings of the code segments and metadata.

[0075] On one hand, metadata includes a code summary of a code segment, a predictive query for the code segment, the filename of the file containing the code segment, the file path of the file containing the code segment, and / or namespace data associated with the code segment. On another hand, a code segment is a file, class, or method of a codebase. On yet another hand, a large language model is a neural transformer model with attention.

[0076] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims.

[0077] It should be understood that, unless otherwise instructed, the representative methods do not necessarily have to be performed in the order presented or in any particular order. Furthermore, the various activities described with respect to the methods can be performed serially or in parallel, or in any combination of serial and parallel operations. In one or more aspects, the method illustrates the operation of the systems and devices disclosed herein.

Claims

1. A system comprising: processor; as well as The memory stores a program configured to be executed by the processor, the program including instructions that, when executed by the processor, perform actions: Receive a query and context from the user interface, wherein the query is related to data in the codebase, and wherein the context identifies the codebase; Search the code segments of the codebase for examples similar to the query and context, wherein the search is based on the embedding similarity of the query and context to the embeddings associated with the code segments of the codebase and the metadata associated with the code segments; Generate suggestions for a large language model, wherein the suggestions include the query, the context of the query, and the example; Receive a response from the large language model given the prompt; as well as The response is displayed in the user interface.

2. The system of claim 1, wherein the metadata of the code segment includes a natural language summary of the code segment.

3. The system of claim 1, wherein the metadata of the code segment includes a filename associated with the code segment.

4. The system of claim 1, wherein the metadata of the code segment includes class definitions associated with the code segment, classes associated with the code segment, and methods associated with the code segment.

5. The system of claim 1, wherein the metadata includes queries associated with the code segment.

6. The system according to claim 1, wherein the code segment is a file of the code library, a method of the code library, or a class of the code library.

7. The system of claim 1, wherein the large language model comprises a neural transformer model with attention.

8. A computer-implemented method, comprising: Obtain queries and context from the user interface, wherein the context identifies the codebase, and wherein the query includes questions related to the codebase; Access a codebase segment table comprising multiple code segments, wherein the code segments are accessed through the embedding of the code segments and associated metadata; Search the code segment table for code segments similar to the query, wherein the search is based on the embedding of the query and context being highly similar to the embedding of code segments from the code segment table and associated metadata; The answer to the query is obtained from a large language model given a prompt, wherein the prompt includes the query, the context, and the similar code snippets and associated metadata; as well as The answer is returned to the user interface.

9. The computer-implemented method of claim 8, wherein the code segment comprises a file from the code library, a class from the code library, or a method from the code library.

10. The computer-implemented method of claim 8, wherein the metadata includes a code digest of the code segment.

11. The computer-implemented method of claim 8, wherein the metadata includes the filename of the code segment and the local file path of the file containing the code segment.

12. The computer-implemented method of claim 8, wherein the metadata includes a predictive query for the code segment generated by the large language model.

13. The computer-implemented method of claim 8, wherein the metadata of the code segment includes class definitions associated with the code segment, classes associated with the code segment, and methods associated with the code segment.

14. The computer-implemented method of claim 8, wherein the large language model is an attention-based neural transformer model.

15. The computer-implemented method of claim 8, wherein the response comprises at least one code element from the code library or a reference to a file of the code library.

Images