System and method for a code snippet search tool

Abstract

This application generally relates to search engines, and more particularly, to systems and methods for a code snippet search tool. Embodiments described herein provide a customized code search system that generates code search results from customized data sources, extracts code snippets from the code search results, and presents the code snippets via a user interface. In some embodiments, the search system employs a machine learning module to generate and highlight search results from different data sources, such as code examples in programming languages. To improve search efficiency, in response to a code search query, the search system can extract code snippets from search results from relevant sources within user interface elements such as user-selectable panels.

Description

Technical Field

[0001] Cross-reference This application claims priority to U.S. Non-Provisional Patent Application No. 18 / 330,225, filed on Jun. 6, 2023, which claims priority to U.S. Provisional Patent Application No. 63 / 349,855, filed on Jun. 7, 2022, and U.S. Provisional Patent Application No. 63 / 446,199, filed on Feb. 16, 2023, both of which are co-pending and by the same applicant.

[0002] This application is related to U.S. Non-Provisional Application No. 17 / 981,102, filed on Nov. 4, 2022, which is co-pending and by the same applicant.

[0003] All of the above applications are hereby expressly incorporated by reference in their entirety.

[0004] Technical Field This application generally relates to search engines, and more particularly, to systems and methods for a code snippet search tool that supports code search from within an integrated development environment (IDE).

Background Art

[0005] Search engines allow users to provide search queries and respond by returning search results. Search sites such as Google.com, Bing.com, and / or the like typically provide users with a list of search results from various types of data sources. For example, these existing search engines typically crawl web data to collect search results relevant to a search query. However, the user must visit each website to determine whether the results provide relevant information. This can be frustrating when searching for certain types of content, such as computer code when a programmer-user is compiling programming code in an IDE.

Brief Description of the Drawings

[0006]

Figure 1

[0007]

Figure 2

[0008]

Figure 3

[0009]

Figure 4

[0010]

Figure 5

[0011]

Figure 6

[0012]

Figure 7A

Figure 7B

Figure 7C

[0013]

Figure 8A

Figure 8B

Figure 8C

Figure 8D

[0014] The embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description. It should be understood that like reference numerals are used to identify like elements illustrated in one or more of the figures, and the displays therein are for the purpose of illustrating the embodiments of the present disclosure and not for the purpose of limiting them.

DETAILED DESCRIPTION OF THE INVENTION

[0015] This application generally relates to search engines, and more specifically, to systems and methods for a code snippet search tool.

[0016] As used herein, the term "network" can include any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or therewith.

[0017] As used herein, the term "module" can include a hardware or software-based framework that performs one or more functions. In some embodiments, a module can be implemented on one or more neural networks.

[0018] A search engine allows a user to provide a search query and returns search results in response. Search sites such as Google.com, Bing.com, and / or the like typically provide a user with a list of search results from all kinds of data sources. Some customized search systems provide a web-based platform that provides a customized search experience for individual users from different data sources. In one embodiment, the search system employs a machine learning module to generate and filter search results from all different data sources. For example, the search platform can take in user queries, user context information, and other context information to determine which data sources are relevant, which corresponding data source application programming interfaces (APIs) to contact, how to parse the user query for each search API, and ultimately how to rank the final data source results.

[0019] For different search queries, a search system can intelligently recommend what types of data sources may be most relevant to that particular search query. For example, if a user types in a search for "Quasi convolutional neural network", the search system can preliminarily determine (or classify via a classifier) that the search term is related to a technical topic. Thus, appropriate data sources may be knowledge bases such as "Wikipedia", discussion forums such as "Reddit" where users can discuss technical topics, archives of scientific manuscripts such as "arXiv", and / or the like, which may be most likely to be relevant or interesting to the user. The search system can then recommend these data sources to the user. Further details of the AI-based customized search platform can be found in U.S. Non-Provisional Application No. 17 / 981,102, filed on November 4, 2022, which is co-pending and by the same applicant, and which is hereby expressly incorporated by reference in its entirety.

[0020] Using a customized search system, a user may engage a dedicated database for a specific search. For example, users such as developers, programmers, etc., can enter a search query regarding a coding program such as "sort a list Python". The customized search system can then determine that the search query can be executed through coding-program-related data sources such as Stack Overflow, W3Schools, Tutorials Point, or other well-known data sources for learning how to program and asking questions related to computer programming, and can send the search query to relevant APIs corresponding to these data sources. In this way, less experienced users such as novice developers or even ordinary people can use the customized search system to enter natural language queries for code projects and search these dedicated code databases without a certain level of specialized knowledge for entering the most effective search strings. These dedicated code databases are often scattered and difficult for ordinary people to use.

[0021] On the one hand, existing search engines typically provide a list of URL links that potentially contain content related to a search query. Thus, in order to review or search through the search results, users often have to visit each website to determine whether the content from each website provides relevant information. For example, when a user is searching for coding-related content, the user often has to review the entire web page to see if there is any content from that web page and which parts of that content can be applied in a coding environment and make a judgment. The user may then need to manually copy, paste, and edit the content (e.g., code segments) from the web page into the coding environment. Thus, this search process not only requires a certain level of expertise from the user in order to understand the content on the web page, but also constantly working through different windows such as the search engine, the web page of the search results, and the IDE window can be time-consuming and inconvenient for the user.

[0022] Embodiments described herein provide a customized code search system that generates code search results from a customized data source, extracts code snippets from the code search results, and presents the code snippets via a user interface. In certain embodiments, the search system employs a machine learning module to generate and highlight search results from different data sources, such as code examples in a programming language. To improve search efficiency, in response to a code search query, the search system can extract code snippets from search results from relevant sources within user interface elements such as user-selectable panels. In this way, instead of having to follow each search result link to visit and review the content, the user can directly apply and incorporate the code snippets from the user interface panel into the IDE.

[0023] For example, when a user enters the query "sort a list in Python", the search engine determines that the search query is related to a coding program and can identify multiple coding program data sources to search, such as Stack Overflow, W3Schools, Tutorials Point, etc. Instead of returning some web links from these data sources that discuss the "sorting a list" algorithm, the search engine can parse the search result web links, such as applicable Python snippets, and return exemplary Python snippets in one or more viewable web widgets, such as a side panel within the search browser. The user can click on a panel of code snippets from a data source that provides the code snippet, such as Stack Overflow, and view a list of search results, such as discussion threads and code examples related to the search query, specifically provided by the data source "Stack Overflow". In another example, when the user clicks on a panel about "Tutorials Point", code snippets from Tutorials Point related to the search query can be provided.

[0024] The embodiments described herein further provide an in-IDE code search tool that is integrated into an IDE environment to automatically search for code snippets and assist in an ongoing coding project within the IDE window. In one embodiment, the search client component may be integrated with an IDE implemented on a user device, and the IDE monitors user activities related to code segments within the IDE. The monitored coding activities (e.g., a portion of a coding segment, cursor movement, user idle time, etc.) may then be provided to a search system, which may then determine a code search query based on the monitored user activities by a neural network-based prediction model. The search system may then perform a search on relevant coding data sources based on the code search query and receive search results. The search system may further extract code snippets from the web page of the content by following the search result links and return the code snippets to the client component. The client component may display the code snippets in the user interface within the IDE, whereby the user can choose to incorporate the code snippets into the current coding project within the IDE.

[0025] For example, pauses by the user may indicate that the user is thinking about how a certain coding pattern functions. The search system can identify potential search queries that may be useful to the user using the context above and / or below where the user is currently typing. The search system can then perform a search and display the search results to the user. These search results can be presented to the user within a window pane as part of the IDE without the user having to open a separate window. The code corresponding to the search results may be presented to the user, the user may select a preferred search result, and the search result can be directly inserted into the written code without the user having to copy and paste the code into the written code.

[0026] In this way, the automatic in-IDE code search system significantly improves search and coding efficiency as well as the user programming experience. Further, the in-IDE code search system uses various neural network-based modules, resulting in neural network technology that can perform customized code searches and return code snippets that are immediately applicable for the IDE. Thus, as further described with respect to the figures, neural network technology has been improved in search engines and computer-aided technologies.

[0027] FIG. 1 is a simplified diagram showing a code search framework 100 that facilitates the data flow between a search server that implements the code search process described in FIGS. 2-8D according to an embodiment described herein and related entities. The code search framework 100 includes a user 130 operating a user device 120, a search server 110, and one or more data sources 103a-n connected to the server 110 through one or more application programming interfaces (APIs) 112a-n.

[0028] User device 120 can interact with search server 110 by providing user activity 122 via a client component on which an IDE application running on user device 120 is installed. In one implementation, user activity 122 may include a user's manual input regarding a search provided by the user, for example, "sort a list Python".

[0029] In another implementation, user activity 122 may include monitored user coding activity within the IDE on user device 120. For example, a client component of a customized search system may be delivered from search server 110 and integrated into the IDE running on user device 120, thereby enabling the user to utilize the search system when coding within the IDE without the user initiating a search. For example, user activity 122 may include what the user is coding, how long the user has been typing, when the user has moved to a new line, when and for how long the user has paused, whether the current line contains functional code, user cursor movement up and down indicating attention to a particular code segment, the content of the lines before and after the current line selected by the user, and / or the like. This information indicates to the search system when a search may be useful to the user and can provide search results to the user without the user first initiating a search. The search system client component observes the coding activity for search server 110 and determines at decision points when to execute a search and provide search results to the user based on the input received from the IDE.

[0030] In another implementation, the user device 120 can further provide search context collected by, for example, code projects previously written by the user, code files currently open, other search terms entered by the user on a separate browser window, etc., and can provide context information useful in determining potential search needs for individual code being written. In some embodiments, this context information can be used together with information collected by the search system regarding user preferences, previous searches by the user, trends in search activity, and other context information to determine additional useful search elements to assist the user while writing code.

[0031] In one embodiment, the search server 110 can determine a search data source. For example, when a search query is input through a search browser window, the search server 110 can use a neural network-based AI model to predict relevant data sources for that search, such as coding-related data sources. Further details on determining a specific data source based on a search query can be found in U.S. Non-Provisional Application No. 17 / 981,102, filed on November 4, 2022, which is co-pending and by the same applicant.

[0032] In another example, when the search server 110 receives user coding activity 122 from a client component within an IDE, the search server 110 can determine a pre-defined data source as coding-related to the search. The determined data source may further receive prior user interaction. For example, the user does not approve search results from a certain data source, the user pre-configures a preferred data source, etc.

[0033] In one embodiment, when receiving user activity 122, the search server 110 can determine when to generate a search query and whether to generate it. For example, as further described in connection with FIGS. 2-5, the search server 110 can host one or more neural network-based prediction modules. The prediction module can generate a coded search query based on the user activity 122 and / or other context information when the prediction module determines that a search is to be performed at a time that depends on the received user activity 122. For example, the user activity 122 indicates that the user has an active IDE window but has been inactive for more than a time threshold, the user has scrolled up and down a set of lines more times than a certain number to review, an error has been detected at the current coding location, etc.

[0034] The search server 110 can then generate a customized search query according to each determined data source and send the customized search queries 111a - n to the respective data sources 103a - n through the respective APIs 112a - n. In response, the data sources 103a - n can return the query results 112a - n to the search server 110 in the form of links to web pages and / or cloud files.

[0035] In one embodiment, instead of presenting a link to a search result (e.g., a web page) to the user device 120, the search server 110 can extract code snippets from the search results and return actual code snippets 125 for display on the user device 120. For example, the client component on the user device 120 can display the code snippets in a panel within a search browser (such as shown in FIGS. 8A - 8D) and in a panel within an IDE (such as shown in FIGS. 7A - 7C).

[0036] In this way, the user 130 can submit the selection 126, for example, by selecting to directly incorporate a code snippet presented on the user interface panel into the ongoing coding project in the IDE.

[0037] Figure 2 is a schematic diagram showing a computing device implementing a search server within the code search framework described in FIG. 1, according to an embodiment described herein. As shown in FIG. 2, the computing device 200 includes a processor 210 coupled to a memory 220. The operation of the computing device 200 is controlled by the processor 210. Also, although the computing device 200 is shown as having only one processor 210, it is understood that the processor 210 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), etc. within the computing device 200. The computing device 200 may be implemented as a stand-alone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0038] Memory 220 may be used to store software executed by computing device 200 and / or one or more data structures used during the operation of computing device 200. Memory 220 may include one or more types of machine-readable media. Some common forms of machine-readable media include floppy (registered trademark) disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media to which a processor or computer is adapted to read.

[0039] Processor 210 and / or memory 220 may be arranged in any suitable physical arrangement. In some embodiments, processor 210 and / or memory 220 may be implemented on the same substrate, within the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, processor 210 and / or memory 220 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, processor 210 and / or memory 220 may be arranged in one or more data centers and / or cloud computing facilities.

[0040] In some examples, the memory 220 may include a non-transitory tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 210), causes the one or more processors to execute a method described in more detail herein. For example, as illustrated, the memory 220 may include instructions for a code search module 230 that can be used to implement and / or emulate systems and models and / or to implement any of the methods further described herein. The code search module 230 may receive an input 240, such as input data (e.g., code activity data) via the data interface 215, and may generate an output 250 that may be a predicted search query. Examples of input data may include the code activity data 122 of FIG. 1, and examples of output data may include search queries 111a-n. In another example, the input 240 may be a received search query, and the output 250 may be a code snippet 125 in response to the search query.

[0041] The data interface 215 may include a communication interface, a user interface (such as a voice input interface, a graphical user interface, etc.). For example, the computing device 200 may receive an input 240 (such as a training data set) from a networked database via the communication interface. Alternatively, the computing device 200 may receive an input 240, such as a search query input by the user, from the user via the user interface.

[0042] In some embodiments, the code search module 230 is configured to generate an output code snippet to a user device (e.g., 120 in FIG. 1). The code search module 230 may further include a prediction sub-module 231, an extraction sub-module 232, a search sub-module 233, and a ranking sub-module 234. The prediction sub-module 231 may generate a search query based on the received input coding activity (e.g., 122 in FIG. 1). For example, the prediction sub-module 231 can also determine when code search should be triggered, and the generation of the search query indicates the point in time when it is determined that code search is triggered. The search sub-module 233 can determine one or more data sources for code search based on, for example, a preferred user configuration, the user's past behavior indicating preferences, the search query, the coding data source type, etc. The search sub-module 233 may further generate a query customized according to each data source, send the customized query to the corresponding API (e.g., 112a - n in FIG. 1), and receive search results from those APIs. The extraction sub-module 232 may extract code snippets from each search result, which was originally in the form of a link to a web page and / or a cloud file. The ranking sub-module 234 may rank the code snippets based on, for example, completeness, the reputation of the data source, relevance, etc. Additional functions of the sub-modules 231 - 234 may be further described in connection with FIG. 5.

[0043] Some examples of computing devices, such as computing device 200, may include a non - transient tangible machine - readable medium that, when executed by one or more processors (e.g., processor 210), can cause the one or more processors to execute a process of a method, including executable code. Some common forms of machine - readable media that can include a process of a method are, for example, floppy (registered trademark) disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD - ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROM, EPROM, FLASH - EPROM, any other memory chip or cartridge, and / or any other media adapted to be read by a processor or computer.

[0044] FIG. 3 is a simplified diagram showing a neural network structure implementing the code search module 230 described in FIG. 2 according to an embodiment described herein. In an embodiment, one or more of the code search module 230 and / or its sub - modules 231 - 234 can be implemented via the artificial neural network structure shown in FIG. 2. A neural network comprises a computing system constructed on a set of connected units or nodes called neurons (e.g., 244, 245, 246). Neurons are often connected by edges, and the edges often have adjustable weights (e.g., 251, 252) associated with them. Neurons are often grouped into layers such that different layers can perform different transformations on their respective inputs and output the transformed input data onto the next layer.

[0045] For example, a neural network architecture may include an input layer 241, one or more hidden layers 242, and an output layer 243. Each layer may include a plurality of neurons, and the neurons between layers are interconnected according to a specific topology of the neural network topology. The input layer 241 receives input data such as user coding activities (e.g., 122 in FIG. 1) and search queries input by the user (e.g., 240 in FIG. 2A). The number of nodes (neurons) in the input layer 241 can be determined by the dimensionality of the input data (e.g., the length of the vector giving an example of the input). Each node in the input layer represents a feature or attribute of the input.

[0046] The hidden layer 242 is an intermediate layer between the input layer and the output layer of the neural network. Note that two hidden layers 242 are shown in FIG. XX for illustrative purposes only, and any number of hidden layers can be utilized in the neural network structure. The hidden layer 242 can extract and transform input data through a series of weighted calculations and activation functions.

[0047] For example, as described with reference to FIG. 2, the code search module 230 receives an input 240 of user coding activity and converts the input into an output 250 of search code snippets. To perform the conversion, each neuron receives an input signal, performs a weighted sum of the inputs according to the weights assigned to each connection (e.g., 251, 252), and then applies an activation function (e.g., 261, 262, etc.) associated with each neuron to the result. The output of the activation function is passed to the next layer of neurons or serves as the final output of the network. The activation functions may be the same or different across different layers. Exemplary activation functions include, but are not limited to, sigmoid, hyperbolic tangent, rectified linear unit (ReLU), leaky ReLU, Softmax, etc. In this way, after several hidden layers, the input data received at the input layer 241 is converted into quite different values that exhibit data characteristics corresponding to the task designed to be performed by the neural network structure.

[0048] The output layer 243 is the final layer of the neural network structure. It generates the output or prediction of the network based on the calculations performed in the preceding layers (e.g., 241, 242). The number of nodes in the output layer depends on the nature of the task being handled. For example, in a binary classification problem, the output layer can consist of a single node representing the probability of belonging to one class. In a multi-class classification problem, the output layer may have multiple nodes, each representing the probability of belonging to a specific class.

[0049] Accordingly, one or more of the code search module 230 and / or its sub-modules 231-234 may comprise a transformational neural network structure of layers of neurons and weights and activation functions that describe non-linear transformations at each neuron. Such a neural network structure is often implemented on one or more hardware processors 210 such as a graphics processing unit (GPU). Examples of neural networks can be [give examples of neural models] and the like.

[0050] In certain embodiments, the code search module 230 and its sub-module 231 may be implemented by hardware, software, and / or combinations thereof. For example, the code search module 230 and its sub-module 231 may be implemented and executed on various hardware platforms 250 such as a CPU (central processing unit), GPU (graphics processing unit), FPGA (field programmable gate array), ASIC (application specific integrated circuit), dedicated AI accelerator such as a TPU (tensor processing unit), and dedicated hardware accelerators specifically designed for neural network computations described herein, but not limited to these, and may comprise a specific neural network structure. Exemplary specific hardware for the neural network structure may include, but is not limited to, Google Edge TPU, Deep Learning Accelerator (DLA), NVIDIA AI-focused GPU, etc. The hardware 250 used to implement the neural network structure is specifically configured according to factors such as the complexity of the neural network, the scale of the task (e.g., training time, input data scale, size of the training dataset, etc.), and the desired performance.

[0051] In one embodiment, one or more of the neural network-based code search module 230 and its sub-modules 231-234 can be trained by sequentially and iteratively updating the parameters (e.g., weights 251, 252, etc., bias parameters and / or coefficients in activation functions 261, 262 associated with neurons) underlying the neural network based on a loss objective. For example, during forward propagation, training data such as past coding activities is supplied to the neural network. The data flows through the layers 241, 242 of the network, and each layer performs calculations based on its weights, biases, and activation functions until the output layer 243 generates an output 250 of the network such as a predicted code search query.

[0052] The output generated by the output layer 243 is compared with the expected output (e.g., a "correct answer" such as giving an example of a corresponding correct label), e.g., the actual code search query corresponding to the coding activity from the training data, to calculate a loss function that measures the discrepancy between the predicted output and the expected output. For example, the loss function can be cross-entropy, mean squared error (MSE), etc. Given the loss, the negative gradient of the loss function is calculated individually for each weight of each layer. Such negative gradients are calculated sequentially and iteratively in the reverse direction, one layer at a time, from the last layer 243 of the neural network to the input layer 241. These gradients quantify the sensitivity of the output of the network to changes in the parameters. The chain rule in differential calculus is applied to efficiently calculate these gradients by propagating the gradients in the reverse direction from the output layer 243 to the input layer 241.

[0053] The parameters of the neural network are updated in the reverse direction from the last layer to the input layer based on the calculated negative gradient using an optimization algorithm to minimize the loss (backpropagation). The backpropagation from the last layer 243 to the input layer 241 can be done for several training samples in several sequential iterative training epochs. In this way, the parameters of the neural network can be gradually updated in the direction that results in less or minimized loss, and the fact that the loss is less or minimized indicates that the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching the maximum number of epochs or achieving satisfactory performance for the validation data. At this point, the trained network can be used to make predictions for new, unseen data, such as in an auto-search-based IDE.

[0054] Therefore, the training process transforms the neural network into an "updated" trained neural network with updated parameters such as weights, activation functions, and biases. Thus, the trained neural network improves neural network technology in a cloud-based search system.

[0055] FIG. 4 is a simplified block diagram of a networked system 400 suitable for implementing the code search framework described in FIG. 1 and other embodiments described herein. In one embodiment, system 400 includes a user device 410 that can be operated by a user 440, data vendor servers 445, 470, and 480, a server 430, and other forms of devices, servers, and / or software components that operate to perform various methodologies according to the embodiments described. Exemplary devices and servers include devices, stand-alone, and enterprise-class servers similar to the computing device 200 described in FIG. 2 that operate an OS such as a MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS. The devices and / or servers shown in FIG. 4 may be deployed in other ways, and the operations and / or services performed or provided by such devices and / or servers may be combined or separated for a given embodiment, and may be performed by more or fewer devices and / or servers, as may be understood. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0056] The user device 410, data vendor servers 445, 470, and 480, and the server platform 430 (e.g., similar to the search server 110 of FIG. 1) may communicate with each other through a network 460. The user device 410 may be utilized by a user 440 (e.g., a driver, system administrator, etc.) to access various features available to the user device 410, which may include processes and / or applications associated with the server 430 for receiving output data anomaly reports.

[0057] The user device 410, the data vendor server 445, and the server 430 may each include one or more processors, memories, and other appropriate components for executing instructions such as program code and / or data stored on one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored on one or more computer-readable media such as memories or data storage devices internal and / or external to the various components of the system 400 and / or accessible through the network 460.

[0058] The user device 410 may be implemented as a communication device that can utilize appropriate hardware and software configured for wired and / or wireless communication with the data vendor server 445 and / or the server 430. For example, in one embodiment, the user device 410 may be an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch having appropriate computer hardware resources, glasses having appropriate computer hardware (e.g., GOOGLE GLASS (registered trademark)), other types of wearable computing devices, an embedded communication device, and / or other types of computing devices capable of transmitting and / or receiving data, such as an IPAD (registered trademark) from APPLE (registered trademark). Although only one communication device is shown, multiple communication devices may function similarly.

[0059] The user device 410 of FIG. 4 includes a user interface (UI) application 412 and / or other applications 416, which may correspond to executable processes, procedures, and / or applications having associated hardware. For example, the user device 410 may receive a message indicating a code snippet (e.g., 125 of FIG. 1) from the server 430 and display the message via the UI application 412 (see, e.g., FIGS. 7A-8D). In other embodiments, the user device 410 may optionally include additional or different modules having dedicated hardware and / or software.

[0060] In various embodiments, user device 410 includes other applications 416 that may be desired in certain embodiments to provide functionality to user device 410. For example, other applications 416 may include an IDE application for software development, which often includes a code editor application that can operate with UI application 412 and a compiler. As another example, other applications 416 may include a security application for implementing client-side security features, a programming client application for interfacing with an appropriate application programming interface (API) through network 460, or other types of applications. Other applications 416 can also include communication applications such as email, text messaging, voice, social networking, and IM applications that enable the user to send and receive email, phone calls, texts, and other notifications through network 460. For example, other applications 416 may be an email or instant messaging application that receives prediction result messages from server 430. Other applications 416 may include a device interface and other display modules that can receive input and / or output information. For example, other applications 416 may include a software program for asset management executable by a processor that includes a graphical user interface (GUI) configured to provide an interface for the user 440 to view code snippets.

[0061] The user device 410 may further include a database 418 stored in the temporary and / or non-temporary memory of the user device 410, which stores various applications and data and may be utilized during the execution of various modules of the user device 410. The database 418 may store user profiles regarding the user 440, predictions previously viewed or saved by the user 440, historical data received from the server 430, etc. In some embodiments, the database 418 may be local to the user device 410. However, in other embodiments, the database 418 may be external to the user device 410 and accessible by the user device 410, including a cloud storage system and / or database accessible through the network 460.

[0062] The user device 410 includes at least one network interface component 417 adapted to communicate with a data vendor server 445 and / or the server 430. In various embodiments, the network interface component 417 may include various other types of wired and / or wireless network communication devices including a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet® device, a broadband device, a satellite device, and / or microwave, radio frequency, infrared, Bluetooth®, and near field communication devices.

[0063] The data vendor server 445 may correspond to a server that hosts a database 419 to provide a training data set including user coding activities, corresponding search queries, and code snippets to the server 430. The database 419 may be implemented by one or more relational databases, distributed databases, cloud databases, etc.

[0064] The data vendor server 445 includes at least one network interface component 426 adapted to communicate with the user device 410 and / or the server 430. In various embodiments, the network interface component 426 can include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet® device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices including microwave, radio frequency, infrared, Bluetooth®, and near field communication devices. For example, in one implementation, the data vendor server 445 may transmit asset information from the database 419 to the server 430 via the network interface 426.

[0065] The server 430 may be housed together with the code search module 230 and its sub-modules described in FIG. 2. In some implementations, the code search module 230 may receive data from the database 419 in the data vendor server 445 via the network 460 and generate code snippets. The generated code snippets may be transmitted to the user device 410 for review by the user 440 via the network 460.

[0066] The database 432 may be stored in the temporary and / or non-temporary memory of the server 430. In one implementation, the database 432 may store data obtained from the data vendor server 445. In one implementation, the database 432 can store the parameters of the code search module 230. In one implementation, the database 432 can store previously searched code snippets, code projects previously written by the user, the user's previous coding activities, and the like.

[0067] In some embodiments, database 432 may be local to server 430. However, in other embodiments, database 432 may be external to server 430 and accessible by server 430, including a cloud storage system and / or database accessible through network 460.

[0068] Server 430 includes at least one network interface component 433 adapted to communicate with user device 410 and / or data vendor servers 445, 470, or 480 through network 460. In various embodiments, network interface component 433 may comprise various other types of wired and / or wireless network communication devices including DSL (e.g., Digital Subscriber Line) modems, PSTN (Public Switched Telephone Network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency (RF), and infrared (IR) communication devices.

[0069] Network 460 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 460 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, network 460 may accommodate small-scale communication networks such as private or local area networks accessible by various components of system 400, or larger-scale networks such as wide area networks or the Internet.

[0070] FIG. 5 is a schematic diagram showing an exemplary architecture of a code search platform implemented in a search server according to the embodiments described in this specification. The code search module 230 may comprise a software and hardware platform implemented in the search server 110 of FIG. 1 and / or the server 430 of FIG. 3. For example, the code search module 230 may be implemented based on the neural network structure shown in FIG. 3.

[0071] In one embodiment, the prediction module 231 may receive any of the user query 502, the user coding activity 504, and / or other context 505. For example, when the user query 502, for example, a user manual input regarding a search provided by the user, for example, "sort a list Python", is received, the prediction module 231 may determine to immediately pass the user query 502 to the search module 233 to initiate a code search.

[0072] As another example, if the user query 502 is not received, the prediction module 231 can predict a search query based on the user coding activity 504 and other context 505. The user coding activity 504 can include what the user is coding, the time the user was typing, when the user moved to a new line, when and for how long the user paused, whether the current line contains functional code, user cursor movement that scrolls up and down indicating attention to a particular code segment, the content of the lines before and after the current line selected by the user, and so on. The other context 505 can include the code projects the user has written previously, the code files that are open simultaneously, other search terms the user has entered on a separate browser window, etc., which can provide context information useful in determining the potential search needs for individual written code. In some embodiments, this context information 505 can further include user preferences, previous searches by the user, trends in search activity, and other context information for determining additional useful search elements to assist the user while coding.

[0073] In one embodiment, the prediction module 231 can concatenate input information such as user coding activity 504, user query 502 (optional), and other context 505 into an input sequence of tokens and generate a predicted code search query. The prediction may be executed periodically, intermittently, and / or continuously depending on the user coding activity 504 that is constantly updated. In one implementation, when there is no direct user query 502 for code search, the prediction module 231 can also make a prediction as to whether the input sequence of user coding activity 504 and other context 505 triggers a code search. Making the prediction is, for example, when the user activity 122 indicates that the user has an active IDE window but has been idle for more than a time threshold, the user has scrolled up and down a set of lines more times than a certain number to review, an error has been detected at the current coding location, etc.

[0074] The prediction module 231 can be trained with a dataset of previous coding activity 504, previous context 505, and (optional) previous user query 502, and the corresponding correct code search query associated with the coding activity.

[0075] The search module 232 may receive a code search query from the prediction module 231 and then determine a list of data sources for the search. In one implementation, the search module 233 can retrieve a pre-defined list of data sources pre-classified as relevant coding libraries, such as StackOverflow, Tutorials Point, etc. In another implementation, the search module 233 can use the prediction module to predict a prioritized list of data sources for the search based on the concatenation of the code search query, coding activity 504, and / or other context information 505, in a similar manner as described in U.S. Non-Provisional Application No. 17 / 981,102, filed on November 4, 2022, which is co-pending and by the same applicant.

[0076] The search module 233 can then send a customized coding search query for each identified data source to the respective search APIs 522a - n and receive a list of search results from each of the search APIs 522a - n.

[0077] In some embodiments, the rank module 234 may optionally rank a list of search apps 522a - n to perform a search. Each search application 522a - n corresponds to a specific data source 103a - n in FIG. 1. For example, search app 522a corresponds to a search application configured to search within the "StackOverflow" database, search app 522b corresponds to a search application configured to search within the "Tutorials Point" database, and so on. The rank module 234 scores a plurality of search apps 522a - n by using an input sequence including the user query 502, coding activity 504, and other context 505, and running the input sequence through a neural network model once for each search app 522a - n. In this way, the rank module 234 can rank the search results from the list of data sources via the search APIs 522a - n.

[0078] For example, if the user always adopts code search results from "StackOverflow" and this information is reflected in other context 505, the rank module 234 can rank the search results from the "StackOverflow" API higher.

[0079] The search results from the search APIs 522a - n are often in the form of links to web pages or cloud files within their respective data sources. The ranked list of search results may be passed from the rank module 234 to the extraction module 232.

[0080] The extraction module 232 can follow the links of the search results and extract code snippets from the content on the web page or cloud file. The code snippets 531a - n may then be distributed according to a ranked list based on their respective data sources. In one implementation, the extraction module 232 may further verify whether the code snippets extracted from the web page are complete, and adjust the ranking by prioritizing high - quality code snippets.

[0081] For example, the code snippets 531a - n are then sent to the user device for display via a graphical user interface or some other type of user output device. For example, the code snippets 531a - n may be grouped and presented in the form of a list of user - actionable elements, each displaying an icon representing its respective search app (data source) within, for example, an IDE window or a browser window.

[0082] FIG. 6 is an exemplary logic flow diagram showing a method of performing real - time code search in an IDE based on the code search framework and architecture shown in FIGS. 1 - 5 according to some embodiments described herein. One or more of the processes of method 600 may be implemented, at least in part, in the form of executable code stored on a non - transitory tangible machine - readable medium that can cause one or more processors to execute one or more of the processes. In some embodiments, method 600 corresponds to the operation of the code search module 230 (e.g., FIGS. 2 - 5) that performs code search in response to monitored user coding activities and automatically returns code snippets.

[0083] In step 602, the client component may be provided with and installed on an integrated development environment (IDE) implemented on a user device (e.g., 120 in FIG. 1), and may monitor user activities (e.g., 122 in FIG. 1) related to the code segment.

[0084] In step 604, a neural network-based prediction model (e.g., 230 in FIGS. 2-5 and / or its sub-module 231) implemented on one or more hardware processors in a search server (e.g., 110 in FIG. 1) may generate a code search query, at least in part, based on a first part of the code segment and the monitored user activities received from the user device. For example, the code search query may be further determined based on user activity events indicating the user's intent for code search, triggered, for example, by a long pause or the user scrolling the code segment up and down. In another example, the code search query is determined further based on identifying an error within the first part of the code segment.

[0085] In step 606, the search server may then send, via the network (e.g., 460 in FIG. 4), one or more search inputs customized from the code search query, via one or more search application programming interfaces (APIs) (e.g., 112a-n in FIG. 1, or 522a-n in FIG. 5), to the corresponding data sources (e.g., 103a-n in FIG. 1) pre-classified for code search.

[0086] In step 608, the search server may receive, from one or more search APIs, search results including links to web pages containing code snippets in response to the code search query.

[0087] In step 610, the search server can extract code snippets from web pages by following links. In some implementations, a rank model (e.g., 234 in FIGS. 2 and 5) ranks the search results, and thus can cause the display of code snippets in accordance with the ranking in the user interface within the IDE.

[0088] In step 612, the search server sends the code snippets to the user device, which can thereby cause the display of user-engagable widgets that display the code snippets in the user interface within the IDE. For example, each user-engagable widget can take the form of a panel with visual elements indicating data sources, such as "Stack Overflow". In step 612, the client component receives a user selection of a first code snippet, which thereby automatically integrates the first code snippet into a code segment upon user selection. For example, the automatic integration of the first code snippet includes replacing a lower portion of the code segment identified as related to a code error with the first code snippet.

[0089] In some implementations, steps 604-610 may be performed at a search server (e.g., 110 of FIG. 1). In another implementation, any of steps 604-610 may be performed by a client component of a user device. For example, a client component installed within an IDE of a user device may determine when to trigger step 604 from step 602. When the client component detects that the user has paused at a line of code for a period of time (e.g., 5 minutes, 10 minutes, etc.), scrolled up and down a segment of code, entered a question in a comment section, etc., the client component may send such coding activity and the associated code segment to the search server. The search server may then generate a code search query using an NLP model.

[0090] Figures 7A - 7C provide exemplary UI diagrams showing automatic code search within an IDE according to the embodiments described herein. In some embodiments, as shown in FIG. 7A, the search system can identify errors in the code written by the user based on the user's coding activities. The detected errors can trigger the prediction module 231 of FIGS. 2 and 5 to generate a code search query based on the user coding activities and other contexts. Provide inline suggestions to the user. Thus, the search system can identify errors in the code written by the user without prompting, resolve the errors, and provide the user with inline suggestions for code that may be syntactically correct. In such embodiments, the search system can utilize the context information 505 of FIG. 5 to identify one or more code snippets from the search results. In some embodiments, the search results may instead be shown as code snippets included inline within the text, and the user can indicate whether to accept the search results, view different search results, or ignore the search results. This code may replace one or more lines of the written code or be inserted into one or more new lines of the code, as determined by the search system or the user.

[0091] To detect errors in the code, for example, the search system can use natural language processing to parse the comments inserted into the code written by the user, identify regions in the code that match and do not match the user's comments, and provide suggestions for code that can achieve the functionality desired by the user as written in the user's comments. Thus, the search system can identify when the written code does not match the user's intent and propose corrections to the code. In some embodiments, the search system may utilize a neural network to identify errors or discrepancies with the user's intent and provide suggestions or corrections to the code.

[0092] In some embodiments, as shown in FIG. 7B, the search system uses natural language processing to parse comments (e.g., "#python open file" [Open Python file]) inserted into the written code by the user, identify potential search queries that may be useful to the user, perform a search, and present the search results in a side panel. As shown in FIG. 7C, the user can select "Try solution 0" [Try solution 0] to automatically incorporate the code snippet into the code.

[0093] FIGS. 8A - 8D provide exemplary UI diagrams showing code search within a search browser window according to embodiments described herein. In one example, the user may be writing a "switch statement" in Java®. In traditional search, a list of results is presented to the user, and the user has to visit each web link to find code examples. As shown in FIG. 8A, for a search of "switch statement java", the search results may be a list of code snippets presented on a panel rather than direct links to websites, and the search system extracts relevant code snippets and displays the results on a panel on the search screen. For example, these code snippets may appear as a collection of panels from tutorial - based websites. The user can then scroll the snippet panel to view different results or choose to view additional information related to the snippets within a particular panel.

[0094] As shown in FIG. 8B, in another example, if the user decides they want to see all snippets from a selected source, the search system can display the snippets in a side panel. These panels can display information that the search system determines is relevant to the search query, such as code snippets and corresponding text that explains those code snippets. A variety of results from a single source can be collected by the search system, multiple panels can be generated, and then the user can scroll through them to view various techniques or information relevant to the search query.

[0095] As shown in FIG. 8C, the user may select the “try for yourself” button. Then, the code snippets within each panel can be copied to the clipboard and the user can paste it into the IDE.

[0096] As shown in FIG. 8D, in another example, for a data source that is a discussion forum such as Stack Overflow, such a forum may contain one or more answers that respond to questions similar to the search query. The panel may include the question and related code snippets, as well as all code snippets from the answers and corresponding explanations and comments. The user can scroll through the various forum questions and responses relevant to the search query in the same way as other code snippets.

[0097] Where applicable, the various embodiments provided by the present disclosure can be implemented using hardware, software, or a combination of hardware and software. Also, where applicable, the various hardware components and / or software components described herein can be combined into composite components comprising software, hardware, and / or both, without departing from the spirit of the present disclosure. Where applicable, the various hardware components and / or software components described herein can be separated into sub-components comprising software, hardware, or both, without departing from the scope of the present disclosure. Further, where applicable, it is contemplated that software components can be implemented as hardware components and vice versa.

[0098] Software according to the present disclosure, such as program code and / or data, can be stored on one or more computer-readable media. It is also contemplated that the software identified herein can be implemented using networked and / or other one or more general-purpose or special-purpose computers and / or computer systems. Where applicable, the order of the various steps described herein can be changed, combined into composite steps, and / or separated into sub-steps to provide the features described herein.

[0099] The various features and steps described in this specification may be implemented as a system comprising one or more memories that store the various information described in this specification, and one or more processors coupled to the one or more memories and a network, the one or more processors, when executed by the one or more processors, being operative to perform a method including the steps described in this specification, and as a non-transitory machine-readable medium comprising a plurality of machine-readable instructions adapted to cause the one or more processors to execute a method performed by one or more devices such as a hardware processor, a user device, a server, and other devices described in this specification, to perform the steps described in this specification.

Claims

**Claim 1** A method implemented on a processor for performing real-time code search in an integrated development environment (IDE), the method comprising: monitoring user activities related to code segments via a client component having an IDE implemented on a user device; determining a code search query, at least in part, based on a first portion of the code segment and the monitored user activities, by a neural network-based prediction model implemented on one or more hardware processors; sending, via a first search application programming interface (API) over a network, a customized first search input from the code search query to a first data source pre-classified for code search; receiving, from the first search API, a first search result including a first link to a first web page including a first code snippet, in response to the code search query; extracting the first code snippet from the first web page by following the first link; causing a display of a first user-interactable widget that displays the first code snippet in a user interface within the IDE. A method. **Claim 2** The method of claim 1, wherein the code search query is further determined based on a user activity event indicating a user's intent for the code search. **Claim 3** The method of claim 1, wherein the code search query is further determined based on identifying an error within a first portion of the code segment. **Claim 4** Sending, via a second search application programming interface (API) over the network, a customized second search input from the code search query to a second data source pre-classified for code search; Receiving, from the second search API, a second search result including a second link to a second web page including a second code snippet, in response to the code search query; Extracting the second code snippet from the second web page by following the second link; Ranking the first search result and the second search result by a ranking model; In a user interface within the IDE, causing display of a first user-engagable widget that displays the first code snippet and a second user-engagable widget that displays the second code snippet according to the ranking; The method according to claim 1.

5. The method according to claim 1, wherein the first user-engagable widget includes a visual element indicating the first data source.

6. The monitored user activity is sent to a search server, the neural network-based prediction model is implemented at the search server, and the method: Further including sending the first code snippet to the user device, thereby causing display of a first user-engagable widget. The method according to claim 1.

7. The method according to claim 1, wherein the neural network-based prediction model is implemented by the client component on the user device.

8. The method according to claim 1, wherein the client component receives a user selection of the first code snippet, thereby causing automatic integration of the first code snippet into the code segment upon the user selection.

9. The method according to claim 8, wherein the automatic integration of the first code snippet includes replacing a lower portion of the code segment identified as related to a code error with the first code snippet.

10. A system for performing real-time code search in an integrated development environment (IDE), the system comprising: A communication interface that receives user activity regarding a code segment via a client component having an IDE implemented on a user device; A memory storing a neural network-based model and a plurality of processor-executable instructions; One or more processors that execute the instructions to perform operations, the operations including: Determining a code search query, at least in part, based on a first part of the code segment and monitored user activity by a neural network-based prediction model implemented on one or more hardware processors; Sending, via a network through a first search application programming interface (API), a first search input customized from the code search query to a first data source pre-classified for code search; Receiving, from the first search API, a first search result including a first link to a first web page containing a first code snippet in response to the code search query; Extracting the first code snippet from the first web page by following the first link; Causing the display of a first user-interactable widget that displays the first code snippet in a user interface within the IDE, System. **Claim 11** The system of claim 10, wherein the code search query is further determined based on a user activity event indicative of a user's intent for the code search. **Claim 12** The system of claim 10, wherein the code search query is further determined based on identifying an error within the first part of the code segment. **Claim 13** The operations are: Sending, via a network through a second search application programming interface (API), a second search input customized from the code search query to a second data source pre-classified for code search; Receiving, from the second search API, a second search result including a second link to a second web page containing a second code snippet in response to the code search query; Extracting the second code snippet from the second web page by following the second link; Ranking the first search result and the second search result by a ranker model; In the user interface within the IDE, further including causing display of the first user-interactable widget that displays the first code snippet and a second user-interactable widget that displays the second code snippet, according to the ranking. The system according to claim 10.

14. The system according to claim 10, wherein the first user-interactable widget includes a visual element indicating the first data source.

15. The system according to claim 10, wherein the client component receives a user selection of the first code snippet, thereby causing automatic integration of the first code snippet into the code segment upon the user selection.

16. The system according to claim 15, wherein the automatic integration of the first code snippet includes replacing a lower portion of the code segment identified as related to a code error with the first code snippet.

17. A processor-readable non-transitory storage medium storing instructions executable by a plurality of processors for performing real-time code search in an integrated development environment (IDE), the instructions being executable by one or more processors to perform operations, the operations including: monitoring user activities related to a code segment via a client component having an IDE implemented on a user device; determining a code search query, at least in part, based on a first portion of the code segment and the monitored user activities by a neural network-based prediction model implemented on one or more hardware processors; sending, through a network via a first search application programming interface (API), a customized first search input from the code search query to a first data source pre-classified for code search; receiving, from the first search API, a first search result including a first link to a first web page including a first code snippet in response to the code search query; extracting the first code snippet from the first web page by following the first link; causing a display of a first user-interactable widget that displays the first code snippet in a user interface within the IDE; a processor-readable non-transitory storage medium. The processor-readable non-transitory storage medium of claim 17, wherein the code search query is further determined based on a user activity event indicative of a user's intent for the code search. The processor-readable non-transitory storage medium of claim 17, wherein the code search query is further determined based on identifying an error within a first portion of the code segment. The operations are: sending, via a second search application programming interface (API) and across the network, a customized second search input from the code search query to a second data source pre-classified for code search; receiving, from the second search API, a second search result including a second link to a second web page including a second code snippet, in response to the code search query; extracting the second code snippet from the second web page by following the second link; ranking the first search result and the second search result by a ranker model; further causing, in a user interface within the IDE, a display of the first user-interactable widget that displays the first code snippet and a second user-interactable widget that displays the second code snippet, according to the ranking; The processor-readable non-transitory storage medium of claim 1. ​ ​ ​

Images