System and method for search tool for code snippets

Abstract

The present application relates generally to search engines, and more particularly to systems and methods for search tools for code snippets. Embodiments described herein provide a customized code search system that generates code search results from a customized data source, extracts code snippets from the code search results, and presents the code snippets via a user interface. In one embodiment, a search system employs a machine learning module to generate and highlight search results from different data sources including code examples, e.g., in a programming language. To improve search efficiency, in response to a code search query, a search system may extract code snippets from search results from related sources in a user interface element, such as a user-selectable panel.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. non-provisional patent application No. 18 / 330,225 filed on June 6, 2023, which claims priority to co-pending and co-owned U.S. Provisional Application No. 63 / 349,855 filed on June 7, 2022 and co-pending and co-owned U.S. Provisional Application No. 63 / 446,199 filed on February 16, 2023.

[0003] This application is related to co-pending and co-owned U.S. non-provisional application No. 17 / 981,102 filed on November 4, 2022.

[0004] All of the above-mentioned applications are hereby expressly incorporated herein by reference in their entirety. Technical Field

[0005] The present application relates generally to search engines, and more particularly to a system and method for a search tool for code snippets that supports code searching from within an Integrated Development Environment (IDE). Background Art

[0006] Search engines allow users to provide search queries and return search results in response. Search sites such as Google.com, Bing.com, and / or the like typically provide users with a list of search results from a variety of data sources. For example, these existing search engines typically crawl web data to collect search results related to the search query. However, users have to visit various websites to determine whether the results provide relevant information. This can be frustrating when searching for specific types of content (such as when a programmer user searches for computer code when compiling programming code in an IDE, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a simplified diagram illustrating a code search framework according to one embodiment described herein that facilitates Figures 2 to 8D The data flow between the search server described in and the related entities that implement the code search processing.

[0008] Figure 2 is an example of implementing an embodiment according to the present invention. Figure 1 A simplified diagram of a computing device of a search server in the code search framework described in .

[0009] Figure 3 is an example of implementing an embodiment according to the present invention. Figure 2A simplified diagram of the neural network structure of the code search module 230 described in FIG.

[0010] Figure 4 is suitable for implementing Figure 1 A simplified block diagram of a networked system for the code search framework described in and other embodiments described herein.

[0011] Figure 5 is a simplified diagram illustrating an example architecture of a code search platform implemented at a search server according to embodiments described herein.

[0012] Figure 6 is an example of a method based on some embodiments described herein. Figures 1 to 5 An example logical flow chart of a method for performing real-time code search in an IDE using the code search framework and architecture shown in FIG.

[0013] 7A to 7C An example UI diagram illustrating automatic code search within an IDE according to embodiments described herein is provided.

[0014] FIG. 8A to FIG. 8D An example UI diagram illustrating code searching within a search browser window is provided according to embodiments described herein.

[0015] Embodiments of the present disclosure and their advantages are best understood by reference to the following detailed description. It should be understood that similar reference numerals are used to identify the same elements illustrated in one or more of these figures, wherein the display in these figures is for the purpose of illustrating embodiments of the present disclosure, rather than for the purpose of limiting embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] The present application relates generally to search engines, and more particularly to systems and methods for search tools for code snippets.

[0017] As used herein, the term "network" may include any hardware or software based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented on or with the framework.

[0018] As used herein, the term "module" may include a hardware or software based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0019] Search engines allow users to provide search queries and return search results in response. Search sites such as Google.com, Bing.com, and / or the like typically provide users with a list of search results from a variety of data sources. Some customized search systems provide a web-based platform that provides a customized search experience for individual users from different data sources. In one embodiment, the search system uses a machine learning module to generate and filter search results from all different data sources. For example, the search platform can ingest user queries, user context information, and other context information to coordinate: which data sources are relevant, which (which) corresponding data source application programming interfaces (APIs) should be contacted, how to parse user queries for each search APP API, and finally the final ranking order of the data source results.

[0020] For different search queries, the search system can intelligently recommend what kind of data source may be most relevant to a specific search query. For example, when a user types a search for "Quasi convolutional neural network (quasi convolutional neural network)", the search system can preliminarily determine (or classify via a classifier) ​​that the search term is related to a technical topic. Therefore, suitable data sources can be knowledge bases such as "Wikipedia", discussion forums where users such as "Reddit" can discuss technical topics, archives of scientific manuscripts such as "arXiv", and / or analogs that are most likely to be relevant to or interest the user. Then, the search system can recommend these data sources to the user. Additional details of a custom search platform based on AI can be found in the co-pending and co-owned U.S. non-provisional application No. 17 / 981,102 filed on November 4, 2022, all of which are expressly incorporated herein by reference.

[0021] Using the custom search system, users can access (engage with) a dedicated database to perform a specific search. For example, users such as developers, programmers and / or the like can enter a search query related to a coding program, such as "sort a list Python" and / or the like. The custom search system can then determine that the search query can be carried out via a data source related to the coding program (such as Stack Overflow, W3Schools, Tutorials Point, or other well-known data sources for learning how to program and raising questions related to computer programming, etc.), and the search query is sent to the relevant API corresponding to these data sources. In this way, users with less experience (such as novice developers or even laymen, etc.) can also use the custom search system to enter a natural language query for a code project to search through these dedicated code databases (which are often scattered and difficult for laymen to use), even if there is no certain level of expertise to enter the most effective search string.

[0022] On the other hand, existing search engines usually provide a list of URL links that potentially contain content relevant to the search query. Like this, in order to check or retrieve search results, users often have to visit each website to determine whether the content from the corresponding website provides relevant information. For example, when a user is searching for coding related content, the user may often need to check the entire web page, find and determine whether there is content from the web page and what part of the content can be applied in the coding environment. Then, the user may need to manually copy, paste and edit the content (for example, code segment) from the web page in the coding environment. Thereby, not only does this search process require the user to have a certain level of expertise to be able to understand the content on the web page, but this search process is also time-consuming and inconvenient for the user to constantly work through different windows such as search engines, web pages of search results and IDE windows.

[0023] Embodiments described herein provide a custom code search system that generates code search results from a custom data source, extracts code snippets from the code search results, and presents the code snippets via a user interface. In one embodiment, the search system employs a machine learning module to generate and highlight search results based on different data sources including, for example, code examples in programming languages. In order to improve search efficiency, in response to a code search query, the search system may extract code snippets from search results from related sources in user interface elements such as user-selectable panels. In this way, instead of having to access and check the content that follows each search result link, the user may apply and incorporate the code snippets from the user interface panel directly into the IDE.

[0024] For example, if a user enters the query "sort a list in Python", the search engine may determine that the search query is related to coding programs and determine multiple coding program data sources to search, such as Stack Overflow, W3Schools, Tutorials Point, etc. Instead of returning multiple web links discussing algorithms for "sorting a list" from these data sources, the search engine may parse the search result web links (e.g., applicable Python snippets) and return example Python snippets in one or more viewable web widgets (e.g., a side panel within a search browser). The user may click on a panel of code snippets from a data source that provides code snippets (e.g., Stack Overflow) and view a list of search results specifically provided from the data source "Stack Overflow" (such as discussion threads and code examples related to the search query). For another example, if the user clicks on a panel for "Tutorials Point", code snippets from Tutorials Point related to the search query may be provided.

[0025] Embodiments described herein further provide an IDE code search tool integrated into the IDE environment to automatically search for code snippets and assist the ongoing coding project in the IDE window. In one embodiment, the search client component can be integrated with the IDE implemented on the user device, and the search client component then monitors the user activity related to the code snippet in the IDE. Then, the coding activity monitored (for example, a part of the coding segment, cursor movement, user pause time and / or the like) can be provided to the search system, and the search system can then utilize the prediction model based on the neural network to determine the code search query based on the user activity monitored. Then, the search system can search and receive search results on the relevant coding data source based on the code search query. The search system can further extract the code snippet from the web page of the content following the search result link, and the code snippet is returned to the client component. The client component then displays the code snippet at the user interface in the IDE, which allows the user to choose to merge the code snippet into the current coding project in the IDE.

[0026] For example, a user's pause may indicate that they are thinking about how a certain coding aspect works. The search system can use the context above and / or below where the user is currently typing to discern potential search queries that may be useful to the user. The search system can then conduct the search and display the search results to the user. These search results can be displayed to the user in a window pane that is part of the IDE without requiring the user to open a separate window. The code corresponding to the search results can be displayed to the user, and the user can select a preferred search result that can be inserted directly into the written code without requiring the user to copy and paste the code into the written code.

[0027] Thus, the automatic in-IDE code search system greatly improves the search and coding efficiency and the user programming experience. In addition, the in-IDE code search system employs various neural network-based modules, which results in a neural network technology that can perform customized code searches and return easily applicable code snippets for the IDE. Thus, as further discussed with respect to the accompanying drawings, neural network technology has been improved in search engines and computer-aided technology.

[0028] Figure 1 is a simplified diagram illustrating a code search framework 100 according to one embodiment described herein, which facilitates Figures 2 to 8D The code search framework 100 includes a user 130 operating a user device 120, a search server 110, and one or more data sources 103a to 103n connected to the server 110 via one or more application programming interfaces (APIs) 112a to 112n.

[0029] User device 120 may interact with search server 110 by providing user activity 122 via a client component installed with an IDE application running on user device 120. In one implementation, user activity 122 may include user manual input regarding a user-provided search, such as "sort a list Python."

[0030] In another implementation, the user activity 122 may include monitored user coding activities within an IDE on the user device 120. For example, a client component of a customized search system may be delivered from the search server 110 to be integrated into an IDE running at the user device 120, which allows the user to utilize the search system while coding within the IDE without the user initiating a search. For example, the user activity 122 may include what the user is coding, how long the user has been typing, when the user moves to a new line, when the user pauses and how long the pause lasts, whether the current line contains functional code, user cursor movement that scrolls up and down indicating user attention to a particular code segment, the content of the lines before and after the current line selected by the user, and / or the like. This information may indicate to the search system when a search will be useful to the user, and provide search results to the user without the user first initiating a search. The search system client component will observe the coding activities for the search server 110 to determine when to search and provide search results to the user at a decision point based on the input received from the IDE.

[0031] In another implementation, the user device 120 may further provide search contexts collected through code projects previously written by the user, code files opened at the same time, other search terms entered by the user on a separate browser window, and / or the like, which may provide context information that helps determine potential search needs for the individual writing code. In some embodiments, this context information may be used together with information about user preferences, previous searches by the user, trends in search activity, and other context information collected by the search system to determine additional useful search elements that will assist the user when writing code.

[0032] In one embodiment, the search server 110 can determine the search data source. For example, when a search query is entered through a search browser window, the search server 110 can engage a neural network-based AI model to predict relevant data sources for the search, such as encoding-related data sources, etc. Additional details of determining a specific data source based on a search query can be found in co-pending and commonly owned U.S. non-provisional application No. 17 / 981,102 filed on November 4, 2022.

[0033] For another example, when the search server 110 receives the user coding activity 122 from the client component in the IDE, the search server 110 can determine the data sources that have been pre-defined as coding-related for the search. The determined data sources can be further limited by previous user interactions, for example, the user does not approve the search results from a particular data source, the user pre-configures the preferred data source, and / or the like.

[0034] In one embodiment, the search server 110 can determine when and whether to generate a search query when receiving the user activity 122. For example, as described in Figures 2 to 5 As further described, the search server 110 may host one or more neural network-based prediction modules. When the prediction module determines that a search should be performed at a certain point in time based on the received user activity 122, the prediction module may generate a coding search query based on the user activity 122 and / or other contextual information. For example, when the user activity 122 indicates that the user has an active IDE window but has been paused for more than a time threshold, the user has scrolled up and down to check a set of lines more than a certain number of times, an error has been detected at the current coding position, and / or the like.

[0035] Then, the search server 110 can generate customized search queries based on the determined data sources, and send the customized search queries 111a to 111n to the respective data sources 103a to 103n through the respective APIs 112a to 112n. In response, the data sources 103a to 103n can return the query results 112a to 112n to the search server 110 in the form of links to web pages and / or cloud files.

[0036] In one embodiment, instead of presenting a link to the search results (e.g., a web page) to the user device 120, the search server 110 may extract the code snippet from the search results and return the actual code snippet 125 for display at the user device 120. For example, a client component at the user device 120 may display the code snippet 125 within a search browser (e.g., as shown in FIG. FIG. 8A to FIG. 8D ) and within the IDE (for example, 7A to 7C The code snippet is displayed in the panel shown in the figure.

[0037] In this way, the user 130 may submit the selection 126 , for example, by choosing to merge the code snippet presented on the user interface panel directly into an ongoing coding project at the IDE.

[0038] Figure 2 is an example of implementing an embodiment according to the present invention. Figure 1 A simplified diagram of a computing device of a search server in the code search framework described in FIG. Figure 2As shown, computing device 200 includes a processor 210 coupled to memory 220. The operation of computing device 200 is controlled by processor 210. And although computing device 200 is shown with only one processor 210, it should be understood that processor 210 can represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), and / or the like in computing device 200. Computing device 200 can be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0039] Memory 220 may be used to store software executed by computing device 200 and / or one or more data structures used during operation of computing device 200. Memory 220 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media from which a processor or computer is suitable for reading.

[0040] The processor 210 and / or the memory 220 may be arranged in any suitable physical arrangement. In some embodiments, the processor 210 and / or the memory 220 may be implemented on the same board, in the same package (e.g., a system-in-package), on the same chip (e.g., a system-on-chip), and / or the like. In some embodiments, the processor 210 and / or the memory 220 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 210 and / or the memory 220 may be located in one or more data centers and / or cloud computing facilities.

[0041] In some examples, memory 220 may include a non-transitory tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 210), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, memory 220 includes instructions for code search module 230 that may be used to implement and / or simulate systems and models and / or implement any of the methods described further herein. Code search module 230 may receive input 240 such as input data (e.g., code activity data) via data interface 215 and generate output 250 that may be a predictive search query. Examples of input data may include Figure 1The code activity data 122 in the input 240 and examples of the output data may include search queries 111a to 111n. In another example, the input 240 may be a received search query and the output 250 may be a code snippet 125 in response to the search query.

[0042] The data interface 215 may include a communication interface, a user interface (such as a voice input interface, a graphical user interface, and / or the like). For example, the computing device 200 may receive input 240 (such as a training data set) from a networked database via a communication interface. Or the computing device 200 may receive input 240 (such as a search query entered by the user) from a user via a user interface.

[0043] In some embodiments, the code search module 230 is configured to generate a code for a user device (e.g., Figure 1 The code search module 230 may further include a prediction submodule 231, an extraction submodule 232, a search submodule 233, and a ranking submodule 234. The prediction submodule 231 may be based on the received input coding activity (e.g., Figure 1 122 in the example above) to generate a search query. For example, the prediction submodule 231 may also determine when to trigger a code search, and the generation of the search query indicates the time point at which the code search is triggered. The search submodule 233 may determine one or more data sources for the code search based on, for example, a user configuration of preferences, past user behavior indicating preferences, the search query, the type of coded data source, and / or the like. The search submodule 233 may further generate a customized query based on each data source, and transmit the customized query to a corresponding API (e.g., Figure 1 The extraction submodule 232 may extract the code snippets from each search result that is originally in the form of a link to a web page and / or cloud file. The ranking submodule 234 may rank the code snippets, for example, based on completeness, reputation of the data source, relevance, and / or the like. Figure 5 The additional functions of the submodules 231 to 234 are further described.

[0044] Some examples of computing devices, such as computing device 200, etc., may include a non-transitory tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 210), may cause the one or more processors to perform the processes of the method. Some common forms of machine-readable media that may include the processes of the method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media from which a processor or computer is suitable for reading.

[0045] Figure 3 is an example of implementing an embodiment according to the present invention. Figure 2 A simplified diagram of the neural network structure of the code search module 230. In one embodiment, the code search module 230 and / or one or more of its submodules 231 to 234 may be connected to the code search module 230 via Figure 2 The artificial neural network structure shown in is implemented. The neural network includes a computing system built on a collection of connected units or nodes called neurons (e.g., 244, 245, 246). Neurons are often connected by edges, and adjustable weights (e.g., 251, 252) are often associated with the edges. Neurons are often aggregated into layers so that different layers can perform different transformations on corresponding inputs and output the transformed input data to the next layer.

[0046] For example, the neural network architecture may include an input layer 241, one or more hidden layers 242, and an output layer 243. Each layer may include multiple neurons, and the neurons between layers are interconnected according to a specific topology of the neural network topology. The input layer 241 receives input data (e.g., Figure 2 240), such as user coding activities (e.g., Figure 1 122 in ), search queries input by users, and / or the like. The number of nodes (neurons) in the input layer 241 may be determined by the dimension of the input data (e.g., the length of a vector giving an input example). Each node in the input layer represents a feature or attribute of the input.

[0047] The hidden layer 242 is an intermediate layer between the input layer and the output layer of the neural network. Figure 3 Only two hidden layers 242 are shown for illustrative purposes, and any number of hidden layers may be utilized in a neural network structure. The hidden layers 242 may extract and transform input data through a series of weighted calculations and activation functions.

[0048] For example, Figure 2 As discussed in , the code search module 230 receives an input 240 of user coding activity and transforms the input into an output 250 of searching for code snippets. To perform the transformation, each neuron receives an input signal, performs a weighted sum of the input according to the weights assigned to each connection (e.g., 251, 252), and then applies the activation function associated with the corresponding neuron (e.g., 261, 262, etc.) to the result. The output of the activation function is passed to the next layer of neurons or used as the final output of the network. The activation function can be the same or different across different layers. Example activation functions include, but are not limited to, Sigmoid, hyperbolic tangent, rectified linear unit (ReLU), leaky ReLU, softmax and / or the like. In this way, after multiple hidden layers, the input data received at the input layer 241 is transformed into quite different values ​​that indicate data characteristics corresponding to the tasks that the neural network structure is designed to perform.

[0049] The output layer 243 is the final layer of the neural network structure. The output layer 243 generates the output or prediction of the network based on the calculations performed in the previous layers (e.g., 241, 242). The number of nodes in the output layer depends on the nature of the task being solved. For example, in a binary classification problem, the output layer can consist of a single node representing the probability of belonging to one class. In a multi-class classification problem, the output layer can have multiple nodes, each node representing the probability of belonging to a specific class.

[0050] Therefore, the code search module 230 and / or one or more of its submodules 231 to 234 may include a transformation neural network structure having layers of neurons and weights and activation functions describing nonlinear transformations at each neuron. Such a neural network structure is often implemented on one or more hardware processors 210 (such as a graphics processing unit (GPU) or the like). The example neural network may be an example and / or analog of a neural model.

[0051] In one embodiment, the code search module 230 and its submodules 231 may be implemented by hardware, software, and / or a combination thereof. For example, the code search module 230 and its submodules 231 may include a specific neural network structure implemented and run on various hardware platforms 350, such as but not limited to a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), an ASIC (Application-Specific Integrated Circuit), a dedicated AI accelerator such as a TPU (Tensor Processing Unit), and a dedicated hardware accelerator specifically designed for the neural network calculations described herein, and / or the like. Example specific hardware for the neural network structure may include, but is not limited to, Google Edge TPU, Deep Learning Accelerator (DLA), NVIDIA's AI-focused GPU, and / or the like. The hardware platform 350 used to implement the neural network structure is specifically configured depending on factors such as the complexity of the neural network, the scale of the task (e.g., training time, input data scale, size of the training data set, etc.) and the expected performance.

[0052] In one embodiment, the neural network-based code search module 230 and one or more of its submodules 231 to 234 can be trained by iteratively updating the underlying parameters of the neural network (e.g., weights 251, 252, etc., bias parameters and / or coefficients in activation functions 261, 262 associated with neurons) based on a loss target. For example, during forward propagation, training data such as past encoding activity is fed into the neural network. The data flows through the layers 241, 242 of the network, where each layer performs calculations based on its weights, biases, and activation functions until the output layer 243 produces the output 250 of the network (such as a predicted code search query, etc.).

[0053] The output generated by the output layer 243 is compared to the expected output (e.g., "ground truth", such as the corresponding ground truth label) (e.g., the actual code search query corresponding to the coding activity from the training data) to calculate a loss function that measures the difference between the predicted output and the expected output. For example, the loss function can be a cross entropy, a mean square error (MSE), and / or the like. Taking into account the loss, the negative gradient of the loss function is calculated separately for each weight of each layer. The negative gradient is calculated iteratively one layer at a time from the last layer 243 of the neural network to the input layer 241 backward. These gradients quantify the sensitivity of the network's output to changes in parameters. The chain rule of calculus is applied to efficiently calculate these gradients by propagating the gradients backward from the output layer 243 to the input layer 241.

[0054] An optimization algorithm is used to update the parameters of the neural network backward from the last layer to the input layer based on the calculated negative gradient (back propagation) to minimize the loss. Back propagation from the last layer 243 to the input layer 241 can be performed on multiple training samples in multiple iterative training cycles (epochs). In this way, the parameters of the neural network can be gradually updated in the direction that brings less or minimized losses, indicating that the neural network has been trained to generate predicted output values ​​that are closer to the target output value with improved prediction accuracy. Training can continue until a stopping criterion is met (such as reaching a maximum number of cycles (epochs) or achieving satisfactory performance on validation data, etc.). At this point, the trained network can be used to predict new unseen data (such as IDE based on automatic search, etc.).

[0055] Thus, the training process transforms the neural network into an "updated" trained neural network with updated parameters, such as weights, activation functions, and biases. Thus, the trained neural network improves the neural network technology in the cloud-based search system.

[0056] Figure 4 is suitable for implementing Figure 1 4 and other embodiments described herein. In one embodiment, the system 400 includes a user device 410 that can be operated by a user 440, data provider servers 445, 470 and 480, a server 430, and other forms of devices, servers and / or software components that operate to perform various methods according to the described embodiments. Exemplary devices and servers may include devices that can be similar to Figure 2 The computing device 200 described in Or other suitable devices, stand-alone and enterprise-level servers based on the OS of the device and / or server. It is understood that Figure 4The devices and / or servers illustrated in the examples may be deployed in other ways, and the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, and may be performed by a greater or lesser number of devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0057] User device 410, data provider servers 445, 470 and 480, and server platform 430 (e.g., similar to Figure 1 The search server 110 in the example embodiment may communicate with each other via a network 460. The user device 410 may be utilized by a user 440 (e.g., a driver, a system administrator, etc.) to access various features available to the user device 410, which may include processing and / or applications associated with the server 430 to receive output data anomaly reports.

[0058] User device 410, data provider server 445, and server 430 may each include one or more processors, memories, and other appropriate components for executing instructions (such as program code and / or data stored on one or more computer-readable media) to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media such as memory or data storage devices internal and / or external to various components of system 400, and / or may be accessed via network 460.

[0059] The user device 410 may be implemented as a communication device that may utilize appropriate hardware and software configured for wired and / or wireless communication with the data provider server 445 and / or the server 430. For example, in one embodiment, the user device 410 may be implemented as an autonomous vehicle, a personal computer (PC), a smart phone, a laptop / tablet computer, a watch with appropriate computer hardware resources, glasses with appropriate computer hardware (e.g., GOOGLE ), other types of wearable computing devices, implantable communication devices, and / or other types of computing devices capable of transmitting and / or receiving data (such as from of Although only one communication device is shown, multiple communication devices may operate similarly.

[0060] Figure 4The user device 410 includes a user interface (UI) application 412 and / or other applications 416, which may correspond to executable processes, procedures, and / or applications with associated hardware. For example, the user device 410 may receive an instruction code snippet (e.g., Figure 1 125) and displays the message via UI application 412 (e.g., see 7A to 8D ). In other embodiments, the user device 410 may include additional or different modules having dedicated hardware and / or software as needed.

[0061] In various embodiments, the user device 410 includes other applications 416 that may be desired in a specific embodiment to provide features to the user device 410. For example, other applications 416 may include an IDE application for software development, which often includes a code editor application that can be operated with the UI application 412, and a compiler. For another example, other applications 416 may include a security application for implementing client-side security features, a programming client application for engaging with an appropriate application programming interface (API) through a network 460, or other types of applications. Other applications 416 may also include communication applications, such as email, texting, voice, social networking, and IM applications that allow users to send and receive emails, calls, texts, and other notifications through a network 460. For example, other applications 416 may be email or instant messaging applications that receive prediction result messages from a server 430. Other applications 416 may include device interfaces and other display modules that can receive input and / or output information. For example, other applications 416 may include a software program for asset management that can be executed by a processor, including a graphical user interface (GUI) configured to provide an interface to a user 440 to view code snippets.

[0062] The user device 410 may further include a database 418 stored in a transient and / or non-transitory memory of the user device 410, which may store various applications and data and be utilized during execution of various modules of the user device 410. The database 418 may store a user profile related to the user 440, predictions previously viewed or saved by the user 440, historical data received from the server 430, and / or the like. In some embodiments, the database 418 may be local to the user device 410. However, in other embodiments, the database 418 may be external to the user device 410 and accessible to the user device 410, including a cloud storage system and / or database accessible via the network 460.

[0063] The user device 410 includes at least one network interface component 417 adapted to communicate with the data provider server 445 and / or the server 430. In various embodiments, the network interface component 417 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices (including microwave, radio frequency, infrared, Bluetooth, and near field communication devices).

[0064] Data provider server 445 may correspond to a server hosting database 419 to provide training data sets including user coding activities, corresponding search queries, and code snippets to server 430. Database 419 may be implemented by one or more relational databases, distributed databases, cloud databases, and / or the like.

[0065] The data provider server 445 includes at least one network interface component 426 adapted to communicate with the user device 410 and / or the server 430. In various embodiments, the network interface component 426 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices (including microwave, radio frequency, infrared, Bluetooth, and near field communication devices). For example, in one implementation, the data provider server 445 may send asset information from the database 419 to the server 430 via the network interface 426.

[0066] Server 430 may accommodate Figure 2 In some implementations, the code search module 230 may receive data from the database 419 at the data provider server 445 via the network 460 to generate code snippets. The generated code snippets may also be sent to the user device 410 via the network 460 for the user 440 to review.

[0067] The database 432 may be stored in a transient and / or non-transitory memory of the server 430. In one implementation, the database 432 may store data obtained from the data provider server 445. In one implementation, the database 432 may store parameters of the code search module 230. In one implementation, the database 432 may store previously searched code snippets, code projects previously written by a user, previous coding activities of a user, and / or the like.

[0068] In some embodiments, database 432 may be local to server 430. However, in other embodiments, database 432 may be external to and accessible to server 430, including a cloud storage system and / or a database accessible via network 460.

[0069] The server 430 includes at least one network interface component 433 adapted to communicate with the user device 410 and / or the data provider server 445, 470, or 480 via the network 460. In various embodiments, the network interface component 433 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices (including microwave, radio frequency (RF), and infrared (IR) communication devices).

[0070] Network 460 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 460 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, network 460 may correspond to a small-scale communication network (such as a private network or a local area network, etc.) accessible by various components of system 400, or a larger-scale network (such as a wide area network or the Internet, etc.).

[0071] Figure 5 2 is a simplified diagram illustrating an example architecture of a code search platform implemented at a search server according to an embodiment described herein. The code search module 230 may include Figure 1 The search server 110 and / or Figure 3 For example, the code search module 230 may be based on the software and hardware platform implemented at the server 430 in Figure 3 It is implemented using the neural network structure shown in .

[0072] In one embodiment, prediction submodule 231 may receive any of user query 502, user coding activity 504, and / or other context 505. For example, upon receiving user query 502 (e.g., user manual input related to a search provided by a user, e.g., "sort a list Python"), prediction submodule 231 may determine to immediately pass user query 502 to search submodule 233 to initiate a code search.

[0073] For example, if user query 502 is not received, prediction submodule 231 can predict search query based on user coding activity 504 and other context 505. User coding activity 504 can include what user is coding, how long user has typed, when user moves to new row, when user pauses and how long to pause, whether current row contains function code, user cursor movement of just scrolling up and down indicating user's attention to specific code segment, content of row before and after current row selected by user, and / or the like. Other context 505 can include code project previously written by user, code file opened at the same time, other search words inputted by user on separate browser window, and / or the like, which can provide useful context information when determining potential search demand for individual coding. In certain embodiments, this context information 505 can further include user preference, user's previous search, trend of search activity and other context information, to determine additional useful search elements that will assist user when coding.

[0074] In one embodiment, the prediction submodule 231 can concatenate input information such as user coding activities 504, user queries 502 (optional), and other contexts 505 to the input sequence of tokens and generate a predicted code search query. The prediction can be performed periodically, intermittently, and / or continuously based on the continuously updated user coding activities 504. In one implementation, in the absence of a direct user query 502 for code search, for example, when the user activity 122 indicates that the user has an active IDE window but has been paused for more than a time threshold, the user has scrolled up and down to check a group of lines for more than a certain number of times, an error has been detected at the current coding position, and / or the like, the prediction submodule 231 can also perform predictions related to whether the input sequence of user coding activities 504 and other contexts 505 will trigger a code search.

[0075] The prediction submodule 231 may be trained on a dataset of previous coding activities 504, previous context 505, and (optionally) previous user queries 502, as well as corresponding ground truth code search queries associated with the coding activities.

[0076] The search submodule 233 can receive a code search query from the prediction submodule 231 and then determine a list of data sources to search. In one implementation, the search submodule 233 can retrieve a predefined list of data sources that have been pre-classified as relevant to a coding library, such as StackOverflow, Tutorials Point, etc. In another implementation, in a manner similar to that described in co-pending and co-owned U.S. non-provisional application No. 17 / 981,102 filed on November 4, 2022, the search submodule 233 can use a prediction module to predict priority data sources for the search based on a concatenation of the code search query, coding activity 504, and / or other contextual information 505.

[0077] The search submodule 233 may then send an encoded search query customized for each identified data source to each search API 522a - 522n and receive a list of search results from each search API 522a - 522n.

[0078] In some embodiments, the ranking submodule 234 may optionally rank the list of search applications 522a to 522n for searching. Figure 1 522a corresponds to a specific data source 103a to 103n in . For example, search application 522a corresponds to a search application configured to search within a database of "StackOverflow"; search application 522b corresponds to a search application configured to search within a database of "Tutorials Point"; and / or the like. The ranking submodule 234 uses an input sequence including a user query 502, encoding activity 504, and other context 505 to score multiple search applications 522a to 522n by running the input sequence through a neural network model once for each search application 522a to 522n. In this way, the ranking submodule 234 can rank the search results from the list of data sources via the search APIs 522a to 522n.

[0079] For example, if users consistently employ code search results from “StackOverflow” and this information is reflected in the other context 505 , the ranking submodule 234 may rank search results from the “StackOverflow” API higher.

[0080] Search results from the search APIs 522a to 522n often take the form of links to web pages or cloud files in various data sources. A ranked list of search results may be passed from the ranking submodule 234 to the extraction submodule 232 .

[0081] The extraction submodule 232 can follow the links of the search results and extract the code snippets from the content on the web page or cloud file. The code snippets 531a to 531n can then be delivered according to the ranking list based on their respective data sources. In one implementation, the extraction submodule 232 can further verify whether the code snippets extracted from the web page are complete, and can adjust the ranking by giving priority to high-quality code snippets.

[0082] For example, the code snippets 531a to 531n are then transmitted to a user device for display via a graphical user interface or some other type of user output device. For example, the code snippets 531a to 531n can be grouped and presented in the form of a list of user-engageable elements, each of which displays an icon representing each corresponding search application (data source), such as within an IDE window or within a browser window.

[0083] Figure 6 is an example of a method based on some embodiments described herein. Figures 1 to 5 An example logical flow chart of a method for performing real-time code search in an IDE using the code search framework and architecture shown in . One or more of the processes of method 600 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which executable code, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes. In some embodiments, method 600 is coupled to a code search module 230 (e.g., Figures 2 to 5 ) operation.

[0084] At step 602, a client component may be provided and connected to a user device (e.g., Figure 1 120) is installed together with an IDE implemented on the client, and the client component can monitor user activities related to the code segment (for example, Figure 1 122 in the above).

[0085] At step 604, on a search server (eg, Figure 1 A neural network-based prediction model (e.g., Figures 2 to 5230 and / or its submodule 231) can generate a code search query based at least in part on the first portion of the code segment and the monitored user activity received from the user device. For example, the code search query is further determined based on a user activity event indicating a user intent for a code search (e.g., triggered by a long pause, a user scrolling up and down the code segment, and / or the like). For another example, the code search query is further determined based on identifying an error within the first portion of the code segment.

[0086] At step 606, the search server can then access the network (e.g., Figure 4 460) via one or more search application programming interfaces (APIs) (e.g., Figure 1 112a to 112n or Figure 5 522a to 522n) transmit one or more search inputs customized according to the code search query to corresponding data sources that have been pre-categorized for code search (e.g., Figure 1 103a to 103n).

[0087] At step 608 , the search server may receive search results responsive to the code search query from one or more search APIs, including links to web pages containing code snippets.

[0088] At step 610, the search server may extract the code snippet from the web page following the link. In some implementations, the ranking model (e.g., Figure 2 and Figure 5 234) in can rank the search results, and thus make the code snippets displayed in the user interface within the IDE according to the ranking.

[0089] At step 612, the search server may send the code snippet to the user device, thereby causing a user-engageable widget displaying the code snippet to be displayed at a user interface within the IDE. For example, each user-engageable widget may take the form of a panel having a visual element indicating a data source (e.g., "Stack Overflow" and / or the like). At step 612, the client component receives a user selection of the first code snippet, thereby causing the first code snippet to be automatically integrated into the code snippet upon user selection. For example, the automatic integration of the first code snippet includes replacing a sub-portion of the code snippet that has been identified as being associated with a code error with the first code snippet.

[0090] In some implementations, steps 604 to 610 may be performed on a search server (e.g., Figure 1110 in . In another implementation, any of steps 604 to 610 may be performed by a client component at a user device. For example, a client component installed in an IDE at a user device may determine when step 604 is triggered from step 602. When the client component detects that the user has paused at a certain line of code for more than a period of time (e.g., 5 minutes, 10 minutes, etc.), scrolled up and down a code segment, entered a question in a comment section, and / or the like, the client component may send such coding activities and related code portions to a search server. The search server may then engage an NLP model to generate a code search query.

[0091] Figures 7A to 7C An example UI diagram illustrating automatic code search within an IDE according to an embodiment described herein is provided. Fig. 7A As shown, the search system can identify errors in the user's written code based on the user's coding activities. The detected errors can trigger Figure 2 and Figure 5 The prediction submodule 231 in the example can generate code search queries based on the user's coding activity and other contexts, and provide in-line suggestions to the user. Thus, the search system can identify errors in the user's written code without prompting, and provide the user with in-line suggestions with grammatically correct code that can resolve the errors. In such an embodiment, the search system can utilize Figure 5 In some embodiments, the search results may be shown as code snippets included inline in the text, where the user can indicate whether to accept the search results, view different search results, or ignore the search results. As determined by the search system or the user, the code may replace one or more lines of written code, or may be inserted in one or more new lines of the code.

[0092] To detect errors in code, for example, the search system can use natural language processing to parse comments inserted by a user into the written code, identify areas in the code that match and do not match the user's comments, and provide suggestions for the code that can implement the user's desired functionality as written in the user's comments. Thus, the search system can identify when the written code does not match the user's intent and propose fixes to the code. In some embodiments, the search system can utilize a neural network to identify errors or mismatches with the user's intent and provide suggestions or fixes to the code.

[0093] In some embodiments, Figure 7BAs shown, the search system uses natural language processing to parse comments inserted by users into written code (e.g., "#python open file") and identifies potential search queries that may be useful to the user, thereby performing a search and displaying search results in a side panel. Figure 7C As shown, the user can select "Try Solution 0" to automatically merge the code snippet into the code.

[0094] FIG. 8A to FIG. 8D An example UI diagram illustrating a code search within a search browser window according to embodiments described herein is provided. In one example, a user may be writing a "switch statement" in Java. In a traditional search, a list of results may be presented to the user, who would then have to visit various web links to find code examples. Fig. 8A As shown, in a search for "switch statement java", the search results can be a list of code snippets presented on a panel, rather than a direct link to a website, the search system extracts relevant code snippets and displays the results in a panel on the search screen. For example, these code snippets may appear in a collection of panels from a tutorial-based website. The user can then scroll through the snippet panels to view different results or choose to see more information related to the snippet in a specific panel.

[0095] like Figure 8B As shown, in another example, when the user may determine that they want to see all snippets from the selected source, the search system can then display these snippets in side panels. These panels can display information that the search system determines to be relevant to the search query, such as code snippets and corresponding text explaining the code snippets. Various results from a single source can be collected by the search system, and multiple panels can be generated, and the user can then scroll through these multiple panels to view various methods or information related to the search query.

[0096] like Figure 8C As shown, the user can select the "Try for yourself" button so that the code snippet in each panel can be copied to the clipboard, and the user can paste the code snippet into the IDE.

[0097] like Fig.8DAs shown, for another example, for a data source that is a discussion forum such as Stack Overflow, such a forum may contain one or more answers in response to questions similar to the search query. The panel may include questions and related code snippets, as well as all code snippets from the answers and corresponding explanations and comments. The user can scroll through different forum questions and responses related to the search query in the same way as other code snippets.

[0098] Where applicable, the various embodiments provided by the present disclosure may be implemented using hardware, software, or a combination of hardware and software. In addition, where applicable, the various hardware components and / or software components set forth herein may be combined into composite components including software, hardware, and / or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and / or software components set forth herein may be separated into subcomponents including software, hardware, or both without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components, and hardware components may be implemented as software components.

[0099] Software (such as program code and / or data) according to the present disclosure may be stored on one or more computer-readable media. It is also contemplated that the software identified herein may be implemented using one or more general or special-purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the ordering of the various steps described herein may be changed, combined into composite steps, and / or separated into sub-steps to provide the features described herein.

[0100] The various features and steps described herein may be implemented as a system comprising one or more memories storing the various information described herein and one or more processors coupled to the one or more memories and a network, wherein the one or more processors are operable to perform the steps described herein; implemented as a non-transitory machine-readable medium comprising a plurality of machine-readable instructions, which, when executed by one or more processors, are adapted to cause the one or more processors to perform methods comprising the steps described herein, as well as methods performed by one or more devices (such as hardware processors, user devices, servers, and other devices described herein).

Claims

1. A processor-implemented method for performing real-time code search in an integrated development environment (IDE), the method comprising: monitoring user activity associated with the code snippet via a client component implemented on a user device in conjunction with the IDE; determining, by a neural network-based predictive model implemented on one or more hardware processors, a code search query based at least in part on the first portion of the code segment and monitored user activity; Transmitting, via a first search application programming interface (i.e., a first search API), a first search input customized according to the code search query to a first data source that has been pre-categorized for code search, over a network; receiving, from the first search API, a first search result in response to the code search query, the first search result comprising a first link to a first webpage containing a first code snippet; extracting the first code snippet from the first web page following the first link; and A first user-engageable widget displaying the first code snippet is caused to be displayed at a user interface within the IDE.

2. The method according to claim 1, wherein: The code search query is further determined based on a user activity event indicative of a user intent for the code search.

3. The method according to claim 1, wherein: The code search query is further determined based on identifying an error within the first portion of the code segment.

4. The method according to claim 1, further comprising: transmitting, via a second search application programming interface (second search API) and through the network, a second search input customized according to the code search query to a second data source that has been pre-categorized for code search; receiving, from the second search API, a second search result in response to the code search query, the second search result comprising a second link to a second webpage containing a second code snippet; extracting the second code snippet from the second web page following the second link; Ranking the first search result and the second search result by a ranker model; as well as The first user-engageable widget displaying the first code snippet and the second user-engageable widget displaying the second code snippet are caused to be displayed at a user interface within the IDE according to the ranking.

5. The method according to claim 1, wherein: The first user engageable widget includes a visual element indicative of the first data source.

6. The method according to claim 1, wherein: The monitored user activities are transmitted to a search server, and wherein the neural network-based predictive model is implemented at the search server, and wherein the method further comprises: The first code snippet is transmitted to the user device, thereby causing display of a first user-engageable widget.

7. The method according to claim 1, wherein: The neural network-based predictive model is implemented by the client component at the user device.

8. The method according to claim 1, wherein: The client component receives a user selection of the first code snippet, thereby causing the first code snippet to be automatically integrated into the code segment upon the user selection.

9. The method according to claim 8, wherein: Automatic integration of the first code snippet includes replacing a sub-portion of the code segment that has been identified as being associated with a code error with the first code snippet.

10. A system for real-time code search in an integrated development environment (IDE), the system comprising: a communication interface to receive user activity associated with the code segment via a client component implemented on a user device together with the IDE; A memory storing a neural network-based model and a plurality of processor executable instructions; as well as One or more processors execute the instructions to perform operations including: determining, by a neural network-based predictive model implemented on one or more hardware processors, a code search query based at least in part on the first portion of the code segment and monitored user activity; transmitting, via a first search application programming interface (API), a first search input customized according to the code search query to a first data source that has been pre-categorized for code search over a network; receiving, from the first search API, a first search result in response to the code search query, the first search result comprising a first link to a first webpage containing a first code snippet; extracting the first code snippet from the first web page following the first link; and A first user-engageable widget displaying the first code snippet is caused to be displayed at a user interface within the IDE.

11. The system according to claim 10, wherein: The code search query is further determined based on a user activity event indicative of a user intent for the code search.

12. The system according to claim 10, wherein: The code search query is further determined based on identifying an error within the first portion of the code segment.

13. The system according to claim 10, wherein: The operations also include: transmitting, via a second search application programming interface (second search API) and through the network, a second search input customized according to the code search query to a second data source that has been pre-categorized for code search; receiving, from the second search API, a second search result in response to the code search query, the second search result comprising a second link to a second webpage containing a second code snippet; extracting the second code snippet from the second web page following the second link; Ranking the first search result and the second search result by a ranker model; and The first user-engageable widget displaying the first code snippet and the second user-engageable widget displaying the second code snippet are caused to be displayed at a user interface within the IDE according to the ranking.

14. The system according to claim 10, wherein: The first user engageable widget includes a visual element indicative of the first data source.

15. The system according to claim 10, wherein: The client component receives a user selection of the first code snippet, thereby causing the first code snippet to be automatically integrated into the code segment upon the user selection.

16. The system of claim 15, wherein: Automatic integration of the first code snippet includes replacing a sub-portion of the code segment that has been identified as being associated with a code error with the first code snippet.

17. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for performing real-time code search in an integrated development environment (IDE), the instructions being executed by one or more processors to perform operations comprising: monitoring user activity associated with the code snippet via a client component implemented on a user device in conjunction with the IDE; determining, by a neural network-based predictive model implemented on one or more hardware processors, a code search query based at least in part on the first portion of the code segment and monitored user activity; Transmitting, via a first search application programming interface (API), a first search input customized according to the code search query to a first data source that has been pre-categorized for code search over a network; receiving, from the first search API, a first search result in response to the code search query, the first search result comprising a first link to a first webpage containing a first code snippet; extracting the first code snippet from the first web page following the first link; and A first user-engageable widget displaying the first code snippet is caused to be displayed at a user interface within the IDE.

18. The processor-readable non-transitory storage medium of claim 17, wherein: The code search query is further determined based on a user activity event indicative of a user intent for the code search.

19. The processor-readable non-transitory storage medium of claim 17, wherein: The code search query is further determined based on identifying an error within the first portion of the code segment.

20. The processor-readable non-transitory storage medium of claim 1, wherein: The operations also include: transmitting, via a second search application programming interface (second search API) and through the network, a second search input customized according to the code search query to a second data source that has been pre-categorized for code search; receiving, from the second search API, a second search result in response to the code search query, the second search result comprising a second link to a second webpage containing a second code snippet; extracting the second code snippet from the second web page following the second link; Ranking the first search result and the second search result by a ranker model; and The first user-engageable widget displaying the first code snippet and the second user-engageable widget displaying the second code snippet are caused to be displayed at a user interface within the IDE according to the ranking.

Images