Artificial intelligence methods and systems for software development environments

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for providing, to an artificial intelligence model, a request with a prompt that includes contextual information from the software development environment comprising a file containing software code, a project containing the file, a summary of the project, a programming language identifier, a project entitlement, or some combination thereof; in response to the request with the prompt, receiving generative content from the artificial intelligence model that is based on the contextual information, wherein the generative content includes code, a project setting, an asset identifier, or documentation; and presenting the generative content including the code, the project setting, the asset identifier, or documentation, that is based on the contextual information, in the user interface of the software development environment.

Description

ARTIFICIAL INTELLIGENCE METHODS AND SYSTEMS FOR SOFTWARE DEVELOPMENT ENVIRONMENTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to US Provisional Application No. 63 / 631,236, filed on April 8, 2024, US Provisional Application No. 63 / 646,229, filed on May 13, 2024, US Provisional Application No. 63 / 657,898, filed on June 9, 2024, and US Provisional Application No. 63 / 694,105, filed on September 12, 2024, the contents of all of which are herein incorporated by reference in their entirety.BACKGROUND

[0002] Software applications can execute on devices used in a variety of contexts. For instance, a particular software application can execute in multiple different countries, with different interfaces for different languages. In some examples, the particular software application can execute on different hardware, different operating systems, e.g., as newer versions of operating systems are released, or a combination of both.SUMMARY

[0003] In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of providing, to an artificial intelligence model, a request with a prompt that includes contextual information from the software development environment comprising a file containing software code, a project containing the file, a summary of the project, a programming language identifier, a project entitlement, an asset identifier, or some combination thereof; in response to the request with the prompt, receiving generative content from the artificial intelligence model that is based on the contextual information, the generative content includes code, a project setting, the asset identifier, or documentation; and presenting the generative content including the code, the project setting, the asset identifier, or the documentation, that is based on the contextual information, in the user interface of the software development environment.

[0004] In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving, by the software development environment that depicts at least a portion of software code from a file in a code entry region, generative content that includes code; presenting, in a user interface of the software development environment, the generative content user interface element thattriggers a change in a presentation of the code from the generative content with software code from the file; determining a subset of the software code from the file that is not the generative content; determining a generative content location for the generative content using the subset of the software code from the file; and presenting, in the user interface of the software development environment, the generative content location for the code from the generative content.

[0005] In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of accessing, by the software development environment that depicts at least a portion of software code from a file in a code entry region of a user interface, a generative content location for presentation of generative content and that was determined using a subset of the software code from the file; and presenting, by the software development environment, the generative content location at which presentation of the generative content in the code entry region of the user interface will change in response to selection of a generative content user interface element.

[0006] In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of determining, by the software development environment executing on a device and presenting a user interface for editing software code, that at least some of the software code is of a different type than a type for which an artificial intelligence model included in the software development environment was trained; and in response to determining that at least some of the software code is of the different type than the type for which the artificial intelligence model was trained, presenting, by the software development environment, the artificial intelligence model message that indicates that the at least some of the software code is of the different type than the type for which the artificial intelligence model was trained.

[0007] Other implementations of this aspect include corresponding computer systems, apparatus, computer program products, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0008] The foregoing and other implementations can each optionally include one or more of the following features, alone or in combination.

[0009] In some implementations, providing the request can include providing a portion of software code highlighted in the user interface, a code subset before a cursor location in the user interface, a code subset after the cursor location in the user interface, all code from the file, all code from the project, a summary of code from the project, an asset catalog for the project, a current cursor location, a current user interface state, error data for an error presented in the user interface, or code from a second file that includes code other than code presented in the user interface.

[0010] In some implementations, providing the request can include providing code from the second file to cause the artificial intelligence model to provide generative content including code from the second file.

[0011] In some implementations, the artificial intelligence model might not be trained on at least a portion of the code in the second file.

[0012] In some implementations, the second file can include at least one function, class, structure, data type, or comment on which the artificial intelligence model was not trained.

[0013] In some implementations, the method can include: determining, using at least one of the prompt that includes contextual information or the generative content, a recommendation type; and selecting, from a plurality of presentation types, a presentation type for the recommendation type. Presenting at least the portion of the generative content can include presentation with the presentation type.

[0014] In some implementations, providing the request can include providing, by the development environment executing on a developer device and to the artificial intelligence model, the request.

[0015] In some implementations, providing the request can include providing the request to the artificial intelligence model executing on a system separate from the developer device.

[0016] In some implementations, the request for the generative content can identify one or more settings for the development environment or data that identifies a target platform for a project that includes the software code or both.

[0017] In some implementations, the method can include determining a cursor location of a cursor in the user interface; and selecting a code subset before the cursorlocation. Providing the contextual information can include providing the request for a code completion that includes the code subset of the software code.

[0018] In some implementations, providing the request can include includes type information for text before the cursor location.

[0019] In some implementations, the prompt can be one of a plurality of prompts presented in a user interface. The prompt provided in the request can be selected from the plurality of prompts.

[0020] In some implementations, providing the project entitlement can include providing an identification of one or more data sources when the project executes on an end-user device.

[0021] In some implementations, a response to the prompt can indicates whether generative content is available.

[0022] In some implementations, the artificial intelligence model can be a large language model.

[0023] In some implementations, presenting the generative content including the code, the project setting, the asset identifier, or the documentation can include: presenting, in a first file for the project, a first portion of code from the generative content; and presenting, in a second different file for the project, a second portion of code from the generative content.

[0024] In some implementations, the method can include detecting selection of the generative content user interface element that triggers addition of the generative content to or removal of the generative content from the code entry region of the user interface; and in response to detecting selection of the user interface element, updating, in the user interface, the presentation of the generative content at the generative content location by adding the generative content to the generative content location or removing the generative content from the generative content location.

[0025] In some implementations, the method can include: in response to detecting selection of the user interface element that triggers addition of the generative content to the code entry region of the user interface, storing, to memory, at least a portion of the generative content to the file depicted in the code entry region of the user interface.

[0026] In some implementations, the method can include, in response to detecting selection of the user interface element that triggers removal of the generative content from the code entry region of the user interface, deleting, from memory, at least a portionof the generative content from the file depicted in the code entry region of the user interface.

[0027] In some implementations, the method can include determining the subset of the software code from the file using a cursor location in the code entry region of the user interface or first software code from the file that is highlighted in the user interface.

[0028] In some implementations, the method can include maintaining the generative content that was determined using, as input to an artificial intelligence model, contextual information from the software development environment including a file containing software code, a project containing the file, a summary of the project, a programming language identifier, a project entitlement, or some combination thereof.

[0029] In some implementations, the method can include receiving, from a system that is separate from a developer device executing the software development environment, the generative content location that was determined using the subset of the software code from the file. Presenting the generative content location can be responsive to receiving the generative content location from the system.

[0030] In some implementations, the method can include receiving, from the system, the generative content. Presenting the generative content location can be responsive to receiving the generative content location and receiving the generative content.

[0031] In some implementations, the method can include receiving, from a system that is separate from a developer device executing the software development environment, an updated copy of the file that includes the generative content and indicates the generative content location.

[0032] In some implementations, the method can include determining, by a developer device executing the software development environment and using the subset of the software code from the file, the generative content location.

[0033] In some implementations, presenting can include presenting, at the generative content location in the code entry region of the user interface, at least a portion of the generative content.

[0034] In some implementations, the method can include maintaining location data for two or more generative content locations including the generative content location. Presenting at least the portion of the generative content can include presenting, at each of the two or more generative content locations, a corresponding different portion of the generative content.

[0035] In some implementations, the method can include detecting, by the software development environment, a time period since the software development environment received input; and determining whether the time period satisfies a time period threshold. Presenting at least the portion of the generative content can be responsive to determining that the time period satisfies the time period threshold.

[0036] In some implementations, the method can include maintaining, for a project that includes the file, a plurality of files including the file, each file of which defines a function, a class, or a data type; and receiving, from an artificial intelligence model executing on the developer device that is executing the software development environment, the generative content that includes a code summary that includes a proper subset of code from a second file from the plurality of files for the project. Presenting at least the portion of the generative content can include presenting at least some of the code summary that includes the proper subset of code from the second file.

[0037] In some implementations, presenting at least the portion of the generative content can include presenting, for each of a plurality of generative content portions and at the generative content location, the respective generative content portion.

[0038] In some implementations, the method can include detecting, for at least some generative content portions from the plurality of generative content portions, a relevance score that indicates a predicted relevance of the respective generative content portion to a project that includes the file. Presenting the respective generative content portions from the plurality of generative content portions can be according to the relevance scores.

[0039] In some implementations, presenting the generative content location can include: presenting, in a non-code entry portion of the user interface and at least partially concurrently with presentation of at least some of the software code from the file in a code entry portion of the user interface, the generative content location.

[0040] In some implementations, presenting, in the non-code entry portion of the user interface, the generative content location can include presenting, in the non-code entry portion of the user interface, difference data that indicates changes to the at least some of the software code from the file recommended by the generative content.

[0041] In some implementations, the difference data can include data from a difif file.

[0042] In some implementations, presenting the generative content location can include presenting, in a popup non-code entry portion of the user interface, the generative content location.

[0043] In some implementations, the method can include presenting the generative content that has a type determined using data for text before a cursor location of a cursor in the user interface.

[0044] In some implementations, presenting the generative content location can include: presenting, for a first portion of the generative content, a first generative content location in the file for a project; and presenting, for a second different portion of the generative content, a second generative content location in a second different file for the project.

[0045] In some implementations, the method can include presenting a first user interface element that triggers addition or removal of the first portion of the generative content from the first generative content location in the file; and presenting a second user interface element that triggers addition or removal of the second different portion of the generative content at the second generative content location in the second different file.

[0046] In some implementations, the method can include detecting, using at least some text from the software code, the different type for the software code.

[0047] In some implementations, detecting the different type for the software code can use a location of a cursor in the user interface.

[0048] In some implementations, the type can include at least one of a programming language, or a human language.

[0049] In some implementations, determining can include determining that all of the software code is of the different type than the type for which the artificial intelligence model included in the software development environment was trained.

[0050] In some implementations, determining can include determining that a confidence for any recommendations generated by the artificial intelligence model included in the software development environment would not satisfy a threshold confidence required to present a recommendation in the user interface for editing the software code.

[0051] In some implementations, the artificial intelligence model can include a large language model.

[0052] In some implementations, the method can include, in response to determining that at least some of the software code is of the different type than the typefor which the artificial intelligence model was trained, determining, by the software development environment, to skip providing a prompt for the software code to the artificial intelligence model.

[0053] This specification uses the term “configured to” in connection with systems, apparatus, and computer program components. That a system of one or more computers is configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform those operations or actions. That one or more computer programs is configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform those operations or actions. That specialpurpose logic circuitry is configured to perform particular operations or actions means that the circuitry has electronic logic that performs those operations or actions.

[0054] The subject matter described in this specification can be implemented in various implementations and may result in one or more of the following advantages. In some implementations, the systems and methods described in this specification can reduce computational resource usage, e.g., compared to other systems, by providing generative content such as programming recommendations that were generated using contextual information. The provision can include presenting the generative content such as a programming recommendation, a recommended location for recommended code or another type of code edit, or both, which can reduce computational resource usage, e.g., time and resources used to present search results for code in a user interface, to present different files in a user interface as a developer determines where to insert the code in a project or file, or both. In some implementations, the systems and methods described in this specification can reduce computational by presenting a message that indicates a software code type difference compared to other systems. For instance, when a type for the software code is different than a type for which a model was trained, a system can present a message and determine to skip using computation resources for a recommendation that is not likely to be responsive to a received prompt. In some implementations, the systems and methods described in this specification can use a local recommendation model to reduce potential latency, e.g., introduced by network communication when a recommendation model is on a remote system.

[0055] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the descriptionbelow. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0056] FIG. 1 depicts an example environment in which a developer device presents a user interface for an integrated development environment (“IDE”).

[0057] FIGS. 2A-B depict example user interfaces for an integrated development environment.

[0058] FIG. 3 is a flow diagram of an example process for presenting generative content.

[0059] FIG. 4 is a flow diagram of an example process for using a type of software code.

[0060] FIG. 5 is a block diagram of a computing system that can be used in connection with computer-implemented methods described in this specification.

[0061] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0062] Developers can use an integrated development environment (“IDE”) to write software code. Sometimes the developer might not know how to write a particular function, how a function was defined in another file of a software project, or a combination of both. As a result, the developer can use a lot of computational resources, e.g., processor cycles and time, to search for a solution regarding the particular function, look at the other file, or both.

[0063] An IDE can use an artificial intelligence model to generate generative content, such as programming recommendations for presentation in a user interface of the IDE. The generative content can include code, documentation, a recommended project setting, an asset identifier, or any combination of these. As part of code, the generative content can include references to other assets for a project, such as images maintained for a project. The IDE can provide, as input to the artificial intelligence model, a prompt, contextual information, or both. In response, the IDE can receive, from the artificial intelligence model, output that indicates the programming recommendation. Some examples of programming recommendations can include code to insert in an existing file presented in the IDE’s user interface, code for a new file for the same project, or both.The code to insert in the existing file can include one or more complete lines of code, e.g., that define the particular function; one or more partial lines of code, e.g., that complete existing code; lines for an entire file; or a combination of two or more of these, e.g., for different files.

[0064] The contextual information can be any appropriate contextual information, such as data from the file currently presented in the user interface; data from other files in the same project as the file presented in the user interface, e.g., all files in the project; a title of a file, e.g., the current file or another file; permissions data that indicates the other files in the same project and for which generative code can be generated; or a combination of two or more of these. In some examples, the contextual information can identify entitlements for the project, application specific data, a programming language version, or a combination of two or more of these. The application specific data can be one or more settings, e.g., for the project or the IDE or a combination of both.

[0065] FIG. 1 depicts an example environment 100 in which a developer device 102 presents a user interface for an integrated development environment (“IDE”) 104, e.g., a software development environment. The IDE uses an artificial intelligence model 106 to present generative content, generally referred to as programming recommendations, in the IDE’s user interface. The programming recommendations can include recommended edits to the code, such as recommendations for how to complete a line of code, e.g., presented in line with the code whether the line has any existing code on it or not; recommendations for code to add to an existing or a new file; recommendations for code to remove from a file, e.g., in addition to code to add to the file; or a combination of these. In some examples, the programming recommendation can include documentation for code. A recommendation for how to complete an empty line of code can be based on prior lines of code, a name for the current file, or a combination of both.

[0066] In this specification, a programming recommendation can include any of multiple different types of data that relate to programming. For instance, a programming recommendation can identify one or more of documentation data, source code data, or a combination of both. The documentation data can include data that indicates, for example, how to program. The source code data is sometimes referred to as code in this specification.

[0067] In some examples, the IDE 104 is a native IDE for the developer device 102 in that the IDE 104 executes on the developer device 102. As a result, the IDE 104can access contextual information about a project for which a file is presented in the IDE’s user interface. This contextual information can include project files 108 for the project, settings data 110, data entitlements 112 for the project, permissions data 113 for the project, other appropriate contextual information, or a combination of two or more of these.

[0068] As a result of the access to the contextual information, the IDE 104 can provide the contextual information as input to the artificial intelligence model 106, e.g., as part of a request for a programming recommendation. The artificial intelligence model 106 can be any appropriate type of artificial intelligence model, such as a machine learning model, or a large language model, to name a few examples. When implemented as a large language model, the artificial intelligence model 106 can be a neural network. The artificial intelligence model 106 can have any appropriate architecture, e.g., a transformer-based architecture such as a decoder-only transformer-based architecture, a recurrent neural network architecture, a deep learning architecture such as Mamba, a generative pre-trained transformer architecture, a multimodal large language model architecture, an autoregressive architecture, or a combination of two or more of these. The artificial intelligence model 106 can be trained for natural language processing tasks.

[0069] For instance, the artificial intelligence model 106 can receive a prompt and the contextual information as input. In some examples, the artificial intelligence model 106 can receive other input, e.g., data from a code database, developer documentation, application programming interface specifications, or a combination of two or more of these.

[0070] By providing contextual information to the artificial intelligence model 106, the IDE 104 can receive more accurate programming recommendations compared to other systems. For example, the artificial intelligence model 106 can be trained on a sample of code files. However, these code files do not include all potential permutations of how to implement any particular function let alone the various variables and functions defined for a specific project. As a result, a general artificial intelligence model can be aware of some ways to implement functions while being unaware of any project specific functions, which results in the general artificial intelligence model being unable to generate recommendations that are specific to the project for which a file is presented in the IDE’s user interface.

[0071] When the IDE 104 provides the contextual information to the artificial intelligence model 106, the IDE 104 enables the artificial intelligence model 106 togenerate tailored programming recommendations for the project. In contrast to the general artificial intelligence model that is unaware of any project specific functions, the artificial intelligence model 106 can generate programming recommendations given the actual data for the project, e.g., the functions, classes, structures, data types, variables, comments, images, other text, other assets, or a combination of two or more of these. As a result, when the artificial intelligence model 106 receives the contextual information, the artificial intelligence model 106 can, in some implementations, be a specialized model specific to the project to which the contextual information corresponds.

[0072] The programming recommendations can be for any one or more files in the project. For instance, given file dependencies in the project; programming organization, e.g., which functions are defined and called in which files; or a combination of both, the artificial intelligence model 106 can determine a programming recommendation that identifies changes in multiple different files. The programming organization can indicate coding practices, naming conventions, or the like. A first change can indicate one or more first edits to a data structure in a first file or implementation of a new method that uses that data structure. A second change can indicate one or more second edits to the use of that data structure in a second file, e.g., to the initialization of that data structure given the changes to the data structure itself or use of the new method or both.

[0073] When selecting the one or more files, the artificial intelligence model 106 can use the contextual information. The contextual information can indicate permissions data that identifies the files in the project for which the IDE 104 has write permissions, identified as files that can be edited as part of a programming recommendation, or a combination of both. By using the permissions data, the artificial intelligence model 106 can generate programming recommendations that are specific to the file, project, or both, being developed. For instance, if a project uses one or more third party files that should not be edited, by using the permissions data that indicates that these third-party files should not be edited, the artificial intelligence model 106 can generate a programming recommendation that only includes proposed code edits to the non-third party files in the project (assuming there aren’t any other permissions constraints on these files).

[0074] Although the artificial intelligence model 106 can receive the contextual information during inference for generation of a programming recommendation, the artificial intelligence model 106 will not, in some implementations, use that contextual information to update the model itself, e.g., for training. For example, the artificialintelligence model 106 will not use the contextual information for training absent explicit feedback, e.g., from a developer, that enables such use.

[0075] The artificial intelligence model 106 can be implemented on any appropriate combination of devices. In some implementations, one or more components of the artificial intelligence model 106 are implemented on the developer device 102, e.g., as part of the IDE 104. For instance, the IDE 104 can use a smaller artificial intelligence model 106 implemented on the developer device 102 to generate programming recommendations.

[0076] In some examples, one or more components of the artificial intelligence model 106 are implemented on a remote system 114, e.g., the cloud. In these examples, the developer device 102 can send, to the remote system, a prompt, the contextual information, or both.

[0077] The prompt can be any appropriate type of prompt, such as a user defined query, a suggested query, e.g., given some of the contextual information, or another appropriate type of data that forms part of a request for a programming recommendation. One example of a prompt is a request for the IDE 104 to “write a hello world function” or “complete the hello world function” that is already written, in part, in a code file. For instance, the developer device 102 can receive, using one or more input devices, data for the prompt, e.g., from the developer.

[0078] The prompt can be different data than the contextual information. For example, although the prompt can be generated, e.g., by the IDE 104, using the contextual information, in some implementations in which the developer device 102 provides a prompt to the remote system 114, the prompt is separate data from the contextual information.

[0079] In some examples, the developer device 102, e.g., the IDE 104, can provide the contextual information to the remote system 114 without providing the prompt. In these examples, the remote system 114 can determine a prompt for the artificial intelligence model 106 using the contextual information.

[0080] The IDE 104 receives, from the artificial intelligence model 106, the programming recommendation. When the artificial intelligence model 106 is implemented at least in part on the remote system 114, the IDE 104 can receive the programming recommendation from the remote system 114 via a network 118.Communication with the remote system 114 can use one or more network communication protocols, e.g., for transmission, reception, or both, of data.

[0081] Upon receiving the programming recommendation from the artificial intelligence model 106, the IDE 104 can determine whether to present the programming recommendation. In some examples, the IDE 104 always presents the programming recommendation. In some examples, the IDE 104 can receive a relevance score with the programming recommendation. The relevance score can indicate a likelihood that the programming recommendation is relevant to the project or file presented in the IDE’s user interface, e.g., given the contextual information.

[0082] FIGS. 2A-B depict example user interfaces 200a-b for an integrated development environment, e.g., the IDE 104. The user interfaces 200a-b include two portions: a recommendation portion 202a and a code entry portion 202b.

[0083] The code entry portion 202b receives input from an input device operated by a developer and that defines code for a software file, e.g., “Person View. swift”. The software file is part of a project, e.g., “Person View”, for which a developer device 102 maintains one or more project files 108. The project can be for an application that will execute on one or more end-user devices 116, such as a desktop, a laptop, a tablet, or a mobile phone.

[0084] The recommendation portion 202a can be a portion of the user interfaces 200a-b for which the developer device 102 provides one or more programming recommendations for presentation, e.g., to the developer, in contrast to the code entry portion 202b that receives input defining code entered into the IDE 104 by the developer using an input device. The recommendation portion 202a can be presented in any appropriate location in the user interfaces 200a-b. The recommendation portion 202a can be a non-code entry portion of the user interfaces 200a-b. The recommendation portion 202a can include a prompt field 204. The prompt field 204 receives input identifying a prompt for a recommendation.

[0085] The prompt field 204 can receive any type of free form text, e.g., and is not limited to selection of a predetermined phrase. For instance, the prompt field 204 can receive input for any one of the phrases “complete the hello world function”, “how do I add location information to a user interface”, “how can I present calendar information as part of the hello world function”, or “what is function greet(persom)”. In these examples, the first and third phrase include text that is specific to the project, e.g., “the hello world function”, while the second phrase regarding location information does not include anything specific to the file presented in the user interface 200a. The IDE 104 or another component can determine whether the third phrase includes anything specific about thefile presented in the user interface, e.g., based on whether the function “greet(person: )” is included in the file.

[0086] Upon detection of a search command for the prompt field 204, e.g., selection of a search button or selection of the “enter” key, the IDE 104 can send data for the prompt and contextual data for the file to the artificial intelligence model 106. In response, the IDE 104 receives generative content 206, shown in FIG. 2 A, e.g., one or more programming recommendations. The generative content 206 can be any appropriate type of programming recommendation. In FIG. 2A, the programming recommendation indicates that the hello world function should include “alignment: .leading” as input to “VStack”, e.g., having an automatic code recommendation type, as described in more detail below.

[0087] The recommendation portion 202a or another portion of the user interface 200 can include one or more user interface elements 208-210 for the generative content 206. Upon detecting selection of a user interface element 208-210, the IDE 104 can insert the code in a corresponding file, or remove the code from a corresponding file, e.g., when the code was previously inserted. For instance, an insert user interface element 208 can trigger insertion of the generative content 206 in the file currently presented in the user interface 200a. A new file user interface element 210 can trigger insertion of the generative content 206 in a new file for the project.

[0088] A remove user interface element (not shown) can trigger removal of the generative content 206 from the file currently presented in the user interface 200a. The insert user interface element 208 can change to the remove user interface element when the generative content 206 is added to the file. In some examples, when the programming recommendation is automatically inserted into a file, e.g., one or more files for the project, the user interface 200a can present the remove user interface element without having previously presented the insert user interface element for that particular programming recommendation.

[0089] The IDE 104 can determine which user interface elements, how many, or both, to present for the generative content 206 using contextual information. The contextual information can include data for the portion of the file that is currently depicted in the user interface 200a, e.g., the visible view port, compared to other data that is not depicted in the view port; data for the file as a whole; data from other files for the project, e.g., a proper subset of files for the project or all files for the project; an asset catalog for the project; a data model structure; code selected in the code entry portion202b of the user interface 200a; names of one or more objects for the project; or a combination of two or more of these. A name of an object can include a name for a function, a class, a structure, a data type, or a variable. A data model structure can be defined by a schema for the file, the project, or both.

[0090] An example of code selected in the code entry portion 202b can include the code at a location 212b. The IDE 104 can detect that the code at the location 212b is selected, e.g., highlighted based on input received from at least one input device. The input device can be controlled by a user, e.g., developer.

[0091] In some implementations, the IDE 104 can determine a likelihood that the generative content 206 might be used in other files for the project. When the IDE 104 determines that the likelihood satisfies, e.g., is greater than or equal to, a likelihood threshold, the IDE 104 can present the new file user interface element 210, one or more add to other files user interface elements, or any combination of both. When the IDE 104 determines that the likelihood does not satisfy the likelihood threshold, the IDE 104 can determine to skip presentation of the new file user interface element 210.

[0092] In some examples, the likelihood can represent, in addition to or instead of representing a likelihood of use in other files, a likelihood that the generative content 206 should be placed in the currently depicted file. This likelihood can be based on coding practices, naming conventions, other data such as the contextual data, e.g., given an analysis of how the project files 108 are structured generally, or a combination of these.

[0093] The add to other files user interface elements can cause edits to code in other existing files in the project, e.g., for which there are editing permissions. For instance, the generative content can include recommended edits to one or more other files in the project. The user interface 200a can include one add to other file(s) user interface element for each of the one or more other files, or a group of the other files, e.g., in the same package of the project, for all of the other files, or any combination of two or more of these. Selection of the corresponding add to other file(s) user interface element can trigger modification of the corresponding other files, presentation, in the user interface 200a of the other file with the recommended code edits, or any combination of both.

[0094] When the user interface 200a presents recommended code edits, the user interface can present one or more corresponding user interface elements for confirming addition of the edits, removing the edits, or both. For example, the user interface 200a can, after the developer device 102 receives generative content, display a view generative content user interface element. This user interface element can include or otherwise berelated to a generative content location for the generative content, such as the programming recommendation. The developer device 102 can receive input indicating selection of the view generative content user interface element. In response, the developer device 102 can present, in the current file or across multiple files, one or more code edits defined in the generative content. Although some of the examples described in this specification refer to a single file, similar examples apply for generative content that includes data for multiple files.

[0095] The developer device 102 can present, in the user interface 200a, one or more user interface elements for the presented code edits. These user interface elements can include a confirm edits user interface element, a remove edits user interface element, or both. The confirm edits user interface element can trigger saving the edits to the file, removal of any emphasis of the edits that is displayed in the user interface 200a, e.g., highlighting or underlining that flags the edits, or both. The remove edits user interface element can remove the proposed code edits from the file.

[0096] In some instances, the user interface 200a can present different portions of the generative content, e.g., programming sub-recommendations, at different locations in the same file. For instance, when a programming recommendation adds a new variable to a data structure, the programming recommendation can include a first edit in an initialization of an instance of the data structure and a second edit in how the data structure is accessed. When a programming recommendation includes different subrecommendations at different locations, the user interface 200ca can include one or more user interface elements for each of the sub-recommendations. For example, the user interface 200a can first display the view generative content user interface element. Upon detecting selection of the view generative content user interface element, the developer device 102 can update the display of the current file to include the different portions of the generative content at the different locations and, for at least some of the different portions, present corresponding user interface elements as described above.

[0097] The user interface 200a can include generative content location information 212a-b. The recommended location can indicate where the IDE 104 recommends placing the recommended code. The recommended location can indicate one or more lines, columns, characters, or a combination of these, at which the generative content 206 is recommended to be placed, e.g., such as the generative content location 212a shown in the recommendation portion 202a of the user interface 200a. In someexamples, the recommended location can include data that indicates whether the code will overwrite some existing code, e.g., replacing “18mazing” with “amazing”, or not.

[0098] In some examples, the user interface 200a can indicate the generative content location 212b in the code entry portion 202b. For instance, the user interface 200a can present a user interface element that shows where the recommended code will be added, e.g., without replacing any existing text in the file, or the existing text that will be replaced. In FIG. 2A, the generative content location 212b in the code entry portion 202b indicates the existing text recommended for replacement by the generative content 206.

[0099] The code entry portion 202b can include a user interface element that triggers insertion of the programming recommendation, e.g., an add to file user interface element. For instance, the generative content location 212b in the code entry portion 202b can trigger, upon selection, replacement of the text identified by the generative content location 212b with the generative content 206.

[0100] The user interface 200a can depict one or more other recommendations for the project, the file, or both. The IDE 104 can determine the recommendations using data for the prompt 204, other data for the project, or a combination of both.

[0101] For instance, in some examples, the IDE 104 can provide a request to the artificial intelligence model 106 that identifies the contextual information for the file, the project, or both. The request can include prompt data for the artificial intelligence model 106, e.g., “check for coding issues” or not.

[0102] In response, the IDE 104 can receive, from the artificial intelligence model 106, a programming recommendation for the file, the project, or both. The recommendation can indicate suggested code to add to the file, remove from the file, or a combination of both, e.g., to optimize an existing function written in the file. The recommendation can indicate one or more settings for the IDE 104, e.g., to optimize the IDE 104 for the particular project being developed, one or more settings for the project, or a combination of both.

[0103] In some examples, the one or more settings for the project can include one or more entitlements. An entitlement for a project, e.g., an application, can define rights or privileges that grant an executable, e.g., for the application, specific capabilities. An entitlement can identify one or more data sources on an end-user device 116 that will be used by the project when the project executes on the end-user device 116. For instance, when the prompt 204 is “how do I add location information to a user interface”, the IDE 104 can present a programming recommendation for a corresponding function along withan entitlement recommendation 214 to add a “map” entitlement to the project. In some examples, after analyzing the code using the artificial intelligence model 106, the IDE can present the entitlement recommendation 214, e.g., without presenting a programming recommendation.

[0104] When an entitlement is added to a project, an end-user device 116 will present information about the entitlement with information about the project. In this way, the end-user device 116 indicates types of personal information that correspond to a project. This can enable a corresponding end-user to determine whether to install an application generated from the project files 108.

[0105] In some implementations, the recommendation portion 202a can present one or more suggested prompts. The IDE 104 can use any appropriate process to determine the suggested prompts. Upon detecting selection of a suggested prompt, the IDE 104 can present the selected suggested prompt in the prompt field 204 and transmit a request for generative content for the prompt, e.g., to the remote system 114. Upon receiving generative content, the IDE 104 can present the corresponding generative content 206 in the recommendation portion 202a of the user interface 200a. The presentation of the corresponding generative content 206 can include presentation of any appropriate user interface elements, e.g., 208-210.

[0106] FIG. 2B depicts an example of another type of programming recommendation. In FIG. 2B, the programming recommendation can be part of a code completion recommendation. Here, the IDE 104 can receive, one or more programming recommendations from the artificial intelligence model 106. In response, the IDE 104 can present, in the code entry portion 202b of the user interface 200b, multiple code completion recommendations 218.

[0107] The code completion recommendation 218 can be for any appropriate type of data in the code entry portion 202b. For instance, the code completion recommendation 218 can be for a line without any text or a line with some text. The code completion recommendation 218 can be for an empty file, e.g., given the name of the file, other files in the project, or a combination of both. The code completion recommendation 218 can be for a file with at least some text. The data in the code entry portion 202b can be comments, programming code, or a combination of both.

[0108] In some implementations, the IDE 104 can request, from the artificial intelligence model 106, one or more programming recommendations upon detecting one or more threshold criteria. The threshold criteria can be a lack of user input, e.g., textinput, for a threshold time period, a lack of mouse movement for a threshold time period, detection of a potential programming error, or a combination of two or more of these. The threshold time period, e.g., for text input or mouse movement or both, can be one second.

[0109] For instance, the IDE 104 can determine that a cursor 220 in the code entry portion 202b has not moved for the threshold time period. The IDE 104 can determine that the user interface 200b is in focus for the developer device 102. In response to one or both of these determinations, the IDE 104 can present one or more code completion recommendations 218. A location of the cursor can be at a line in the existing file with one or more complete lines of code, one or more partial lines of code, an empty line without any code, or a file without any code.

[0110] The IDE 104 can present the code completion recommendations 218 at a recommended location, e.g., defined by a location of the cursor 220. For instance, the IDE 104 can determine a location of the cursor 220 in the user interface 200b and present, as the recommended location, the code completion recommendations 218 at the cursor location. In some examples, the user interface 200b can include location information 212 for the cursor which can indicate the recommended location for the code completion recommendations 218.[OHl] The IDE 104 can present the code completion recommendations 218 in any appropriate manner. For instance, the IDE 104 can present a first code completion generative content 222 in line with the cursor 220 location and other code completion generative content recommendations below the first code completion generative content 222. In some examples, the IDE 104 can present all of the code completion recommendations 218 below a location of the cursor.

[0112] The IDE 104 can determine a relevance score that indicates a likelihood that a corresponding programming recommendation is relevant to the file, the project, or both. When the IDE 104 presents multiple code completion recommendations 218, the IDE 104 can present the multiple code completion recommendations 218 according to the relevance scores for the corresponding recommendations. For instance, the IDE 104 can present, as the first code completion generative content 222, the code completion recommendation with the highest relevance score. The IDE 104 can present, as the second code completion recommendation that is just below the first code completion recommendation, the code completion recommendation with the second highest relevance score, and so on.

[0113] When the IDE 104 presents programming recommendations in the recommendation portion 202a, the IDE 104 can present multiple programming recommendations that are each responsive to the same prompt. The IDE 104 can present the multiple programming recommendations according to corresponding relevance scores. The IDE 104, the artificial intelligence model 106, or a combination of both, can determine the relevance scores using the contextual information for the file, the project, or both.

[0114] In some examples, the user interface 200b can present a summary of code, e.g., instead of or in addition to presenting a code complete option. For instance, the IDE 104 can maintain the project files 108 for the project. Upon detecting at least some of the one or more threshold criteria, the IDE 104 can request, from the artificial intelligence model 106, a code summary for a location in the code entry portion 202b of the user interface, e.g., given the location of the cursor 220, code presented in a view port for the IDE 104, or a combination of both. In response, the IDE 104 can receive, from the artificial intelligence model 106, a code summary that includes a subset of code from another file in the project. The code summary can present key features of the code, e.g., inputs, outputs, or a combination of both. The code summary can present a high-level review of code from another file without requiring presentation of all content from the other file, potentially saving computational resource usage, e.g., that would be required by switching back and forth repeatedly between files, by presentation of both files at the same time, or a combination of both. In this way, the IDE can present one or more code summaries instead of or in addition to the code complete recommendations 218.

[0115] The IDE 104 can receive input indicating that a package should be made for the project. The package can enable execution of the project on a device, e.g., the developer device 102 or one of the end-user devices 116. In some examples, the package can include a compiled application, e.g., along with any files required for execution.

[0116] The developer device 102 can provide the package to an end-user device 116. The provision can be by way of an application store, a website, or another appropriate system or process that provides the package to the end-user device 116. Upon receipt of the package, the end-user device 116 can execute the package, e.g., using any corresponding entitlements included with the package.

[0117] In some implementations, a recommendation type can be predicted. The recommendation type can represent a data type that is likely responsive to a determination to present a programming recommendation. The determination to present a programmingrecommendation can be responsive to receipt of user input for a programming recommendation, or a determination that a threshold time period has passed and a programming recommendation should be presented. The artificial intelligence model 106 can predict the recommendation type, e.g., given an input request for a programming recommendation.

[0118] There can be any appropriate number of recommendation types. For instance, the recommendation types can include a documentation recommendation type, a non-automatic code recommendation type, and an automatic code recommendation type. The non-automatic code recommendation type, the automatic code recommendation type, or both, can optionally include documentation, e.g., or might include only code data.

[0119] The developer device 102 can receive, from the artificial intelligence model 106, output that indicates a recommendation type. The output can include data that identifies, e.g., by name, index, or other appropriate data, the recommendation type. In some examples, the output identifies the recommendation type by indicating data responsive to the prompt provided to the artificial intelligence model 106. For instance, when the output is a programming recommendation that only includes or otherwise identifies code, the output can indicate an automatic code recommendation type given that the output only identifies code.

[0120] The developer device 102, e.g., the IDE 104, can use the output to determine a presentation format for the code recommendation. For instance, each of the multiple recommendation types can have a corresponding, e.g., and different, presentation format.

[0121] The automatic code recommendation type can include code for addition to one or more files in a project. Presentation of a programming recommendation with the automatic code recommendation type can include the developer device 102 automatically inserting the code into the one or more files in the project. For instance, while the IDE 104 presents existing code from the project in a code entry portion of the user interface, the IDE 104 can insert the recommended code into the presented subset of existing code. This can include making live changes to the existing code in the code entry portion of the user interface. The IDE 104 can present the live changes with an indication of what changes were made to the existing code, e.g., such that the indication represents a form of track changes, diff changes, or another appropriate type of presentation that identifies the recommended code that was added to the existing code.

[0122] In some examples, the IDE 104 can receive the programming recommendation. The IDE 104 can determine a recommendation type for the programming recommendation. The integrated development environment can present data for the programming recommendation using the recommendation type. In examples in which the recommendation type is automatic code, the integrated development environment can automatically, e.g., without user input, begin to insert lines of code from the programming recommendation into the existing code presented in the user interface, e.g., into data representing the corresponding file.

[0123] When the user input indicates a request for “insert a button,” the programming recommendation can have an automatic code recommendation type. The IDE 104 can automatically insert code into one or more files for the button upon receiving output from the artificial intelligence model 106.

[0124] The documentation recommendation type can include one or more links to documentation, data extracted from documentation, e.g., a summary of how to program in a particular manner or a particular algorithm, or a combination of both. Presentation of a programming recommendation with the documentation recommendation type can include presenting data for the programming recommendation only in one or more non-code entry portions of a user interface.

[0125] When the user input indicates a request for “what is a button,” the programming recommendation can have a recommendation type of documentation. The IDE 104 can present documentation that describes what a software button is, what actions a button can trigger, and how to write code for a button, to name a few examples.

[0126] The non-automatic code recommendation type can be a type for a programming recommendation that includes both documentation and code, or just recommended code that is not automatically inserted into existing code. In some implementations, instead of automatically inserting code into the source code presented in the user interface, the IDE 104 can present the code in one or more non-code entry portions of the user interface. For instance, the IDE 104 can present a panel or a popup that includes the recommended code, e.g., a diff presentation of at least a portion of the recommended code from the programming recommendation.

[0127] When the user input indicates a request for “how to insert a button,” the programming recommendation can have a non-automatic code recommendation type. The IDE 104 can determine the recommendation type. In response to determining the recommendation type, the IDE 104 can present recommended code and documentation inone or more non-code entry portions of the user interface. The programming recommendation can include both a description of what a button is and how to write code for a button along with recommended code for a button.

[0128] In some examples, recommended code can be customized given the project. For instance, if the project has a particular style of how to write code, e.g., written by another developer, or a particular comment style, the artificial intelligence model 106 can use the contextual information that includes data for the project to determine recommended code that aligns with either or both of those styles.

[0129] In some implementations, the IDE 104 can determine the recommendation type before providing input, e.g., including contextual data, to the artificial intelligence model 106. In these implementations, the IDE 104 can infer the recommendation type using data for the user input or another trigger for the prompt for a code recommendation, e.g., that no code has been entered and the IDE 104 should request a code completion recommendation. The artificial intelligence model 106 can use the recommendation type when generating the programming recommendation output.

[0130] In some implementations, a device in the environment 100 can select a least one of multiple artificial intelligence models 106 to process a request for a programming recommendation. For instance, the environment 100 can maintain multiple artificial intelligence models 106 each of which were trained to generate programming recommendations for different prompt types, e.g., trained using different training data, different processes, or a combination of both. Some examples of prompt types can be code completion, e.g., “complete the hello world function”; how do I, e.g., “how do I add location information to a user interface” or “how can I present calendar information as part of the hello world function”; or an explanation request, e.g., “what is function greet(persom)”. In some examples, the prompt types can include prompt trigger types such as a user defined prompt, e.g., when a prompt filed 204 in the IDE 104 receives input; or a code completion prompt, e.g., when the IDE 104 automatically generates a prompt without a prompt request from an input device, e.g., based on an amount of time since the input device provided input.

[0131] The device can use data from the prompt, e.g., received from the developer device’s 102 input device, from the contextual information, or a combination of both, to select at least one of the multiple artificial intelligence models 106. For example, the device can select one or more artificial intelligence models given the prompt type. When the prompt type is a code completion type, the device can select a first model from theartificial intelligence models. When the prompt type is an explanation request type, the device can select a second model from the artificial intelligence models. When the prompt type is a how do I type, the device can select a third model from the artificial intelligence models, can select both the first and second models, or a combination of both. When selecting both the first and second models, each of the models can provide a different type of output. For instance, the first model can generate code and the second model can generate documentation. The device can combine the outputs from both models to create the programming recommendation.

[0132] In some examples, when the device is the developer device 102, at least some of the artificial intelligence models 106 can be at different physical locations. For instance, the developer device 102 can maintain data for a code completion prompt while the remote system 114 maintains one or more models for one or more user defined prompts. This can enable more efficient presentation of programming recommendations using available computational resources, e.g., by using local resources for automatically generated programming recommendations for which there should be lower latency and using remote resources for user defined prompts for which a higher latency is acceptable. When the developer device 102 selects the local artificial intelligence model 106, the developer device 102 provides input to that local artificial intelligence model 106. When the developer device 102 selects the remote artificial intelligence models, the developer device 102 provides the request to the remote system 114. The remote system 114 can then use a single artificial intelligence model 106, select at least one of multiple artificial intelligence models, or a combination of both, to process the request.

[0133] The device can determine an order in which to provide input to different artificial intelligence models when two or more artificial intelligence models are selected. For instance, at least some output of one artificial intelligence model can be provided to another artificial intelligence model as input. This data provided to the other artificial intelligence model can include data that will be part of the programming recommendation presented by the IDE 104, other data generated by the first artificial intelligence model, e.g., specifically for use by the other artificial intelligence model, or a combination of both. By selecting an order in which the artificial intelligence models are triggered, the device can generate more accurate programming recommendations than might otherwise be generated.

[0134] In some examples, the device can perform prompt engineering using data for the selected artificial intelligence model 106, provide different input data to each ofmultiple selected artificial intelligence models 106, or a combination of both. For example, the device can use the prompt type to add, remove, or a combination of both, data for the input that will be provided to the corresponding artificial intelligence model. As a result, when multiple artificial intelligence models are used to process data for a single prompt, at least some of the models might receive different data as input.

[0135] An asset catalog for the project can include or otherwise identify assets for use in the project, such as images. The artificial intelligence model 106 can receive the asset catalog, or data for the asset catalog that identifies the assets in the catalog. The artificial intelligence model can use the data for the asset catalog to determine what assets are identified by the asset catalog, types of those assets, whether one of those assets might be relevant to a prompt, or any combination of these. For instance, when the prompt is to add an extra list item in a table row for “a loaf of bread” and one of the images in the asset catalog depicts a loaf of bread, the artificial intelligence model 106 can generate a response that identifies, e.g., includes, the image of the loaf of bread, and a corresponding image label for inclusion in the table row, e.g. which row can be automatically generated.

[0136] The device can be any appropriate device. The device that selects the at least one of multiple artificial intelligence models to process the request can be the developer device 102, a device or combination of devices in the remote system 114, or a combination of both.

[0137] The remote system 114 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described in this specification are implemented. The developer devices 102 can include personal computers, mobile communication devices, and other devices that can send and receive data over the network 118. The network 118, such as a local area network (“LAN”), wide area network (“WAN”), the Internet, or a combination thereof, connects the developer device 102, the remote system 114, and the end-user devices 116. The input device can be any appropriate type of input device, such as a mouse, a keyboard, a microphone, a touch screen, or another appropriate type of input device. The remote system 114, the developer device 102, or a combination of both, can use a single computer or multiple computers operating in conjunction with one another, including, for example, a set of remote computers deployed as a cloud computing service.

[0138] The developer device 102 can include several different functional components, including the IDE 104, and the artificial intelligence model 106. The IDE104, the artificial intelligence model 106, or a combination of these, can include one or more data processing apparatuses, can be implemented in code, or a combination of both. For instance, each of the IDE 104 and the artificial intelligence model 106 can include one or more data processors and instructions that cause the one or more data processors to perform the operations discussed herein. In implementations in which the artificial intelligence model 106 executes, at least in part, on the remote system 114, the artificial intelligence model 106 can be similarly implemented as it would have had the artificial intelligence model 106 been implemented on the developer device 102.

[0139] The various functional components of the remote system 114 can be installed on one or more computers as separate functional components or as different modules of a same functional component. For example, the artificial intelligence model 106 can be implemented as computer programs installed on one or more computers in one or more locations that are coupled to each through a network. In cloud-based systems for example, these components can be implemented by individual computing nodes of a distributed computing system.

[0140] FIG. 3 is a flow diagram of an example process 300 for presenting generative content. For example, the process 300 can be used by the IDE 104, e.g., executing on the developer device 102, from the environment 100.

[0141] An IDE provides, to an artificial intelligence model, a request with a prompt that includes contextual information from the software development environment (302). The provision can be provision of the request to an artificial intelligence model or multiple artificial intelligence models. As described in more detail elsewhere in this specification, the request can be responsive to receipt of user input or a determination to present a programming recommendation without detecting user input. The user input can request a programming recommendation or be part of a general query, e.g., input in a prompt field of the IDE’s user interface. In some examples, the contextual information can include type information that indicates a type of text from the file depicted in the IDE’s view port. The text can be text before a cursor location, text depicted in the view port, text from a portion of the file that is not depicted in the view port, or a combination of these.

[0142] The contextual information can be any appropriate type of contextual information for the programming recommendation. For instance, the contextual information can include a file containing software code, a project containing the file, a summary of the project, a programming language identifier, a project entitlement, orsome combination thereof. The contextual information can be for the file presented in the IDE’s view port, for the project to which the file belongs, or a combination of both. The contextual information can include text, e.g., code, images, or a combination of both. In some examples, the text can be text from another file or group of files in the project that includes the file, e.g., a proper subset of files from the project or all files in the project. In some implementations, the text can be a summary of text from other files in the project, e.g., along with all text from the file presented at least in part in the IDE’s view port.

[0143] For instance, when the request is a request to present code, whether in a code entry portion of the IDE, e.g., inline with existing code, or in a recommendation portion of the IDE, e.g., not inline with existing code, the artificial intelligence model can use the contextual information to generate a code recommendation that has a lower likelihood of collisions than would otherwise be generated. For instance, the contextual information can include text from one or more header files for the project that indicate the variables, class definitions, function definitions, or a combination of two or more of these, for the project. By using this contextual information, the artificial intelligence model can reduce a likelihood of collisions in code generated by the artificial intelligence model compared to other systems. In some examples, by sending a subset of information from other files in the project, e.g., the header information or another type of summary for the variables, classes, functions, or a combination of these, the IDE can reduce computational resource usage, e.g., bandwidth, computational cycles, or both, compared to other systems that might send all files for the project.

[0144] The IDE receives generative content from the artificial intelligence model that is based on the contextual information (304). The IDE can receive the data from the artificial intelligence model or the multiple artificial intelligence models to which the request was provided. The generative content can be any appropriate type of generative content. For instance, the data can include recommended, e.g., suggested, code as part of the generative content. The data can include documentation, e.g., data that indicates how to write a particular function. In some examples, the data can include a recommended project setting, e.g., an instruction to cause the IDE to implement the recommended setting for the project. The setting can be for an entitlement or another appropriate project setting. The data can include an asset identifier, e.g., that indicates one or more assets for use in the project. In some instances, when the data includes an asset identifier the data can include corresponding code that references the asset, e.g., using the asset identifier.An asset identifier can indicate a location at which an asset is stored or another appropriate type of identifier that uniquely identifies the asset.

[0145] In some examples, the IDE receives data for generative content. The data can indicate whether a programming recommendation is available. For instance, as described in more detail below with respect to FIG. 4, the data can indicate that a type for the code is different than a type for which an artificial intelligence recommendation model was trained and that a programming recommendation is unavailable. In some examples, the data for the generative content, e.g., a programming recommendation, can include location information for recommended code.

[0146] The data for the programming recommendation can be any appropriate type of data. For instance, the data can indicate a line or multiple lines of code to add to a file, e.g., a currently presented file or another file. In some examples, when the currently presented software code is not necessarily stored in a file but is otherwise stored, the data can indicate a line or multiple lines of code to add to the currently presented software code. The data can represent a diff file showing recommended changes for the currently presented software code to add the recommended code to the presented software code or other software code for the project, e.g., whether part of the same code subset presented in the user interface, e.g., the same file, or another subset of code, e.g., another file.

[0147] The IDE presents a generative content user interface element that triggers a change in a presentation of the code from the generative content with software code from the file (306). For instance, the IDE can include, in the user interface, the user interface element that triggers presentation of the programming recommendation in the view port for the file. This presentation can include presentation of the recommended code with existing code from the file, e.g., when the user interface element is an “insert” interface element, such as described with reference to FIG. 2A. The presentation can be responsive to receipt of the programming recommendation.

[0148] In some examples, the IDE can present a user interface element that triggers presentation of the programming recommendation in a new file, e.g., the new file user interface element 210. In these examples, although the IDE can add the recommended code to a new file, the IDE can also add some code for the programming recommendation to the existing file that was presented in the user interface when the user interface element was presented. For instance, the user interface element can trigger creation of a new file with one or more functions, data structures, or other code that can be used in multiple files in the project. The user interface element can also triggeraddition of code to the existing file that references at least some of the code in the new file, e.g., a call to a function defined in the new file.

[0149] The IDE determines a subset of the software code from the file that is not the generative content (308). For example, the IDE determines the software code subset to which the programming recommendation corresponds. The subset can be a subset of code that was highlighted, e.g., by a cursor, when the IDE received input defining a prompt for the programming recommendation, code within a threshold distance of a cursor in the IDE, e.g., in a code entry portion of the IDE’s user interface, or other appropriate code. The code within the threshold distance of the cursor can include code before the cursor, code after the cursor, or a combination of both. In situations that include the combination of both, the IDE can use different thresholds for code before and code after the cursor.

[0150] The IDE determines a generative content location for the generative content using the subset of the software code from the file (310). The recommended location of the generative content can include a cursor location; a location after the subset of the software code; a location that encompasses the subset of software code, e.g., when at least a portion of the subset will be replaced with the recommended code; a location before the subset of software code; a location inside the subset of software code, e.g., when an argument to a function call is the recommended code; another appropriate location; or a combination of two or more of these. The IDE can determine a combination of locations when the recommended code might be inserted in two or more locations, e.g., when different portions of the recommended code will be added to, replace, or a combination of both, text from the file.

[0151] Although the examples here specifically mention software code as part of the programming recommendation, the subset, or both, other appropriate text can be included in a programming recommendation. For instance, a programming recommendation can include one or more comments or other text describing the code, e.g., both of which can have a code recommendation type. The one or more comments can be only part of, or the entirety of, the programming recommendation.

[0152] The IDE can determine the recommended location in any appropriate manner. For instance, when the artificial intelligence model executes on the developer device, the IDE can determine the recommended location using the contextual information, e.g., such as the software code subset.

[0153] In some implementations, the IDE can receive the recommended location from another component or system. When the IDE receives the programming recommendation from another system or component, the other system or component might have determined the recommended location using the subset of the software code or other appropriate contextual information.

[0154] The IDE can receive the recommended location from the artificial intelligence model. This can include the artificial intelligence model providing the IDE with an updated version of the file that includes the recommended code.

[0155] In some examples, the IDE can receive the recommended location from the remote system. The recommended location can be a coordinate location, e.g., in line and column coordinates. The recommended location can be identified by inclusion of the recommended code in an updated version of the file.

[0156] The updated file might or might not explicitly identify the recommended location. When the IDE receives an updated file, the updated file can identify the recommended location, e.g., a start location and a length of the programming recommendation. In some examples, the IDE can determine, or otherwise present, the recommended location by comparing, e.g., using “diff ’ or a similar tool, the file and the updated file to determine the differences between the two. The IDE can use a result of the comparison to determine the recommended location(s), the programming recommendation, or both.

[0157] The IDE presents the generative content location for the generative content (312). The presentation can be responsive to determining the recommended location. The presentation of the recommended location can be in a code entry portion, a recommendation portion, or a combination of both, of the IDE’s user interface. The IDE can present the recommended location by presenting a diff file that includes the programming recommendation. In the diff file, the programming recommendation can be a single contiguous block of code or multiple separate code blocks.

[0158] The IDE can present the programming recommendation substantially concurrently with other appropriate data for the file, the project, or both. For instance, the IDE can present the recommended location substantially concurrently with the recommended location. The IDE can present the programming recommendation substantially concurrently with an entitlement recommendation, e.g., that was determined using the programming recommendation; one or more setting recommendations, e.g., for the IDE; or a combination of both.

[0159] The IDE can present any appropriate amount of the programming recommendation that was generated by the artificial intelligence model. For instance, a programming recommendation can include multiple lines of code. Presentation of the programming recommendation can include presentation, by the IDE, of one or more first lines of code. The presentation of the one or more first lines of code can include presentation of a user interface element that indicates that additional lines of code can be presented upon selection of the user interface element.

[0160] The IDE can receive input indicating selection of the generative content user interface element (314). The input can be any appropriate type of input, e.g., as described elsewhere in this specification. The generative content user interface element can be an insert user interface element. Selection of the insert user interface element can cause the IDE to insert the generative content to a code entry portion of the IDE.

[0161] In some examples, in response to receiving the generative content, the IDE can determine a user interface portion of the IDE at which to present the generative content. The user interface portion can be a recommendation portion, a code entry portion, or another appropriate portion of the IDE. The IDE can determine the user interface portion using any appropriate process, data, or both. For instance, the generative content can include data that indicates the portion of the IDE in which to present the generative content. The IDE can use data about the request, a prompt for the request, or both, to determine the portion of the IDE in which to present the generative content. When the IDE determines to present the generative content in a portion other than the code entry portion, the IDE can present the insert user interface portion.

[0162] The IDE can present a remove from code user interface element (316). Once the generative content is displayed in a code entry portion of the IDE, the IDE can present the remove from code user interface element. Selection of the remove from code user interface element can cause the IDE to stop presenting the generative content in the code entry portion of the IDE. The remove from code user interface element can be a reject user interface element that rejects the generative content. In some instances, selection of the remove from code user interface element can cause presentation of at least some data for the generative content in another portion of the IDE, other than the code entry portion of the IDE.

[0163] The IDE can receive input indicating selection of a save to memory user interface element (318). The save to memory user interface element can indicate acceptance of the generative content. For example, when the IDE does not receive inputselecting the remove from code user interface element for generative content, the IDE can receive input selecting a save to memory user interface element. This save to memory user interface element can be a “confirm generative content” user interface element that indicates acceptance of the generative content, e.g., recommended code edits.

[0164] The IDE stores, in memory, data from the generative content as part of the software code for the file (320). For instance, the IDE updates a copy of the file or files in the project files with data from the programming recommendation. This can occur responsive to detection of one or more inputs, e.g., selection of the user interface element that triggers presentation of the programming recommendation in the file, or selection of another user interface element saving the changes to the file.

[0165] The order of operations in the process 300 described above is illustrative only, and presenting the programming recommendation can be performed in different orders. For example, the IDE can perform operations 306 and 312 substantially concurrently. In these examples, the process 300 can include performing operation 304, followed by at least operation 310 and optionally operation 308, and then perform operations 312 and 306, e.g., the latter two of which can be performed in any order, optionally at least partially concurrently.

[0166] In some implementations, the process 300 can include additional operations, fewer operations, or some of the operations can be divided into multiple operations. For example, the process can include operations 302 and 304. In some examples, the process can include operations 302, 304, optionally 306, and 312. In some implementations, the process can include operations 302, 304, optionally 306, 310, and 312. Any of the described embodiments can include any combination of operations 314, 316, 318, and 320, including, optionally, none of these operations.

[0167] In some instances, the generative content, e.g., the programming recommendation, can include alternative recommendations. For instance, an autocomplete programming recommendation can include a first autocomplete option and a second autocomplete option. The IDE can include one or more user interface elements each of which trigger addition of the corresponding autocomplete option to the file presented in the IDE.

[0168] In some examples, different portions of the generative content can be presented in different portions of the IDE. For instance, when the generative content includes both code and documentation, the IDE can present the code in the code entry portion of the IDE and the documentation in the recommendation portion of the IDE.

[0169] In some implementations, the process 300 receives additional input such as edits to the code in the file or another file in the project. The process 300 can receive the edits to the code before, after, or at least partially concurrently with the performance of one of the operations in the process 300. For instance, the process 300 can receive the input before operation 302, between operations 302 and 304, at least partially concurrently with operation 312 or 320, after operation 314 or 320, or any combination of these.

[0170] In some implementations, the process 300 presents generative content for a single file. The single file can be a file currently displayed in the IDE, e.g., in the IDE’s view port, or another file in the project for the file.

[0171] In some implementations, the process 300 receives generative content for multiple files. In these implementations, the artificial intelligence model can generate generative content for the multiple files when generation of content for a single file likely does not align with the organization of the project, e.g., coding practices or naming conventions of the project. This can include receiving generative content for two or more files in the project, such as a proper subset of files in the project.

[0172] In some examples, the process 300 can select the files from the project to which the generative context applies, e.g., for which the generative content includes edits, using information about one or more files open in the IDE. For instance, the artificial intelligence model can use data about what symbols are available in a file currently presented in the IDE, symbols or other data for other files open in the IDE, or a combination of two or more of these, to select the files for which the generative content includes edits.

[0173] When the generative content is for multiple files in the project, the contextual information can include permissions data that indicates which files for the project can be edited for the generative content. For instance, the permissions data can indicate which files cannot be edited, e.g., are read only; which files can be edited, e.g., for which there is read and write access; or a combination of both. The permissions can be for the developer device, an account for which the IDE is being executed, e.g., an account of the developer using the IDE, or both.

[0174] In some instances, the project might include one or more third-party files. The third-party files can be files that cannot be edited. These files can include operating system files, open source files, or other types of third-party files that can be used to generate the generative content but for which the generative content cannot include edits.

[0175] The software application for which the IDE receives the generative content can be any appropriate type of software application. For instance, the software application can be a home automation software application that controls operation of one or more devices at a property, such as a smart speaker, a television, a sensor such as a camera, a door lock, a light, an air conditioner, an air purifier, a doorbell, a (de)humidifier, a fan, a garage door, stereo equipment, a router, a security system, a smoke alarm, a sprinkler, a thermostat, a window, a window covering, or any combination of these.

[0176] The software application can be for an application configured to receive touch screen input to operate the application. The touch screen input can be used to control at least some of the one or more devices at the property. For instance, the application can receive sensor data from a touch screen display and, in response, change presentation of the application’s user interface.

[0177] As another example, the software application can be application that displays information regarding a business to a user and / or facilitates interactions between the user and the business. For instance, the software application can be an application that displays information regarding a restaurant (e.g., the restaurant’s menu, location, contact information, operating hours, photos, videos, etc.) and allows a user to place food orders with the restaurant. Or, the software application can be an application that displays information regarding a store (e.g., the store’s location, contact information, operating hours, items or services for sale, photos, videos, etc.) and allows a user to place orders with the store.

[0178] As another example, the software application can be an application that allows a user to create, edit, and / or view content, such as textual content, images, videos, audio, or a combination thereof. As further examples, the software application can be a game application (e.g., for playing a video game), a news application (e.g., for browsing news content, such as news articles or videos), and / or a financial application (e.g., for managing personal finances, such as banking and investments).

[0179] Although example applications are described herein, these are merely illustrative examples. In practice, the software application can be any type of software application that provides any type of functionality on a device.

[0180] For implementations in which the application receives mouse or keyboard input, the application can perform corresponding operations in response to receipt of input. For example, upon receiving mouse input, keyboard input, or a combination of both, the application can change a presentation of the application’s user interface.

[0181] FIG. 4 is a flow diagram of an example process 400 for using a type of software code. For example, the process 400 can be used by the IDE 104, e.g., executing on the developer device 102, from the environment 100.

[0182] An IDE detects, using at least some text from software code, a type for the software code (402). The type can be any appropriate type such as programming language, or a human language. Some examples of programming languages include swift, C, C++, objective C, and Java. Some examples of human languages include English, Spanish, Japanese, Arabic, and Greek.

[0183] The IDE determines whether at least some of the software code is of a different type than a type for which an artificial intelligence model included in the software development environment was trained (404). For instance, the IDE determines whether the artificial intelligence model, e.g., a recommendation model, was trained for swift and objective-C. The IDE can make this determination by accessing settings data for the recommendation model. The IDE determines whether the types for the software code, e.g., presented in the IDE, is of a different type.

[0184] The recommendation model can be any appropriate type of recommendation model. For instance, the recommendation model can be the artificial intelligence model described throughout this specification.

[0185] The IDE presents a message that indicates that the at least some of the software code is of the different type than the type for which the artificial intelligence model was trained (406). For example, in response to determining that the software code’s type is different than the type for the artificial intelligence model, the IDE can present the message.

[0186] The message can be any appropriate type of message. For instance, the message can be visual, audible, or a combination of both.

[0187] Operation 406 can be part of the process 300, e.g., part of operation 304. In these examples, the IDE can receive the data for the generative content, e.g., programming recommendation, that indicates that a generative content is unavailable. The IDE can receive this data from a component in IDE, another component executing on the developer device, a component on the remote system, or a combination of these.

[0188] The IDE determines to skip providing a prompt for the software code to the artificial intelligence model (408). For instance, the IDE can determine to skip providing the prompt in response to determining that the software code’s type is different than the type for the artificial intelligence model.

[0189] In some implementations, the IDE can determine to skip providing the prompt without presenting the message. This can occur when the IDE might otherwise provide a programming recommendation without receiving input as a prompt. For instance, the IDE can detect that a cursor has not moved for a threshold time period and determine to skip providing the prompt in response to determining that the software code’s type is different than the type for the artificial intelligence model.

[0190] The IDE provides, to the artificial intelligence model, a request for generative content (410). For instance, in response to determining that the software code’s type is not different, e.g., is the same as, the type for which the artificial intelligence model was trained, the IDE can provide the request. The IDE can perform operation 410 as part of the process 300, e.g., as part of operation 302.

[0191] The order of operations in the process 400 described above is illustrative only, and using the type of software code can be performed in different orders. For example, the IDE can perform operation 408 before or substantially concurrently with, e.g., instead of after, operation 406.

[0192] In some implementations, the process 400 can include additional operations, fewer operations, or some of the operations can be divided into multiple operations. For example, the process 400 can include, as an initial operation, detecting input requesting a programming recommendation. The IDE can detect this input using a prompt field or another appropriate component of the IDE’s user interface. In some examples, the process 400 can include operations 404 and 406; operations 404 and 408; or operations 404 and 410. Any of these implementations can optionally include operation 402.

[0193] In some implementations, the IDE can determine the contextual information using data for a cursor presented in the IDE. For instance, the IDE can determine a subset of software code presented in the IDE, e.g., at least in part in a view port, using the cursor’s location. The code subset of the software code can be one or more of code highlighted in the user interface, whether depicted in the view port or not; a prior code subset before a cursor location in the user interface; or a subsequent code subset after the cursor location in the user interface.

[0194] In some examples, the code subset might not be from the file presented in the view port, e.g., in the IDE’s user interface. For instance, the code subset can include code from another file i) than a file that includes the software code presented in the user interface, and ii) that is in a project that includes the file. In these examples, providing therequest can include providing at least some code from the other file to cause the artificial intelligence recommendation model to determine whether to generate the programming recommendation that includes data from the other file.

[0195] In some implementations, the IDE can provide, as part of the contextual information, code version data, application specific data, or a combination of both. The code version data can be for the project, e.g., for one or more files in the project, for the IDE, for the programming language, or include data for a combination of these. The code version data for a file in the project can be for the file currently presented in the IDE’s user interface, e.g., view port.

[0196] In some implementations, the IDE can provide, as part of the contextual information, settings data. The settings data can identify a target platform for the project that includes the file depicted in the IDE’s user interface. The settings data can identify one or more settings for the IDE.

[0197] The generative content, e.g., programming recommendation, can be any appropriate type of generative content. For instance, the generative content can include edits to code in one or more files for a project. The edits can include recommended code additions, recommended code deletions, or a combination of both.

[0198] In implementations in which the IDE presents generative content that includes content for multiple programming recommendations, the artificial intelligence model can be trained using data indicating which of the multiple programming recommendations was accepted. A training system can train the artificial intelligence model in implementations in which the training system receives data indicating an “opt in” for allowing training given an accepted programming recommendation. This can occur while the training system does not use any code specific to the project as part of the training process.

[0199] As described herein, content is automatically generated by one or more computers in response to a request to generate the content. The automatically-generated content is optionally generated on-device (e.g., generated at least in part by a computer system at which a request to generate the content is received) and / or generated off-device (e.g., generated at least in part by one or more nearby computers that are available via a local network or one or more computers that are available via the internet). This automatically-generated content optionally includes visual content (e.g., images, graphics, and / or video), audio content, and / or text content.

[0200] In some embodiments, novel automatically-generated content that is generated via one or more artificial intelligence (“Al”) processes is referred to as generative content (e.g., generative images, generative graphics, generative video, generative audio, and / or generative text). Generative content is typically generated by an Al process based on a prompt that is provided to the Al process. An Al process typically uses one or more Al models to generate an output based on an input. An Al process optionally includes one or more pre-processing steps to adjust the input before it is used by the Al model to generate an output (e.g., adjustment to a user-provided prompt, creation of a system-generated prompt, and / or Al model selection). An Al process optionally includes one or more post-processing steps to adjust the output by the Al model (e.g., passing Al model output to a different Al model, upscaling, downscaling, cropping, formatting, and / or adding or removing metadata) before the output of the Al model used for other purposes such as being provided to a different software process for further processing or being presented (e.g., visually or audibly), e.g., to a person.

[0201] A prompt for generating generative content can include one or more of one or more words (e.g., a natural language prompt that is written or spoken), one or more images, one or more drawings, and / or one or more videos. Al processes can include machine learning models including neural networks. Neural networks can include transformer-based deep neural networks such as large language models (“LLMs”). Generative pre-trained transformer models are a type of LLM that can be effective at generating novel generative content based on a prompt. Some Al processes use a prompt that includes text to generate either different generative text, generative audio content, and / or generative visual content. Some Al processes use a prompt that includes visual content and / or an audio content to generate generative text (e.g., a transcription of audio and / or a description of the visual content). Some multi-modal Al processes use a prompt that includes multiple types of content (e.g., text, images, audio, video, and / or other sensor data) to generate generative content. A prompt sometimes also includes values for one or more parameters indicating an importance of various parts of the prompt. Some prompts include a structured set of instructions that can be understood by an Al process that include phrasing, a specified style, relevant context (e.g., starting point content and / or one or more examples), and / or a role for the Al process.

[0202] Generative content is generally based on the prompt but is not deterministically selected from pre-generated content and is, instead, generated using the prompt as a starting point. In some embodiments, pre-existing content (e.g., audio, text,and / or visual content) is used as part of the prompt for creating generative content (e.g., the pre-existing content is used as a starting point for creating the generative content). For example, a prompt could request that a block of text be summarized or rewritten in a different tone, and the output would be generative text that is summarized or written in the different tone. Similarly a prompt could request that visual content be modified to include or exclude content specified by a prompt (e.g., removing an identified feature in the visual content, adding a feature to the visual content that is described in a prompt, changing a visual style of the visual content, and / or creating additional visual elements outside of a spatial or temporal boundary of the visual content that are based on the visual content). In some embodiments, a random or pseudo-random seed is used as part of the prompt for creating generative content (e.g., the random or pseud-random seed content is used as a starting point for creating the generative content). For example, when generating an image from a diffusion model, a random noise pattern is iteratively denoised based on the prompt to generate an image that is based on the prompt. While specific types of Al processes have been described herein, it should be understood that a variety of different Al processes could be used to generate generative content based on a prompt.

[0203] Some embodiments described herein can include use of artificial intelligence and / or machine learning (“ML”) systems (sometimes referred to herein as the AI / ML systems). The use can include collecting, processing, labeling, organizing, analyzing, recommending and / or generating data. Entities that collect, share, and / or otherwise utilize user data should provide transparency and / or obtain user consent when collecting such data. The present disclosure recognizes that the use of the data in the AI / ML systems can be used to benefit users. For example, the data can be used to train models that can be deployed to improve performance, accuracy, and / or functionality of applications and / or services. Accordingly, the use of the data enables the AI / ML systems to adapt and / or optimize operations to provide more personalized, efficient, and / or enhanced user experiences. Such adaptation and / or optimization can include tailoring content, recommendations, and / or interactions to individual users, as well as streamlining processes, and / or enabling more intuitive interfaces. Further beneficial uses of the data in the AI / ML systems are also contemplated by the present disclosure.

[0204] The present disclosure contemplates that, in some embodiments, data used by AI / ML systems includes publicly available data. To protect user privacy, data may be anonymized, aggregated, and / or otherwise processed to remove or to the degree possiblelimit any individual identification. As discussed herein, entities that collect, share, and / or otherwise utilize such data should obtain user consent prior to and / or provide transparency when collecting such data. Furthermore, the present disclosure contemplates that the entities responsible for the use of data, including, but not limited to, data used in association with AI / ML systems, should attempt to comply with well-established privacy policies and / or privacy practices.

[0205] For example, such entities may implement and consistently follow policies and practices recognized as meeting or exceeding industry standards and regulatory requirements for developing and / or training AI / ML systems. In doing so, attempts should be made to ensure all intellectual property rights and privacy considerations are maintained. Training should include practices safeguarding training data, such as personal information, through sufficient protections against misuse or exploitation. Such policies and practices should cover all stages of the AI / ML systems development, training, and use, including data collection, data preparation, model training, model evaluation, model deployment, and ongoing monitoring and maintenance. Transparency and accountability should be maintained throughout. Such policies should be easily accessible by users and should be updated as the collection and / or use of data changes. User data should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection and sharing should occur through transparency with users and / or after receiving the informed consent of the users. Additionally, such entities should consider taking any needed steps for safeguarding and securing access to such data and ensuring that others with access to the data adhere to their privacy policies and procedures. Such entities should subject themselves to evaluation by third parties to certify, as appropriate for transparency purposes, their adherence to widely accepted privacy policies and practices. In addition, policies and / or practices should be adapted to the particular type of data being collected and / or accessed and tailored to a specific use case and applicable laws and standards, including jurisdiction-specific considerations.

[0206] In some embodiments, AI / ML systems may utilize models that may be trained (e.g., supervised learning or unsupervised learning) using various training data, including data collected using a user device. Such use of user-collected data may be limited to operations on the user device. For example, the training of the model can be done locally on the user device so no part of the data is sent to another device. In some implementations, the training of the model can be performed using one or more otherdevices (e.g., server(s)) in addition to the user device but done in a privacy preserving manner, e.g., via multi-party computation as may be done cryptographically by secret sharing data or other means so that the user data is not leaked to the other devices.

[0207] In some embodiments, the trained model can be centrally stored on the user device or stored on multiple devices, e.g., as in federated learning. Such decentralized storage can similarly be done in a privacy preserving manner, e.g., via cryptographic operations where each piece of data is broken into shards such that no device alone (i.e., only collectively with another device(s)) or only the user device can reassemble or use the data. In this manner, a pattern of behavior of the user or the device may not be leaked, while taking advantage of increased computational resources of the other devices to train and execute the ML model. Accordingly, user-collected data can be protected. In some implementations, data from multiple devices can be combined in a privacy-preserving manner to train an ML model.

[0208] In some embodiments, the present disclosure contemplates that data used for AI / ML systems may be kept strictly separated from platforms where the AI / ML systems are deployed and / or used to interact with users and / or process data. In such embodiments, data used for offline training of the AI / ML systems may be maintained in secured datastores with restricted access and / or not be retained beyond the duration necessary for training purposes. In some embodiments, the AI / ML systems may utilize a local memory cache to store data temporarily during a user session. The local memory cache may be used to improve performance of the AI / ML systems. However, to protect user privacy, data stored in the local memory cache may be erased after the user session is completed. Any temporary caches of data used for online learning or inference may be promptly erased after processing. All data collection, transfer, and / or storage should use industry-standard encryption and / or secure communication.

[0209] In some embodiments, as noted above, techniques such as federated learning, differential privacy, secure hardware components, homomorphic encryption, and / or multi-party computation among other techniques may be utilized to further protect personal information data during training and / or use of the AI / ML systems. The AI / ML systems should be monitored for changes in underlying data distribution such as concept drift or data skew that can degrade performance of the AI / ML systems over time.

[0210] In some embodiments, the AI / ML systems are trained using a combination of offline and online training. Offline training can use curated datasets to establish baseline model performance, while online training can allow the AI / ML systems tocontinually adapt and / or improve. The present disclosure recognizes the importance of maintaining strict data governance practices throughout this process to ensure user privacy is protected.

[0211] In some embodiments, the AI / ML systems may be designed with safeguards to maintain adherence to originally intended purposes, even as the AI / ML systems adapt based on new data. Any significant changes in data collection and / or applications of an AI / ML system use may (and in some cases should) be transparently communicated to affected stakeholders and / or include obtaining user consent with respect to changes in how user data is collected and / or utilized.

[0212] Despite the foregoing, the present disclosure also contemplates embodiments in which users selectively restrict and / or block the use of and / or access to data. That is, the present disclosure contemplates that hardware and / or software elements can be provided to prevent or block access to data. For example, in the case of some services, the present technology should be configured to allow users to select to “opt in” or “opt out” of participation in the collection of data during registration for services or anytime thereafter. In some examples, the present technology should be configured to allow users to select not to provide certain data for training the AI / ML systems and / or for use as input during the inference stage of such systems. In some examples, the present technology should be configured to allow users to be able to select to limit the length of time data is maintained or entirely prohibit the use of their data for use by the AI / ML systems. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user can be notified when their data is being input into the AI / ML systems for training or inference purposes, and / or reminded when the AI / ML systems generate outputs or make decisions based on their data.

[0213] The present disclosure recognizes AI / ML systems should incorporate explicit restrictions and / or oversight to mitigate against risks that may be present even when such systems having been designed, developed, and / or operated according to industry best practices and standards. For example, outputs may be produced that could be considered erroneous, harmful, offensive, and / or biased; such outputs may not necessarily reflect the opinions or positions of the entities developing or deploying these systems. Furthermore, in some cases, references to third-party products and / or services in the outputs should not be construed as endorsements or affiliations by the entities providing the AI / ML systems. Generated content can be filtered for potentiallyinappropriate or dangerous material prior to being presented to users, while human oversight and / or ability to override or correct erroneous or undesirable outputs can be maintained as a failsafe.

[0214] The present disclosure further contemplates that users of the AI / ML systems should refrain from using the services in any manner that infringes upon, misappropriates, or violates the rights of any party. Furthermore, the AI / ML systems should not be used for any unlawful or illegal activity, nor to develop any application or use case that would commit or facilitate the commission of a crime, or other tortious, unlawful, or illegal act. The AI / ML systems should not violate, misappropriate, or infringe any copyrights, trademarks, rights of privacy and publicity, trade secrets, patents, or other proprietary or legal rights of any party, and appropriately attribute content as required. Further, the AI / ML systems should not interfere with any security, digital signing, digital rights management, content protection, verification, or authentication mechanisms. The AI / ML systems should not misrepresent machine-generated outputs as being human-generated.

[0215] It is well understood that the use of personally identifiable information should follow privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. In particular, personally identifiable information data should be managed and handled so as to minimize risks of unintentional or unauthorized access or use, and the nature of authorized use should be clearly indicated to users.

[0216] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. A database can be implemented on any appropriate type of memory.

[0217] In this specification, the term “engine” or “software engine” refers to a software implemented input / output system that provides an output that is different from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Each engine can be implemented on any appropriate type of computing device or combination of computing devices, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computerreadable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.

[0218] A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above can be used, with operations re-ordered, added, or removed.

[0219] Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus. One or more computer storage media can include a machine-readable storage device, a machine- readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0220] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can be or include special purpose logic circuitry, e.g., a field programmable gate array (“FPGA”) or an application-specific integrated circuit (“ASIC”). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0221] A computer program, which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form,including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0222] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., a field programmable gate array (“FPGA”) or an application-specific integrated circuit (“ASIC”).

[0223] Computers suitable for the execution of a computer program include, by way of example, general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. A computer can be embedded in another device, e.g., a mobile telephone, a smart phone, a headset, a personal digital assistant (“PDA”), a mobile audio or video player, a game console, a Global Positioning System (“GPS”) receiver, or a portable storage device, e.g., a universal serial bus (“USB”) flash drive, to name just a few.

[0224] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0225] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a liquid crystal display (“LCD”), an organic light emitting diode (“OLED”) or other monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball or a touchscreen, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In some examples, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser.

[0226] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0227] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data, e.g., a Hypertext Markup Language (“HTML”) page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user device, which acts as a client. Data generated at the user device, e.g., a result of user interaction with the user device, can be received from the user device at the server.

[0228] FIG. 5 is a block diagram of computing devices 500, 550 that may be used to implement the systems and methods described in this specification, as either a client or as a server or plurality of servers. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers.Computing device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, smartwatches, head-worn devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and / or claimed in this specification.

[0229] Computing device 500 includes a processor 502, memory 504, a storage device 506, a high-speed interface 508 connecting to memory 504 and high-speed expansion ports 510, and a low-speed interface 512 connecting to low-speed bus 514 and storage device 506. Each of the components 502, 504, 506, 508, 510, and 512, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 502 can process instructions for execution within the computing device 500, including instructions stored in the memory 504 or on the storage device 506 to display graphical information for a GUI on an external input / output device, such as display 516 coupled to high-speed interface 508. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi -processor system).

[0230] The memory 504 stores information within the computing device 500. In one implementation, the memory 504 is a computer-readable medium. In one implementation, the memory 504 is a volatile memory unit or units. In another implementation, the memory 504 is a non-volatile memory unit or units.

[0231] The storage device 506 is capable of providing mass storage for the computing device 500. In one implementation, the storage device 506 is a computer- readable medium. In various different implementations, the storage device 506 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In one implementation, a computer program product is tangibly embodied in an information carrier. The computerprogram product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine- readable medium, such as the memory 504, the storage device 506, or memory on processor 502.

[0232] The high-speed controller 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 512 manages lower bandwidthintensive operations. Such allocation of duties is exemplary only. In one implementation, the high-speed controller 508 is coupled to memory 504, display 516 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 510, which may accept various expansion cards (not shown). In the implementation, low-speed controller 512 is coupled to storage device 506 and low-speed expansion port 514. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0233] The computing device 500 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 520, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 524. In addition, it may be implemented in a personal computer such as a laptop computer 522. Alternatively, components from computing device 500 may be combined with other components in a mobile device (not shown), such as device 550. Each of such devices may contain one or more of computing device 500, 550, and an entire system may be made up of multiple computing devices 500, 550 communicating with each other.

[0234] Computing device 550 includes a processor 552, memory 564, an input / output device such as a display 554, a communication interface 566, and a transceiver 568, among other components. The device 550 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 550, 552, 564, 554, 566, and 568, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0235] The processor 552 can process instructions for execution within the computing device 550, including instructions stored in the memory 564. The processor may also include separate analog and digital processors. The processor may provide, forexample, for coordination of the other components of the device 550, such as control of user interfaces, applications run by device 550, and wireless communication by device 550.

[0236] Processor 552 may communicate with a user through control interface 558 and display interface 556 coupled to a display 554. The display 554 may be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. The display interface 556 may comprise appropriate circuitry for driving the display 554 to present graphical and other information to a user. The control interface 558 may receive commands from a user and convert them for submission to the processor 552. In addition, an external interface 562 may be provided in communication with processor 552, so as to enable near area communication of device 550 with other devices. External interface 562 may provide, for example, for wired communication (e.g., via a docking procedure) or for wireless communication (e.g., via Bluetooth or other such technologies).

[0237] The memory 564 stores information within the computing device 550. In one implementation, the memory 564 is a computer-readable medium. In one implementation, the memory 564 is a volatile memory unit or units. In another implementation, the memory 564 is a non-volatile memory unit or units. Expansion memory 574 may also be provided and connected to device 550 through expansion interface 572, which may include, for example, a SIMM card interface. Such expansion memory 574 may provide extra storage space for device 550, or may also store applications or other information for device 550. Specifically, expansion memory 574 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory 574 may be provided as a security module for device 550, and may be programmed with instructions that permit secure use of device 550. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0238] The memory may include for example, flash memory and / or MRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 564, expansion memory 574, or memory on processor 552.

[0239] Device 550 may communicate wirelessly through communication interface 566, which may include digital signal processing circuitry where necessary.Communication interface 566 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, GPRS, 4G, LTE, or 5G, among others. Such communication may occur, for example, through radio-frequency transceiver 568. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS receiver module 570 may provide additional wireless data to device 550, which may be used as appropriate by applications running on device 550.

[0240] Device 550 may also communicate audibly using audio codec 560, which may receive spoken information from a user and convert it to usable digital information. Audio codec 560 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 550. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 550.

[0241] The computing device 550 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 580. It may also be implemented as part of a smartphone 582, personal digital assistant, or other similar mobile device.

[0242] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0243] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, ProgrammableLogic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0244] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular implementations. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some instances be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0245] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0246] In each instance where an HTML file is mentioned, other file types or formats may be substituted. For instance, an HTML file may be replaced by an XML, JSON, plain text, or other types of files. Moreover, where a table or hash table is mentioned, other data structures, such as spreadsheets, relational databases, or structured files, may be used.

[0247] Particular implementations of the invention have been described. Other implementations are within the scope of the following claims. For example, the operations recited in the claims, described in the specification, or depicted in the figurescan be performed in a different order and still achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

[0248] What is claimed is:

Claims

CLAIMS1. A computer-implemented method for providing generative content in a user interface of a software development environment, the method comprising: providing, to an artificial intelligence model, a request with a prompt that includes contextual information from the software development environment comprising a file containing software code, a project containing the file, a summary of the project, a programming language identifier, a project entitlement, an asset identifier, or some combination thereof; in response to the request with the prompt, receiving generative content from the artificial intelligence model that is based on the contextual information, wherein the generative content includes code, a project setting, the asset identifier, or documentation; and presenting the generative content including the code, the project setting, the asset identifier, or the documentation, that is based on the contextual information, in the user interface of the software development environment.

2. The method of claim 1, wherein providing the request comprising providing a portion of software code highlighted in the user interface, a code subset before a cursor location in the user interface, a code subset after the cursor location in the user interface, all code from the file, all code from the project, a summary of code from the project, an asset catalog for the project, a current cursor location, a current user interface state, error data for an error presented in the user interface, or code from a second file that includes code other than code presented in the user interface.

3. The method of claim 2, wherein providing the request comprises providing code from the second file to cause the artificial intelligence model to provide generative content including code from the second file.

4. The method of claim 3, wherein the artificial intelligence model was not trained on at least a portion of the code in the second file.

5. The method of any of claims 3 or 4, wherein the second file comprises at least one function, class, structure, data type, or comment on which the artificial intelligence model was not trained.

6. The method of any preceding claim, comprising: determining, using at least one of the prompt that includes contextual information or the generative content, a recommendation type; and selecting, from a plurality of presentation types, a presentation type for the recommendation type, wherein: presenting comprises presenting at least the portion of the generative content with the presentation type.

7. The method of any preceding claim, wherein providing the request comprises providing, by the development environment executing on a developer device and to the artificial intelligence model, the request.

8. The method of claim 7, wherein providing the request comprises providing the request to the artificial intelligence model executing on a system separate from the developer device.

9. The method of any preceding claim, wherein the request for the generative content identifies one or more settings for the development environment or data that identifies a target platform for a project that includes the software code or both.

10. The method of any preceding claim, comprising: determining a cursor location of a cursor in the user interface; and selecting a code subset before the cursor location, wherein providing the contextual information comprises providing the request for a code completion that includes the code subset of the software code.

11. The method of claim 10, wherein providing the request comprises includes type information for text before the cursor location.

12. The method of any preceding claim, wherein the prompt is one of a plurality of prompts presented in a user interface, the prompt provided in the request being selected from the plurality of prompts.

13. The method of any preceding claim, wherein providing the project entitlement includes an identification of one or more data sources when the project executes on an end-user device.

14. The method of any preceding claim, wherein a response to the prompt indicates whether generative content is available.

15. The method of any preceding claim, wherein the artificial intelligence model comprises a large language model.

16. The method of any preceding claim, wherein presenting the generative content including the code, the project setting, the asset identifier, or the documentation comprises: presenting, in a first file for the project, a first portion of code from the generative content; and presenting, in a second different file for the project, a second portion of code from the generative content.

17. A computer-implemented for providing a generative content user interface element of a software development environment, the method comprising: receiving, by the software development environment that depicts at least a portion of software code from a file in a code entry region, generative content that includes code; presenting, in a user interface of the software development environment, the generative content user interface element that triggers a change in a presentation of the code from the generative content with software code from the file; determining a subset of the software code from the file that is not the generative content; determining a generative content location for the generative content using the subset of the software code from the file; and presenting, in the user interface of the software development environment, the generative content location for the code from the generative content.

18. A computer-implemented for providing an artificial intelligence model message by a software development environment, the method comprising: determining, by the software development environment executing on a device and presenting a user interface for editing software code, that at least some of the software code is of a different type than a type for which an artificial intelligence model included in the software development environment was trained; and in response to determining that at least some of the software code is of thedifferent type than the type for which the artificial intelligence model was trained, presenting, by the software development environment, the artificial intelligence model message that indicates that the at least some of the software code is of the different type than the type for which the artificial intelligence model was trained.

19. One or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any preceding claim.

20. A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any of claims 1 to 18.

Images