Generation and implementation of geospatial workflows

Abstract

This specification describes implementations for automatically generating multimodal geospatial workflows to accomplish geospatial tasks. [Solution] In various implementations, natural language requests may be processed based on a generative model such as an LLM to generate workflow output tokens that identify high-level actions to complete the geospatial task communicated in the natural language request. First data indicating high-level actions may be processed using one or more generative models to generate dataset output tokens that identify response datasets that are likely to contain data in response to the geospatial task. Second data indicating both high-level actions and response datasets may be processed based on one or more generative models to generate data manipulation output tokens that identify data manipulation instructions for assembling data from the response dataset into a response that satisfies the geospatial task.

Description

TECHNICAL FIELD

[0001] Generative models are a type of machine learning model that often take the form of and / or are described as large language models (LLMs), which can perform various tasks such as language generation, machine translation, and question answering, to name a few examples. These generative models are often trained on vast amounts of diverse data including, but not limited to, data from web pages, e-books, software code, electronic news articles, and the like. Accordingly, these generative models leverage the underlying data on which they were trained when performing these various tasks. For example, when performing a language generation task, a generative model such as an LLM can process natural language (NL)-based input received from a client device, respond to the NL-based input, and generate output in NL or another format that is to be rendered at the client device.

[0002] Geospatial datasets such as maps, high-altitude images, and geospatial databases may be used in combination with one another to perform what is referred to herein as a "geospatial workflow" to accomplish a variety of different geospatial tasks, such as responding to geospatial queries from users. Using data from two or more geospatial datasets, it is possible to determine, for example, to name a few, the number of trees in a particular geographic region, the average canopy coverage of playgrounds in regions with different climates, methane leaks of anthropogenic origin (e.g., in urban areas or areas where oil or gas has been extracted), and trends in real estate pricing in areas near large bodies of water.

SUMMARY OF THE INVENTION

[0003] Designing geospatial workflows can be challenging. Complex geospatial tasks may require correspondingly complex geospatial workflows. Experts such as geospatial data scientists or software engineers may need to dedicate considerable time and resources to designing such complex geospatial workflows. Moreover, complex geospatial workflows are not always scalable beyond their original purpose. This is especially true when the geospatial task that the workflow is intended to accomplish is narrowly defined.

[0004] This specification describes implementations for automatically generating multimodal geospatial workflows to accomplish geospatial tasks. More specifically, this specification describes, though not exclusive, implementations for processing various modalities of data using a sequence of generative models (e.g., LLMs) to identify data manipulation instructions for assembling (i) a geospatial task (e.g., from a natural language request), (ii) high-level actions required to accomplish the geospatial task, (iii) a dataset containing data to respond to the geospatial task, and (iv) data from the response dataset into response data that satisfies the geospatial task. Once the data manipulation instructions are assembled, they can be executed to generate the response data, which can be rendered on one or more output devices.

[0005] In various implementations, the method may be performed using one or more processors and may include processing a natural language request based on one or more generative models, such as one or more single-modal or multimodal large-scale language models (LLMs), to generate a workflow output token that identifies a high-level action to complete a geospatial task communicated in the natural language request; processing first data indicating a high-level action to complete a geospatial task based on one or more generative models to generate a dataset output token that identifies one or more response datasets that are likely to contain data responding to a geospatial task communicated in the natural language request; processing second data indicating both a high-level action to complete a geospatial task and one or more response datasets based on one or more generative models to generate a data manipulation output token that identifies a data manipulation instruction to assemble the data from one or more response datasets into a response that satisfies the geospatial task; executing the data manipulation instruction using one or more response datasets to generate a response that satisfies the geospatial task; and rendering the response that satisfies the geospatial task on one or more computing devices.

[0006] In various implementations, natural language requests may be processed based on a different generative model than the first data. In various implementations, natural language requests may be processed based on a different generative model than the second data. In various implementations, the first data is processed based on a different generative model than the second data.

[0007] In various implementations, the first data includes metadata about multiple candidate datasets that identify one or more response datasets. In various implementations, the metadata may include human-curated content describing one or more of the candidate datasets. In various implementations, the metadata may include data indicating the database schema of one or more of the candidate datasets.

[0008] In various implementations, the dataset output token may represent a measure of the usefulness of each of the multiple candidate datasets, each of which identifies one or more response datasets, in performing high-level actions to complete the geospatial task communicated in the natural language request.

[0009] In various implementations, one or more response datasets may include a first overhead digital depiction of a geographic area, which includes annotations identifying land-based features of the geographic area. In various implementations, the first overhead digital depiction may include raster images. In various implementations, raster images may be captured by satellites or unmanned aerial vehicles (UAVs).

[0010] In various implementations, the first overhead digital representation may include a vector-based map of a geographical area. In various implementations, one or more response datasets may include a second overhead digital representation of at least a portion of a geographical area. In various implementations, data manipulation instructions may include instructions for combining the first overhead digital representation and the second overhead digital representation. In various implementations, data manipulation instructions may include instructions for overlaying one of the first or second overhead digital representations onto the other.

[0011] In various implementations, data manipulation instructions may include source code written in a high-level programming language. In various implementations, the source code may be configured to retrieve data from one or more response datasets and assemble responses to natural language requests.

[0012] In various implementations, one or more response datasets may include at least a first response database and a second response database. In various implementations, data manipulation instructions may include instructions for joining data from the first response database and the second response database. In various implementations, instructions for joining data from the first response database and the second response database may include structured query language (SQL) instructions. In various implementations, instructions for joining data from the first response database and the second response database may include source code written in a high-level programming language.

[0013] In various implementations, the method may further include evaluating a data manipulation instruction based on one or more criteria, constructing an input prompt for one or more generative models, the input prompt comprising at least some of the data manipulation instructions and additional information regarding the decision that the data manipulation instruction cannot satisfy one or more criteria, and processing the input prompt using one or more generative models to generate a new data manipulation instruction. In various implementations, the data manipulation instruction may include source code in a high-level programming language, and one or more criteria may include the ability to compile the source code without errors.

[0014] In addition, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, where one or more processors are operable to execute instructions stored in associated memory, and the instructions are configured to perform one of the methods described above. Some implementations also include one or more non-temporary computer-readable storage media that store computer instructions that can be executed by one or more processors to perform one of the methods described above.

[0015] It should be understood that all combinations of the aforementioned and additional concepts described in detail herein are intended to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter disclosed herein. [Brief explanation of the drawing]

[0016] [Figure 1] This document demonstrates various aspects of the present disclosure and depicts block diagrams of exemplary environments in which some of the implementations disclosed herein can be implemented. [Figure 2] This paper outlines one example of how various components can exchange data to facilitate multimodal assistant engagement across different implementation forms. [Figure 3A] This section outlines exemplary scenarios in which selected aspects of this disclosure may be implemented. [Figure 3B] This section outlines exemplary scenarios in which selected aspects of this disclosure may be implemented. [Figure 4] A flowchart illustrates exemplary ways of implementing a selected aspect of this disclosure in various implementation forms. [Figure 5]This describes exemplary architectures of computing devices in various implementation forms. [Modes for carrying out the invention]

[0017] Designing geospatial workflows can be challenging. Complex geospatial tasks may require correspondingly complex geospatial workflows. Experts such as geospatial data scientists or software engineers may need to dedicate considerable time and resources to designing such complex geospatial workflows. Moreover, complex geospatial workflows are not always scalable beyond their original purpose. This is especially true when the geospatial task that the workflow is intended to accomplish is narrowly defined.

[0018] This specification describes implementations for automatically generating multimodal geospatial workflows to accomplish geospatial tasks. More specifically, this specification describes, though not exclusive, implementations for processing various modalities of data using a sequence of generative models (e.g., LLMs) to identify data manipulation instructions for assembling (i) a geospatial task (e.g., from a natural language request), (ii) high-level actions required to accomplish the geospatial task, (iii) a dataset containing data to respond to the geospatial task, and (iv) data from the response dataset into response data that satisfies the geospatial task. Once the data manipulation instructions are assembled, they can be executed to generate the response data, which can be rendered on one or more output devices.

[0019] In some implementations, based on the response data, a user may issue a follow-up request, for example, as part of a human-to-computer dialog with a chatbot or virtual assistant configured in selected aspects of the present disclosure. Data from one dialog turn may be reused during subsequent dialog turns such that the user can iteratively and / or incrementally perform and / or refine geospatial tasks, for example using trial and error, to converge on a desired solution.

[0020] In various implementations, a user desiring to accomplish a geospatial task may speak or type a natural language request that conveys the geospatial task. This natural language request may be processed as a sequence of tokens using a generative model such as, for example, a "workflow LLM" to generate "workflow output tokens". In some implementations, the workflow LLM may be trained to generate output tokens that directly or indirectly convey, correspond to, or otherwise identify high-level actions for completing the geospatial task corresponding to the geospatial query, based on the natural language request.

[0021] Assume that a user utters or types the natural language query "I want to know the number of trees in Cherokee Park in Louisville, Kentucky". When processed using the workflow LLM, the workflow LLM may generate workflow output tokens that convey one or more workflows, and where multiple workflows are generated, each may be considered a "candidate workflow". Each workflow, when implemented, may include a series of high-level actions that cause response data satisfying the user's request to be generated. The working example related to trees in Cherokee Park may generate the following candidate workflows as alternatives to each other.

[0022] [Table 1]

[0023] In some implementations, as the geospatial workflow generation process described herein proceeds, these candidate workflows may, for example, be evaluated against one another, and the most "preferred" or best candidate workflow may be selected for the implementation.

[0024] In some implementations, the workflow LLM may also be conditioned to adapt the actions of the candidate workflow to the user's context. The user's context may include, for example, metadata about the user's available datasets and / or the application programming interface (API) ecosystem. This context data may be used, for example, together with the natural language request, to condition the workflow LLM. As a result, the workflow LLM may generate output tokens representing the proposed workflow and semantically and contextually appropriate constituent workflow actions.

[0025] In some implementations, once a workflow is identified from the output tokens, data indicative of the workflow, such as the workflow output tokens of the workflow LLM, embeddings generated therefrom, and the like, may be processed using the same workflow LLM or a different generative model. In the latter case, the different generative model may, in some examples, be a "dataset LLM" (or more generally, a "dataset generation model") trained to identify suitable datasets for accomplishing geospatial tasks. In some such implementations, data indicative of the workflow and data indicative of candidate datasets (e.g., dataset metadata) may be used to construct a prompt for the dataset LLM. The dataset LLM may then generate "dataset output tokens" that identify one or more responsive datasets that are likely to contain data responsive to and / or relevant to the geospatial task.

[0026] In the "Candidate Workflow B" described above, various datasets such as web pages, general image repositories, databases, maps, and high-altitude image repositories may be evaluated to determine whether they respond to performing a geospatial task or potentially contain data that is otherwise relevant. In some implementations, metadata about these datasets may be used to condition the dataset LLM along with workflow actions, for example, to assign a score to each dataset or simply classify the datasets as suitable or unsuitable. This metadata may include, for example, a human-written description of the dataset, a schema indicating that the dataset is structured data, and dimensionality reduction embeddings representing the semantics of the dataset. The dataset output token generated based on the dataset LLM may identify satellite and / or aerial imagery as likely to contain data usable for counting trees in Cherokee Park.

[0027] In some implementations, generative models other than LLMs may be used to implement various aspects of this disclosure. For example, a dataset generative model may take the form of a vision language model (VLM) similar to an LLM, except that it can accept input modalities other than text, such as images or image-based embeddings. Such a dataset VLM may be used to query images for content that is potentially responsive to or relevant to a geospatial task. For example, if a workflow seeks information about a river, a VLM may be used to identify satellite and / or aerial imagery that depicts the river and exclude satellite and / or aerial imagery that does not depict the river as a response dataset.

[0028] A single response dataset is selected, and in some implementations, one or more LLMs may be prompted with data that identifies the response dataset, such as metadata, tokens indicating dataset metadata, or dates within the dataset. For example, a “data manipulation LLM” (or more generally, a “data manipulation generative model”) may be prompted with dataset tokens corresponding to satellite and / or aerial imagery, combined with tokens representing workflow actions. A “data manipulation token” generated by the data manipulation LLM based on this prompt may directly or indirectly identify a “data manipulation instruction” to assemble (e.g., combine) data from one or more response datasets into a response that satisfies a natural language request. In the Cherokee Park example, a data manipulation instruction might detect trees in satellite and / or aerial imagery by having computer vision techniques, for example, add bounding boxes or other annotations to detected trees and count the annotations.

[0029] Data manipulation instructions can take various forms. In some implementations, data manipulation instructions may take the form of source code written in high-level programming languages ​​such as C, C++, Python, Java, and Perl. In other implementations, data manipulation instructions may be written using various types of syntactically constrained pseudocode (which can be translated into a high-level programming language for downstream execution). Regardless of the language used, the source code may be configured to be executed to retrieve data from one or more response datasets and assemble a response that satisfies a geospatial task. In other implementations, data manipulation instructions may be written in a domain-specific language (DSL) executable in a particular domain, such as a software application that has a built-in scripting language or can receive mathematical commands that accept data from response datasets as parameters. If the response datasets include a database with a defined schema, data manipulation instructions may include, for example, Structured Query Language (SQL) commands.

[0030] Data manipulation instructions can then be performed (e.g., executed) using the response dataset to generate a response that satisfies a geospatial task. Responses can take various forms depending on factors such as the modality from which the request was made, the requested modality, available output devices, and / or applications. In the Cherokee Park example, the response could be a natural language response such as, "According to the satellite imagery available to me, Cherokee Park has approximately 235,000 trees." Other geospatial tasks may request other types of responses, such as charts, statistics, reports, heatmaps, or vector visualizations on a map (e.g., visual annotations).

[0031] Referring now to Figure 1, a block diagram of an exemplary environment that can demonstrate various aspects of this disclosure and implement the implementations disclosed herein is depicted. The exemplary environment includes a client device 110 and a geospatial knowledge system 120, which is separately depicted in Figure 1. In some implementations, all or aspects of the geospatial knowledge system 120 can be implemented locally on the client device 110. In additional or alternative implementations, all or aspects of the geospatial knowledge system 120 can be implemented remotely from the client device 110 (e.g., on a remote server) as depicted in Figure 1. In those implementations, the client device 110 and the geospatial knowledge system 120 can be coupled to communicate with each other via one or more networks 199, such as one or more wired or wireless local area networks ("LANs" including Wi-Fi, mesh networks, Bluetooth, and short-range wireless communication) or wide area networks ("WANs" including the Internet).

[0032] The client device 110 may be one or more of the following: a desktop computer, a laptop computer, a tablet, a mobile phone, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (optionally having a display), a smart appliance such as a smart TV, and / or a user's wearable device including a computing device (e.g., a user's wristwatch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0033] The client device 110 can run one or more software applications via the application engine 115, submit inputs through those software applications, and / or render outputs in response to those inputs (e.g., audibly and / or visually). The application engine 115 can run one or more software applications separate from the operating system of the client device 110 (e.g., installed "on top of" the operating system), or alternatively, directly implemented by the operating system of the client device 110. For example, the application engine 115 can run a web browser or an automated assistant installed on top of the operating system of the client device 110. In another embodiment, the application engine 115 can run a web browser software application or an automated assistant software application integrated as part of the operating system of the client device 110. The application engine 115 (and one or more software applications run by the application engine 115) can interact with the geospatial knowledge system 120.

[0034] For example, in some implementations, a mapping application (which may be a standalone application or may run in a web browser) may include a chat interface that allows a user to engage in a human-to-computer dialogue with an automated assistant. In various implementations, the automated assistant provided through such a mapping application may be tailored to the mapping application or may be a general-purpose automated assistant capable of operating within the domain of the mapping application (e.g., via the mapping application's API). During such a human-to-computer dialogue, the user may provide inputs that enable the automated assistant to adapt various aspects of this disclosure to automatically create and execute a geospatial workflow. Such a human-to-computer dialogue may, in some examples, include multiple dialogue turns. For example, a first geospatial workflow generated in response to a first NL input may not function properly (e.g., it may crash or generate incorrect information). In such cases, the user may be prompted to provide additional information and / or feedback that enables additional attempts to properly construct the geospatial workflow during subsequent turns of the human-to-computer dialogue.

[0035] In various implementations, the client device 110 may include a user input engine 111 configured to detect user input provided by a user of the client device 110 using one or more user interface input devices. For example, the client device 110 may include one or more microphones that capture audio data, such as audio data corresponding to the user's oral utterances or other sounds in the environment of the client device 110. Additionally or alternatively, the client device 110 may include one or more vision components configured to capture vision data corresponding to digital images and / or motion (e.g., gestures) detected within the field of view of one or more of the vision components. Additionally or alternatively, the client device 110 may include one or more touch-sensitive components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons) configured to capture signals corresponding to touch input directed towards the client device 110.

[0036] Some examples of inputs described herein may be queries for geospatial workflows, provided by a user of client device 110 and formulated based on user input detected via user input engine 111. For example, queries may be typed queries, typed via a physical or virtual keyboard; proposed queries, selected via the touchscreen or mouse of client device 110; spoken voice queries, detected via the microphone of client device 110 (and optionally directed to an automated assistant running at least partially on client device 110); image or video queries, based on vision data captured by the vision components of client device 110 (or based on NL input generated based on processing images using, for example, an object detection model, a caption model, etc.); or any combination thereof. Other examples of NL-based inputs described herein may be prompts for geospatial content, provided by a user of client device 110 and formulated based on user input detected via user input engine 111. For example, a prompt can be a typed prompt, selected via a physical or virtual keyboard; a suggested prompt, selected via the touchscreen or mouse of the client device 110; a spoken prompt, detected via the microphone of the client device 110; or an image prompt, based on an image captured by the vision components of the client device 110.

[0037] In various implementations, the client device 110 may include a rendering engine 112 configured to render response content (e.g., NL-based output, geospatial workflow output, source instructions associated with the geospatial workflow output (e.g., datasets), and / or other content) for audible and / or visual presentation to the user of the client device 110 using one or more user interface output devices. For example, the client device 110 may include one or more speakers that enable the provision of response content for audible presentation to the user via the client device 110. Additionally or alternatively, the client device 110 may include a display or projector that enables the provision of response content for visual presentation to the user via the client device 110.

[0038] In various implementations, the client device 110 may include a context engine 113 configured to determine the context of the client device 110 (e.g., the current context or recent context) and / or the context of the user of the client device 110 (e.g., the active user of the client device 110 when the client device 110 is associated with multiple users). In some of these implementations, the context engine 113 may determine the context based on data stored in the client device data database 110A. The data stored in the client device data database 110A may include, for example, user interaction data characterizing the current or recent interactions of the client device 110 and / or the user of the client device 110; location data characterizing the current or recent locations of the client device 110 and / or the user of the client device 110; user attribute data characterizing one or more attributes of the user of the client device 110; user preference data characterizing one or more preferences of the user of the client device 110; user profile data characterizing the profile of the user of the client device 110; and / or any other data accessible to the context engine 113 via the client device data database 110A or by other means.

[0039] In some implementations, the client device database 110A may contain information such as metadata about datasets available to the client device 110 and / or the user of the client device 110. A dataset can be any structured or unstructured data source, particularly geospatial data, that can be accessed to perform geospatial workflows as described herein. In various implementations, a dataset can take the form of, for example, an API accessible to retrieve or otherwise interact with geospatial data, a structured database (e.g., accessible using SQL), vector or raster image files such as vectorized interactive maps, digital images, videos, satellite imagery, or aerial imagery, or free-form documents such as spreadsheets, articles, reports, and papers. Users may access the dataset by having it stored locally in the client device database 110A, or by having the dataset accessible from another remote computing device / system (e.g., the cloud) using data stored in the client device database 110A. In some implementations, users may be “subscribed” or otherwise permitted to access specific datasets, for example, via an API or other interface. Such subscriptions may be offered to users in a variety of scenarios, such as as part of a user’s employment or registration at a university or other research institution, or as an on-demand (e.g., paid) service through a government agency.

[0040] In various implementations, the context engine 113 can determine the current context based on the current state of the dialog session (e.g., considering one or more recent inputs provided by the user or outputs provided to the user during a human-to-computer dialog session), profile data, and / or the current location of the client device 110. For example, the context engine 113 can determine the current context of "a tree enthusiast counting the number of trees in Cherokee Park, Louisville, Kentucky" based on recently issued queries, profile data, and the expected future location of the client device 110 (e.g., based on recently booked hotel accommodations in Louisville). In another embodiment, the context engine 113 can determine the current context based on which software application is active in the foreground of the client device 110, the current or recent state of the active software application, and / or the content currently being rendered or recently rendered by the active software application. In yet another embodiment, the context engine 113 can determine the current context based on which datasets are available in or via the client device database 110A.

[0041] The context determined by the context engine 113 can be used, for example, when supplementing or rewriting NL-based inputs (or any type of input) formulated based on user input, when generating intermediate inputs (e.g., implicit queries or prompts formulated independently of any explicit NL-based inputs provided by the user of the client device 110), and / or when deciding whether to submit intermediate inputs and / or render the results of implicit NL-based inputs (e.g., geospatial workflow output). In some implementations, the context determined by the context engine 113 can be used when selecting one or more relevant or applicable datasets from datasets available in or via the client device database 110A. As described below, these datasets may be evaluated to determine which contain geospatial data that potentially responds to user queries or is otherwise relevant.

[0042] In various implementations, the client device 110 may include an implicit input engine 114 configured to generate intermediate input independently of any user-explicit NL-based input provided by the user of the client device 110. The implicit input engine 114 may also be configured to submit intermediate input independently of any user-explicit NL-based input requesting the submission of intermediate input, and / or to trigger the rendering of search results or NL-based output for the intermediate input, independently of any explicit NL-based input requesting the rendering of search results or NL-based output. For example, the implicit input engine 114 may use one or more past or current contexts obtained from the context engine 113 when generating intermediate input, when deciding where to submit the intermediate input (e.g., to the LLM engine 142), and / or when deciding to trigger the rendering of NL-based output in response to the intermediate input.

[0043] Furthermore, the client device 110 and / or the geospatial knowledge system 120 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication over one or more of the networks 199. In some implementations, one or more of the software applications may be installed locally on the client device 110, while in other implementations, one or more of the software applications may be hosted remotely (e.g., by one or more servers) and made accessible by the client device 110 over one or more of the networks 199.

[0044] The embodiments in Figure 1 are illustrated or described in relation to a single client device having a single user, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, one or more additional client devices of the user and / or additional users can also implement the technology described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user can form an ecosystem of devices that can employ the technology described herein. These additional client devices and / or computing devices can communicate with client device 110 (for example, via network 199). In another embodiment, a given client device can be used by multiple users in a shared setting (e.g., a group of users, a household, a workplace, a hotel, etc.).

[0045] The geospatial knowledge system 120 is illustrated in Figure 1 as including an input processing engine 140, a dataset processing engine 152, and a data manipulation processing engine 156. Some of these engines can be combined and / or omitted in various implementations. Furthermore, these engines can include various sub-engines. For example, the input processing engine 140 is illustrated in Figure 1 as including an LLM engine 142, a dialogue context engine 146, and an NL-based output engine 150. Similarly, some of these sub-engines can be combined and / or omitted in various implementations. Therefore, it should be understood that the various engines and sub-engines of the geospatial knowledge system 120 illustrated in Figure 1 are depicted to illustrate specific functions and are not intended to be limiting.

[0046] Furthermore, the geospatial knowledge system 120 is illustrated in Figure 1 as interfacing with various databases, including the LLM database 144, the dialogue context database 148, the dataset LLM database 154, and the data manipulation (DM) LLM database 158. While certain engines and / or sub-engines are depicted as having access to specific databases, this is illustrative and not intended to be limiting. For example, in some implementations, each of the various engines and / or sub-engines of the geospatial knowledge system 120 may access each of the various databases. Moreover, some of these databases may be combined and / or omitted in various implementations. Therefore, the various databases interfacing with the geospatial knowledge system 120 illustrated in Figure 1 are depicted to illustrate specific data accessible to the geospatial knowledge system 120 and are not intended to be limiting.

[0047] In various implementations, the geospatial knowledge system 120 can cause the LLM engine 142 to process NL-based input using generative models such as LLMs stored in the LLM database 144 to generate a stream of LLM output. The stream of LLM output may be provided by an NL-based output engine 150 (which may be configured to render the stream of LLM output on a client device 110) and / or used to prompt the same or a different LLM to generate additional information. The LLMs (or more generally, generative models) contained in any of the databases 144, 154, and 158 may include models such as PaLM, BERT, LaMDA, Meena, and / or any other generative models, such as encoder-only, decoder-only, sequence-to-sequence based, and any other generative models that optionally include attention mechanisms or other memory, diffusion models, etc. A generative model may have hundreds of millions, or even hundreds of billions, of parameters. In some implementations, the generative model may include multimodal models such as VLM and / or visual question answering (VQA) models, which may have any of the aforementioned architectures and can be used to process multiple modalities of data, particularly images and text, and / or images and audio, to generate, for example, one or more modalities of output. Some non-limiting examples of VLM that may be applied as described herein include Gemini and / or Flamingo.

[0048] The LLM output stream may include, for example, a probability distribution over a sequence of tokens, such as words, phrases, or other semantic units, that are predicted to respond to an LLM input. In particular, any generative model described herein may include billions of weights and / or parameters learned through training the generative model on a vast amount of diverse data. This allows the generative model to produce an output as a probability distribution over a sequence of tokens. In various implementations, the geospatial knowledge system 120 may cause the dialogue context engine 146 to manage dialogue contexts based on data stored in the dialogue context database 148, which may include identifying new dialogue contexts, shifting between existing dialogue contexts, and managing cascading information across multiple dialogue turns (e.g., data indicating failed or aborted geospatial workflows).

[0049] In some implementations, the LLM engine 142 may be configured to process natural language requests received by the user input engine 111 of the client device 110. These natural language requests may communicate or otherwise identify geospatial tasks. In other implementations, geospatial tasks may be received via other means, such as commands from a remote computing system. A geospatial task can be any task that can be accomplished based on geospatial data. "Geospatial data" may include, for example, raster or vector maps, satellite or aerial imagery, databases storing geospatial content, articles, books, reports, charts, and any other data with geographical components.

[0050] In a non-limiting embodiment, a user of client device 110 may engage in a human-to-computer dialogue with an automated assistant built into the mapping application. Suppose the user is running the mapping application to view a map of a specific city or neighborhood. The user may invoke the automated assistant, for example, using one or more wake words or by activating an invocation input, and provide a geospatial query such as "plot all sewer pipes." This geospatial query may trigger the generation and / or execution of a geospatial workflow that ultimately causes the underground sewer pipes within the geographic area rendered by the user's mapping application to be visually annotated or "plotted."

[0051] In some implementations, one or more LLMs in the LLM database 144 may be “workflow LLMs” trained to generate output tokens based on geospatial (e.g., NL) requests that directly or indirectly communicate, correspond to, or otherwise identify high-level actions for completing geospatial tasks corresponding to geospatial queries. One example is described above regarding the number of trees in Cherokee Park in Louisville, KY.

[0052] In another embodiment, suppose a user is viewing a map of New Hampshire and speaks or types the natural language query "Show me the artificial methane leak." When this query and applicable contextual data (e.g., the user is currently viewing a map of New Hampshire) are processed using a Workflow LLM, the Workflow LLM may generate a Workflow Output Token that communicates one or more workflows. If multiple workflows are generated, each may be considered a "candidate workflow." Each workflow may include a set of high-level actions that, if successfully executed, generate response data that satisfies the user's request. In this embodiment regarding the methane leak in New Hampshire, the following candidate workflows may be generated as alternatives to each other:

[0053] [Table 2]

[0054] To generate a geospatial workflow that provides information in response to a user query, the geospatial knowledge system 120 may find geospatial data relevant to and / or responsive to the user query. For this purpose, the geospatial knowledge system 120 may be configured, for example, to use a context engine 113 to identify datasets that (a) are available to it and (b) are potentially responsive to it.

[0055] To achieve this, in some implementations, the geospatial knowledge system 120 may cause the dataset processing engine 152 to process data indicating one or more datasets available to the client device 110 and / or the user of the client device 110, using dataset LLMs stored in the dataset LLM database 154. For example, in some implementations, the dataset processing engine 152 may be prompted with metadata indicating one or more available datasets. In some implementations, the dataset processing engine 152 may also include, as part of the same input prompt, data indicating high-level actions to complete a geospatial task communicated in a natural language request.

[0056] In some implementations, once the workflow described above is identified from the workflow output token, the workflow-indicating data, such as the workflow output token, and any embedded data generated therefrom, can be directly processed by the dataset processing engine 152 using dataset LLMs stored in the dataset LLM database 154. The dataset LLM can be trained to identify suitable datasets for accomplishing a geospatial task. In some such implementations, the workflow-indicating data and data indicating candidate datasets (e.g., dataset metadata) can be used to construct prompts for the dataset LLM. The dataset LLM can then generate a dataset output token that identifies one or more response datasets that are likely to contain data responding to a geospatial task.

[0057] In the methane leak embodiment, particularly with respect to “Candidate Workflow B,” various datasets such as web pages, general image repositories, databases, maps, and high-altitude image repositories may be evaluated to determine whether they respond to the geospatial task of identifying and depicting artificial methane leaks, or whether they may otherwise contain relevant data. In some implementations, metadata about these datasets may be used to condition the dataset LLM along with workflow actions, for example, to assign a score to each dataset, or simply classify the datasets as suitable or unsuitable. This metadata may include, for example, a human-written description of the dataset, a schema indicating that the dataset is structured data, and dimensionality reduction embeddings representing the semantics of the dataset. The dataset output token generated based on the dataset LLM may identify satellite and / or aerial imagery containing infrared spectral data that is likely to contain data usable for detecting methane leaks, as well as maps that logically and / or visually segment New Hampshire into rural and non-rural areas.

[0058] In other implementations, the VLM may be used, for example, alone or in conjunction with the aforementioned dataset LLM, to determine whether a candidate visual dataset depicts or represents data that is likely to be relevant to or responsive to a geospatial task. For example, one or more workflow actions may be formulated as queries used to construct input prompts for the VLM, along with the image to be queried. The output generated based on the VLM may indicate whether the image contains geospatial features relevant to performing a geospatial task.

[0059] Once a response dataset is identified by the dataset processing engine 152, the data manipulation processing engine 156 may use the data manipulation LLM stored in the data manipulation (DM) LLM database 158 to generate data manipulation output tokens that identify data manipulation instructions for assembling (e.g., joining) data from one or more response datasets into a response that satisfies a natural language request and / or geospatial task. Data from different datasets can be joined in various ways. If two or more of the datasets are images (including satellite and / or aerial imagery, rasters or vectors), joining the data may include spatial alignment, superimposing all or part of one image onto another (including increasing the transparency of one or the other image as needed), projecting annotations from one image (e.g., of detected objects) onto corresponding geographic locations in the other image, etc. If two or more of the datasets are databases, joining the data may include formulating and executing database queries (e.g., in SQL) that join or otherwise extract data as requested or as needed.

[0060] Data manipulation instructions can take various forms, such as source code written in a high-level programming language, instructions written in a DSL embedded in another application, a series of actions performed using an application's API, or a series of input actions performed automatically within an application. In the methane leak embodiment, data manipulation instructions may cause satellite and / or aerial infrared imagery to be plotted on a map segmenting New Hampshire into rural and non-rural areas (e.g., spatially aligned). The data manipulation instructions may then detect and visually annotate methane leaks entering the non-rural areas. These visual annotations may be rendered on the map the user was initially viewing.

[0061] Once generated, the data manipulation instructions can be executed using the response dataset by, for example, the data manipulation processing engine 156 and / or another component of the geospatial knowledge system 120 to produce a response that satisfies the geospatial task. In the methane leak embodiment, the response could be, for example, an additional annotation of the methane leak rendered on top of a map of New Hampshire that the user has already viewed or on a separate map.

[0062] Figure 2 schematically illustrates one embodiment of how the various components of Figure 1 may cooperate to implement a selected aspect of the present disclosure. Starting from the top, the client device 110 may provide the geospatial knowledge system 120 with data representing a natural language request 260 typed or uttered by the user (not depicted). In various implementations, the natural language request 260 may communicate a geospatial task that the user wants to accomplish (or at least wants to prepare a workflow for accomplishing). In other implementations, other modalities of data, such as visual data (e.g., a map the user is looking at, satellite imagery currently viewed by the user), may be provided as input.

[0063] The geospatial knowledge system 120 may cause the LLM engine 142 to process data representing natural language requests 260, for example, as input tokens for a workflow LLM stored in the LLM database 144. The workflow LLM may generate workflow output tokens 262 that identify high-level actions to complete the geospatial task communicated in the natural language request. In some implementations, these high-level actions may be expressed in natural language.

[0064] The workflow output token 262 can then be used by the dataset processing engine 152 to assemble input prompts for dataset LLMs stored in the dataset LLM database 154. In some implementations, the workflow output token 262 may be used directly as an input token for the dataset LLM. In other implementations, a high-level action identified from the workflow output token 262 may be used. In some implementations, the input prompt for the dataset LLM may also include metadata associated with candidate datasets available to the user and / or client device 110.

[0065] The dataset processing engine 152 may use the dataset LLM to generate a dataset output token 264 that identifies one or more response datasets that are likely to contain data responding to a geospatial task communicated in a natural language request. For example, the dataset output token 264 may identify which of the candidate datasets is most likely to contain suitable data for accomplishing the geospatial task. In other implementations, the dataset LLM may be used as a classifier that processes metadata for each candidate dataset separately and outputs pass or fail.

[0066] The dataset output token 264 may be used by the DM processing engine 156 to assemble input prompts for DM LLM from the DM LLM database 158. In some implementations, as indicated by the right arrow, a workflow output token (or data indicating it) may also be provided to the DM processing engine 156 and used to assemble input prompts for DM LLM from the DM LLM database 158. Using DM LLM, the DM processing engine 156 may generate a data manipulation output token 266 that identifies data manipulation instructions for assembling data from one or more response datasets into a response that satisfies a natural language request 260.

[0067] In various implementations, the data manipulation output token 266 may then be processed by, for example, the DM processing engine 156 or another component of the geospatial knowledge system 120 to generate a response 268 to the natural language request 260. The data representing the response 268 may then be provided to the client device 110, for example, so that the rendering engine 112 can render an appropriate audible or visual output.

[0068] In some implementations, the DM processing engine 156 may evaluate the data manipulation output tokens 266 to determine whether they meet various criteria before, for example, assembling a response 268 and sending it back to the client device 110. For example, if the data manipulation output tokens 266 contain or indicate source code of a high-level programming language, the DM processing engine 156 may attempt to compile the source code. If a compiler error is thrown, the DM processing engine 156 may take various corrective actions. In some implementations, the source code and / or compiler errors may be provided as input to an error correction LLM (not depicted), which generates new source code as an output token. The new source code may be evaluated again by the DM processing engine 156, and the process may be repeated, for example, until no more compiler errors are thrown or until the maximum number of loops is reached. If the criteria are met, a response 268 may be generated and provided to the client device 110.

[0069] In some implementations, the geospatial knowledge system 120 provides a multi-turn human-versus-computer dialogue that allows the user to iteratively and / or progressively generate a geospatial workflow, which may be refined in each iteration to achieve the user's geospatial objective. For example, the process depicted in Figure 2 may be iteratively performed to finally provide a response 268 to the user's natural language request 260. If the user is not satisfied with the response 268, the user may provide feedback on the response and / or make additional geospatial requests, for example, in spoken or typed natural language.

[0070] For example, in the methane leak example, the user may be presented with a map annotated with methane leaks in cities, but in oil and gas fields, since they are considered "rural" in the applicable dataset, a map annotated with methane leaks is not presented. However, methane leaks in oil and gas fields can be largely man-made. Therefore, the user may utter a follow-up natural language request 260 such as, "OK, I also need to see methane leaks in oil and gas fields." The process in Figure 2 can be repeated, for example, with various output tokens (e.g., 262, 264, 266) and / or other data generated during the first iteration being reused in the second iteration. For example, the various tokens (e.g., 262, 264, 266) generated during the first iteration may be reused in input prompts processed by various LLMs during the second iteration, along with the user's follow-up natural language requests.

[0071] Figures 3A and 3B illustrate an embodiment of a method by which a geospatial task can be accomplished using the techniques described herein. In this embodiment, it can be assumed that a user (not depicted) operates a client device 110 to browse a map of the neighborhood 370. In Figure 3A, the user issues a natural language request, “Plot all sewer pipes north of Deer Park Avenue.” This may trigger the workflow depicted in Figure 2. The user’s natural language request may be processed by the LLM engine 142 using, for example, a workflow LLM stored in database 144 to generate a workflow output token (e.g., 262 in Figure 2) that identifies a set of high-level actions (e.g., expressed in natural language) that can be performed to accomplish the geospatial task communicated in the user’s natural language request, namely plotting all sewer pipes north of the street called “Deer Park Avenue,” as candidate workflows.

[0072] In some cases, multiple candidate workflows may be generated. For example, a first candidate workflow may be generated assuming that a mapping application operated by a user to view map 370 has access to data about sewer pipes and the functionality to render them on the map. In this case, the first candidate workflow may simply include actions to automatically interact with and / or navigate the graphical user interface (GUI) or API of the mapping application in order to render the appropriate sewer pipes. A second candidate workflow may be generated, including actions to (a) include sewer pipe data and (b) identify maps that at least partially overlap map 370, spatially align those maps, and detect sewer pipes so that annotations indicating the detected sewer pipes can be rendered on map 370.

[0073] Data indicating these high-level actions, such as the workflow output token itself, may then be processed by the dataset processing engine 152, along with metadata associated with candidate datasets accessible, for example, to the user and / or client device 110. Based on the dataset output token (264 in Figure 2) generated from this additional processing, one or more candidate datasets may be selected as a response dataset containing geospatial data that potentially responds to the user's query / geospatial task. In the embodiment of Figure 3A, for example, a municipal map of sewer pipes within the area currently depicted on map 370 may be selected.

[0074] Once a response dataset is selected, the response dataset and workflow data may be processed by the DM processing engine 156, for example, using DM LLM stored in the DM LLM database 158. This additional processing may generate data manipulation output tokens (e.g., 266 in Figure 2) that (directly or indirectly) identify data manipulation commands for manipulating data from the response dataset to construct a response to a user's natural language request (e.g., 268 in Figure 2). In the embodiment of Figure 3A, for example, the response may be a map 370 annotated with dashed lines 372 that spatially represent sewer pipes.

[0075] As described elsewhere in this specification, the geospatial knowledge system 120 may, in some cases, provide a multi-turn chatbot-like experience in which the user can iteratively generate and execute geospatial workflows across multiple turns of human-to-computer dialogue. Figure 3B illustrates one embodiment of how the user in Figure 3A may request additional geospatial processing. In Figure 3B, the user issues a subsequent natural language request: “Here, add the sewer pipe south of Deer Park Avenue.” This may restart the processing depicted in Figure 2, but except in this case, the tokens and / or other data generated during the previous iteration (i.e., the dialogue turn leading to the annotation 372 added to map 370 in Figure 3A) may also be incorporated into various LLM input prompts, in addition to the additional data indicating the user’s subsequent natural language request.

[0076] In the embodiment of Figure 3B, the same response dataset used in Figure 3A (the municipal sewer map) likely contained additional information (sewers south of Deer Park Avenue) necessary to satisfy a subsequent natural language request. However, other subsequent requests may identify additional response datasets if they require or depend on geospatial data not included in the previously used response datasets. For example, suppose the user issues yet another request: "Plot only sewers made of PVC here." The municipal sewer map may not specify the material used for the sewers. However, the information contained in the municipal map could be cross-referenced with another dataset (e.g., a database of city-wide sewers including their locations and the materials they are made of) to determine what the annotated pipes in Figure 3A and / or Figure 3B are made of. The final result could be that only sewers made of PVC are annotated, or that PVC pipes are annotated differently from non-PVC pipes (e.g., different color, dashed vs. solid).

[0077] Figure 4 illustrates a flowchart illustrating exemplary ways of implementing a selected aspect of this disclosure in various implementation forms. For convenience, the operation of Method 400 is described with reference to a system that performs the operation. This system may include one or more processors, memory, and / or other components of a computing device. Furthermore, the operation of Method 400 is shown in a particular order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0078] In block 402, the system may, for example, use the input processing engine 140 and / or the LLM engine 142 to process a natural language request (e.g., 260 in Figure 2) based on one or more generative models, such as one or more LLMs, to generate a workflow output token (e.g., 262 in Figure 2) that identifies a high-level action to complete the geospatial task communicated in the natural language request. In some implementations, the LLM engine 142 may use a workflow LLM specifically trained to generate workflows. In other implementations, the LLM engine 142 may use a more general-purpose LLM to process the data representing the natural language request.

[0079] As mentioned above, in some implementations, the Workflow LLM can be conditioned to base the actions of candidate workflows on the user's context, for example, so that only feasible workflow actions are proposed. Unfeasible workflow actions, such as actions that can only be performed using datasets not available for a particular use, may be discarded. For this purpose, the user's context may include metadata about the user's freely available datasets and / or application programming interface (API) ecosystem. This context data may be used to condition the Workflow LLM, for example, along with natural language requests. As a result, the Workflow LLM may generate output tokens representing the proposed workflow and semantically and contextually appropriate configuration workflow actions.

[0080] In block 404, the system may process first data (e.g., a workflow output token 262, or an embedding generated therefrom) that indicates a high-level action to complete a geospatial task, based on one or more LLMs, such as the same LLM applied in block 402, or separate generative models in the dataset LLM database 154 that include the dataset LLM and / or VLM, by the dataset processing engine 152, for example. Based on this processing, the dataset processing engine 152 may generate a dataset output token (e.g., 264 in Figure 2) that identifies one or more response datasets that are likely to contain data responding to a geospatial task communicated in a natural language request.

[0081] In block 406, the system may, for example, by the DM processing engine 156, process second data indicating both high-level actions for completing a geospatial task and one or more response datasets (e.g., 262 and 264) based on one or more LLMs (e.g., from database 158) to generate data manipulation output tokens (e.g., 266) that identify data manipulation instructions for assembling the data from one or more response datasets into a response that satisfies the geospatial task. In some implementations, the data manipulation output tokens (e.g., 266) may be source code tokens that collectively form executable source code for performing the geospatial task.

[0082] In block 408, the system may evaluate data manipulation instructions identified from data manipulation output tokens based on various criteria, for example, by the DM processing engine 156. If the data manipulation instructions are source code, in some implementations the criteria may include whether the source code can be successfully compiled, or whether the source code can be parsed and contains correct syntax. In other implementations the criteria may include whether the result produced by executing the data manipulation instructions (e.g., 268) successfully satisfies the user's natural language request (e.g., 260).

[0083] In block 410, if the criteria are not met, in some implementations, in block 412, the system may process the data manipulation instructions based on the error-correcting LLM to generate new data manipulation instructions that are, for example, compileable and free of errors and / or bugs. For this reason, in some implementations, the error-correcting LLM may be trained with examples of source code containing errors and / or bugs, and / or examples of source code that do not contain errors and / or bugs.

[0084] If the criteria are met in block 410, method 400 may proceed to block 414. In block 414, the system may, for example, have the DM processing engine 156 execute data manipulation commands using the response dataset to generate a response (e.g., 268) that satisfies the geospatial task. In block 416, the system may have the response that satisfies the geospatial task rendered on one or more computing devices, such as client devices 110, via the rendering engine 112.

[0085] As indicated by the arrow on the left side of Figure 4, in some implementations, various data generated during the operation of blocks 402-416, such as various output tokens 262, 264, 266, response 268, and natural language request 260, can be returned to the LLM engine 142, which can then process new natural language requests (e.g., follow-up requests issued in Figure 3B) that attempt to modify or extend the original geospatial task. In this way, as described above, the system can provide a multi-turn chatbot interface for iteratively and recursively developing increasingly complex and / or detailed geospatial tasks.

[0086] Figure 5 shows a block diagram of an exemplary computing device 510 that may be optionally used to implement one or more embodiments of the technology described herein. In some implementations, one or more of the client device, cloud-based automated assistant component, or other cloud-based software application component, and / or other components may comprise one or more components of the exemplary computing device 510.

[0087] The computing device 510 typically includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0088] The user interface input device 522 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphics tablets, scanners, touchscreens integrated into displays, voice recognition systems, microphones, and / or other types of input devices. Generally, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 510 or a communication network.

[0089] The user interface output device 520 may include a non-visual display such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), liquid crystal display (LCD), projection device, or any other mechanism for creating a visible image. The display subsystem may also provide a non-visual display via an audio output device, etc. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or another machine or computing device.

[0090] The storage subsystem 524 stores programming and data structures that provide some or all of the functions of the modules described herein. For example, the storage subsystem 524 may include logic for carrying out selected embodiments of the methods described herein, as well as logic for carrying out the various components depicted in Figure 1 or Figure 2.

[0091] These software modules are generally executed by processor 514 alone or in combination with other processors. The memory 525 used within the storage subsystem 524 may include several memories, including main random access memory (RAM) 530 for storing instructions and data during program execution, and read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules performing functions in a particular implementation may be stored by the file storage subsystem 526 within the storage subsystem 524 or on other machines accessible by processor 514.

[0092] The bus subsystem 512 provides a mechanism for various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem 512 may utilize multiple buses.

[0093] The computing device 510 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing systems or computing devices. Due to the constantly changing nature of computers and networks, the description of the computing device 510 depicted in Figure 5 is intended only as a specific example to illustrate several implementation forms. Many other configurations of the computing device 510 are possible, having more or fewer components than the computing device depicted in Figure 5.

[0094] While several implementations have been described and illustrated herein, various other means and / or structures may be utilized to perform the function and / or to obtain one or more of the results and / or benefits described herein, and each of such variations and / or modifications shall be considered to fall within the scope of the implementations described herein. More generally, all parameters, dimensions, materials and configurations described herein are intended to be illustrative, and actual parameters, dimensions, materials and / or configurations shall depend on one or more specific applications in which this teaching is used. Those skilled in the art will be able to recognize or investigate many equivalents to the specific implementations described herein simply by using routine experiments. Thus, it will be understood that the aforementioned implementations are presented only as examples, and that within the scope of the appended claims and their equivalents, implementations may be practiced in ways other than those specifically described and claimed. The implementations of this disclosure shall cover each individual feature, system, article, material, kit and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

Claims

1. A method that is carried out using one or more processors, Processing natural language requests based on one or more generative models to generate workflow output tokens that identify high-level actions to complete geospatial tasks communicated in the natural language requests, Based on one or more of the generative models, process first data indicating the high-level actions to complete the geospatial task to generate a dataset output token that identifies one or more response datasets that are likely to contain data responding to the geospatial task communicated in the natural language request; Based on one or more of the generative models, process second data representing both the high-level actions for completing the geospatial task and one or more response datasets to generate data manipulation output tokens that identify data manipulation instructions for assembling the data from the one or more response datasets into a response that satisfies the geospatial task; Using the one or more response datasets, execute the data manipulation command to generate the response that satisfies the geospatial task, The process involves rendering the response that satisfies the geospatial task on one or more computing devices, Methods that include...

2. The method according to claim 1, wherein the natural language request is processed based on a generative model different from the first data.

3. The method according to claim 1 or 2, wherein the natural language request is processed based on a generative model different from the second data.

4. The method according to any one of claims 1 to 3, wherein the first data is processed based on a generative model different from that of the second data.

5. The method according to any one of claims 1 to 4, wherein the first data further comprises metadata relating to a plurality of candidate datasets, from which one or more response datasets are identified.

6. The method according to claim 5, wherein the metadata includes human-curated content describing one or more of the candidate datasets.

7. The method according to claim 5, wherein the metadata includes data indicating one or more database schemas among the candidate datasets.

8. The method according to any one of claims 1 to 7, wherein the dataset output token represents a measure of the usefulness of each of the candidate datasets, from which one or more response datasets are identified, for performing the high-level action to complete the geospatial task communicated in the natural language request.

9. The method according to any one of claims 1 to 8, wherein the one or more response datasets include a first overhead digital depiction of a geographic area, and the first overhead digital depiction includes annotations that identify land-based features of the geographic area.

10. The method according to claim 9, wherein the first overhead digital rendering includes a raster image.

11. The method according to claim 10, wherein the raster image is captured by a satellite or drone.

12. The method according to claim 9, wherein the first overhead digital rendering includes a vector-based map of the geographic area.

13. The method according to claim 9, wherein the one or more response datasets include a second overhead digital depiction of at least a portion of the geographic area.

14. The method according to claim 13, wherein the data manipulation command includes a command for combining the first overhead digital drawing and the second overhead digital drawing.

15. The method according to claim 13, wherein the data manipulation command includes a command for overlaying one of the first overhead digital drawing and the second overhead digital drawing on the other.

16. The method according to any one of claims 1 to 15, wherein the data manipulation instruction includes source code written in a high-level programming language, and the source code is configured to be executed to retrieve data from one or more response datasets and to assemble the response to the natural language request.

17. The method according to any one of claims 1 to 16, wherein the one or more response datasets include at least a first response database and a second response database.

18. The method according to claim 17, wherein the data manipulation command includes a command for combining data from a first response database and a second response database.

19. The method according to claim 18, wherein the instruction for joining data from the first response database and the second response database includes a structured query language (SQL) instruction.

20. The method according to claim 18, wherein the instruction for combining data from the first response database and the second response database includes source code written in a high-level programming language.

21. Evaluating the data manipulation command based on one or more criteria, Based on the determination that the data manipulation instruction cannot satisfy one or more of the criteria, assemble an input prompt for one or more of the generative models, wherein the input prompt includes at least some of the data manipulation instructions and additional information relating to the determination that the data manipulation instruction cannot satisfy one or more of the criteria. The method according to any one of claims 1 to 20, further comprising processing the input prompt using one or more of the generation models to generate a new data manipulation instruction.

22. The method according to claim 21, wherein the data manipulation instruction includes source code in a high-level programming language, and the one or more criteria include the ability to compile the source code without error.

23. The method according to any one of claims 1 to 22, wherein one or more of the generative models include a Large-Scale Language Model (LLM).

24. A system comprising one or more processors and a memory for storing instructions, wherein, in response to the execution of an instruction by the one or more processors, the system causes the one or more processors to perform the method according to any one of claims 1 to 23.

25. At least one non-temporary computer-readable medium containing instructions causing one or more processors to perform the method according to any one of claims 1 to 23.

Images