Foundational Model for Semantic Routing

Abstract

Training data is acquired. The training data includes (a) route information indicating a route from a starting point to a destination, wherein the route includes multiple route segments, each including a subset of a first route segment and a subset of a second route segment; and (b) route feature information describing one or more route features. At least a subset of the first route segment and a portion of the route feature information associated with the subset of the first route segment are processed using a machine learning-prepared semantic routing model to acquire one or more predicted route segments for the subset of the second route segment. One or more parameters of the machine learning-prepared semantic routing model are adjusted based on an optimization function that evaluates the difference between one or more predicted route segments and the subset of the second route segment.

Description

Technical Field

[0001] This application claims priority to U.S. Non-Provisional Patent Application No. 18 / 468,338, filed on September 15, 2023, which application is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to semantic routing. More specifically, the present disclosure relates to a base trained machine learning model trained for semantic understanding of mapping information.

Background Art

[0003] Base models, such as large language models (LLMs), are models with a large number of parameters that are trained using large-scale datasets to perform multiple tasks. Currently, base models are bringing innovation to what can be achieved by assisted AI technology in many fields, from general conversation assistants to multimodal editing of content such as audio, images, and videos. Once trained, the base model can perform a wide variety of tasks within the context of the training data used to train the model. For example, an LLM, once trained, can perform a wide variety of language tasks.

Summary of the Invention

[0004] Aspects and advantages of the embodiments of the present disclosure are set forth in part in the description that follows, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] One exemplary aspect of the present disclosure relates to a method implemented on a computer. The method includes acquiring training data by a computing system comprising one or more computing devices, the training data comprising (a) route information indicating a route from a starting location to a destination location, wherein the route comprises a plurality of route segments including a subset of first route segments and a subset of second route segments, and (b) route feature information describing one or more route features. The method includes processing at least a subset of first route segments and a portion of the route feature information associated with the subset of first route segments using a machine learning-prepared semantic routing model by the computing system in order to acquire one or more predicted route segments for a subset of second route segments. The method includes tuning one or more parameters of the machine learning-prepared semantic routing model by the computing system based on an optimization function that evaluates the difference between one or more predicted route segments and a second subset of route segments.

[0006] Other exemplary aspects of this disclosure relate to a computing system comprising one or more processors and memory. The memory includes a machine-learned semantic routing model, which is trained to process mapping information to produce a model output that includes proposed route segments and / or information associated with route segments. The memory includes one or more tangible, non-temporary computer-readable media that, when executed by one or more processors, store computer-readable instructions that cause one or more processors to perform an action. An action includes obtaining one or more inputs from a client computing device to the machine-learned semantic routing model, the one or more inputs including at least one of request information indicating a request for requested route segments and / or mapping-related information, or route feature information indicating one or more route features. An action includes processing one or more inputs to obtain a model output, the model output including at least one of (a) routing information indicating a route that includes one or more proposed route segments, or (b) semantic mapping information associated with a route. An action includes providing the model output to the client computing device.

[0007] Other exemplary aspects of the present disclosure relate to one or more tangible, non-temporary, computer-readable media that, when executed by one or more processors, store computer-readable instructions causing one or more processors to perform an action. The action includes acquiring training data, which includes (a) route information indicating a route from a starting location to a destination location, wherein the route includes a plurality of route segments, including a subset of a first route segment and a subset of a second route segment; and (b) route feature information describing one or more route features. The action includes processing at least a subset of the first route segments and a portion of the route feature information associated with the subset of the first route segments using a machine-learned semantic routing model in order to acquire one or more predicted route segments for a subset of the second route segments. The action includes tuning one or more parameters of the machine-learned semantic routing model based on an optimization function that evaluates the difference between one or more predicted route segments and a subset of the second route segments.

[0008] Other aspects of this disclosure cover a variety of systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part of this specification illustrate exemplary embodiments of this disclosure and, together with the description, serve to illustrate the relevant principles.

[0010] A detailed description of embodiments intended for those skilled in the art is provided herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1A]The following are block diagrams of exemplary computing systems for training and utilizing machine learning-based semantic routing models according to some embodiments of this disclosure. [Figure 1B] The following are block diagrams of exemplary computing devices for training and pre-training machine learning-based semantic routing models according to some embodiments of the present disclosure. [Figure 1C] The following are block diagrams of exemplary computing devices that generate mapping information or mapping-related information using a machine learning-prepared semantic routing model, according to some embodiments of the present disclosure. [Figure 2A] This is a data flow diagram for pre-training a machine learning-based semantic routing model according to some embodiments of the present disclosure. [Figure 2B] This is a data flow diagram for performing subsequent pre-training on a machine learning-prepared semantic routing model using route information different from the route information in Figure 2A, according to some embodiments of the present disclosure. [Figure 3] This is a block diagram of an exemplary machine learning-trained semantic routing model 300 according to some embodiments of the present disclosure. [Figure 4] The following are flowcharts illustrating exemplary methods for training and / or fine-tuning machine learning-based semantic routing models according to some embodiments of the present disclosure. [Figure 5] The flowcharts illustrate exemplary methods for performing map-related tasks using machine learning-prepared semantic routing models according to some embodiments of this disclosure. [Modes for carrying out the invention]

[0012] The repeated reference numbers across multiple drawings are intended to identify the same features in various embodiments.

[0013] overview This disclosure pertains to semantic routing. More specifically, this disclosure pertains to underlying machine learning models trained for the semantic understanding of mapping information. In particular, training data can be obtained to train underlying mapping models, such as machine learning semantic routing models. The training data may include route information that shows a route containing several route segments (e.g., an actual route previously requested and performed by a user). The training data may also include route feature information. Route feature information may be metadata associated with a route (e.g., preferred route type, transport method, contextual information, etc.). For example, if the route indicated by the route information is a route previously provided to a user, the route feature information may include intermediate locations within the route, entities located along the route (e.g., businesses, points of interest (POIs), landmarks, etc.).

[0014] Training data can be used to train or fine-tune a machine learning-based semantic routing model. A machine learning-based semantic routing model can be a foundational model for map-related tasks (e.g., generating routes and / or route segments, semantic analysis of routes, etc.) that includes numerous parameters and is trained on a large corpus of training data (e.g., mapping data). To train or fine-tune a machine learning-based semantic routing model, portions of route segments close to the destination location of a route can be masked. Training data can be input into the machine learning-based semantic routing model to obtain model output. The model output may include predicted route segments for the masked route segments. One or more parameters of the machine learning-based semantic routing model can be tuned using an optimization function that evaluates the difference between the predicted route segments and the masked route segments. In this way, the machine learning-based semantic routing model can be trained to perform multiple mapping tasks.

[0015] The embodiments of this disclosure bring about several technical effects and advantages. As an exemplary technical effect and advantage, embodiments of this disclosure can significantly improve the efficiency of mapping applications. In particular, conventional mapping applications are generally difficult to navigate and offer relatively few input options for users to specify a particular route or a route with specific characteristics. For example, there are few mapping applications that can provide a function for a user to request a route that follows a scenic coastal route. However, embodiments of this disclosure provide a foundational model that can generate routing information based on a semantic understanding of user requests. In this way, mapping applications can be provided with more accurate and efficient routes that satisfy the semantic meaning of user requests.

[0016] Other exemplary technical effects and benefits include the fact that while conventional mapping applications can sometimes generate the optimal route between a start and end point, they exhibit relatively poor performance when the route needs to be adjusted while traveling or when generating routes for combined transport (e.g., a combination of public transport and private transport). However, embodiments of the present disclosure can predict more efficient route segments for routes that are already in progress. This, in turn, can significantly reduce the user's travel time and, consequently, significantly reduce the consumption of resources required to travel the route (e.g., energy resources for autonomous vehicles, computing resources for on-board computers, fuel resources for conventional vehicles, etc.).

[0017] Exemplary embodiments of this disclosure will be described in more detail here with reference to the figures.

[0018] Exemplary devices and systems FIG. 1A shows a block diagram of an example computing system 100 for training and utilizing a machine-learned semantic routing model in accordance with some embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicatively coupled via a network 180.

[0019] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0020] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be a single processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118, and the instructions 118 are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0021] In some embodiments, the user computing device 102 may store or include one or more trained machine learning semantic routing models 120. For example, the trained machine learning semantic routing model 120 may be or include various trained machine learning models such as neural networks (e.g., deep neural networks), or other types of trained machine learning models including non-linear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some exemplary trained machine learning models can utilize attention mechanisms such as self-attention. For example, some exemplary trained machine learning models may include multi-head self-attention models (e.g., Transformer models).

[0022] In particular, in some embodiments, the trained machine learning semantic routing model 120 may be a model trained to process text content, or may otherwise include such a model, or may be a part of a model trained to process text content, or may otherwise include such a part. For example, models such as LLMs are trained to generate text content based on text input. To do so, an LLM generally includes a decoder portion (and in some examples, an encoder portion and / or a decoder portion) that processes text input to generate an intermediate representation of the text input (e.g., a sequence of tokens, etc.). This intermediate representation is further processed to ultimately generate a text output. However, the decoder portion of the LLM may, in some embodiments, be included in the trained machine learning semantic routing model 120, which allows text input from a user (e.g., "I need a scenic route to the grocery store that avoids highways") to be processed together with conventional mapping information.

[0023] Similarly, in some embodiments, the machine learning-trained semantic routing model 120 may include encoder or decoder units from other underlying models. For example, the machine learning-trained semantic routing model 120 may include encoder and / or decoder units from an underlying computer vision model trained to generate intermediate representations of image data (e.g., video, still images, renderings, textures, etc.). In other examples, the machine learning-trained semantic routing model 120 may include encoder and / or decoder units from an underlying computer vision model trained to generate intermediate representations of audio data (e.g., recordings, speech, etc.). In this way, the machine learning-trained semantic routing model 120 can leverage multimodal inputs to more accurately generate routes for the user.

[0024] It should be noted that certain base models, or parts of base models (e.g., encoder and / or decoder sections), are described as being included in the machine-learned semantic routing model 120. However, in some embodiments, such base models or parts of base models can be stored and instantiated separately from the machine-learned semantic routing model 120. More generally, such base models or parts of base models can be used in conjunction with the machine-learned semantic routing model 120, but can also be instantiated, executed, stored, etc., separately from the machine-learned semantic routing model 120. Specific parts and layers of the machine-learned semantic routing model 120 are described in more detail herein.

[0025] In some embodiments, one or more machine learning-prepared semantic routing models 120 are received from a server computing system 130 via a network 180, stored in user computing device memory 114, and then made available to or otherwise implemented by one or more processors 112. In some embodiments, the user computing device 102 can implement multiple parallel instances of a single machine learning-prepared semantic routing model 120 (for example, to perform parallel machine learning tasks across multiple instances of the machine learning-prepared semantic routing model 120).

[0026] The machine learning-based semantic routing model 120 can perform a wide variety of mapping-related tasks. More specifically, the machine learning-based semantic routing model 120 can process mapping information and / or additional map-related information to generate mapping-related model outputs. In some embodiments, the machine learning-based semantic routing model 120 can process model inputs to generate predicted route segments. These model inputs may include previous or current route segments, information indicating routing requests from users, image data indicating where the route is requested, information describing multiple possible locations, contextual information (e.g., information indicating available transportation options, user preferences, weather conditions, etc.), itinerary requests, or any other type or input method related to mapping.

[0027] In addition or alternatively, in some embodiments, the machine learning-trained semantic routing model 120 can process model inputs to generate semantic mapping information. Generally, semantic mapping information can refer to information that describes a semantic understanding of routes, geographical locations, points of interest, etc. For example, the model input may be a query requesting the route segment that a user is most likely to travel to when searching for a restaurant. The model output may be semantic mapping information containing the predicted route segment corresponding to the query. In another example, the model input may be a query to identify the characteristics of a user navigating a particular route segment. The semantic mapping information may indicate common characteristics of users navigating a particular route segment (e.g., type of vehicle used, mode of transport used, consumer preferences, etc.). In yet another example, the model input may be a request to identify specific features of a route, and the semantic mapping information may describe the features of the route or route segment (e.g., good scenery, dangerous weather conditions, adjacent to a specific POI, low traffic volume, etc.). In yet another example, the model input could be a request describing a specific route or route segment, and the semantic mapping information could include text content, image content, and / or audio content describing the route or route segment.

[0028] Additionally or alternatively, one or more machine learning-prepared semantic routing models 140 may be included in, or otherwise stored in, a server computing system 130 that communicates with a user computing device 102 according to a client-server relationship. For example, a machine learning-prepared model 140 may be implemented by the server computing system 130 as part of a web service (e.g., a mapping service). Thus, one or more models 120 may be stored and implemented in the user computing device 102, and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0029] Furthermore, the user computing device 102 may include one or more user input components 122 that receive user input. For example, a user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may function to implement a virtual keyboard. Other exemplary user input components include a microphone, a conventional keyboard, or other means by which the user can provide user input.

[0030] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 134 may include one or more non-temporary computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138, and the instructions 138 are executed by the processors 132 to cause the server computing system 130 to perform operations.

[0031] In some embodiments, the server computing system 130 includes one or more server computing devices, or is otherwise implemented by them. In examples where the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0032] As described above, the server computing system 130 can store, or otherwise include, one or more machine learning-prepared semantic routing models 140. For example, model 140 may be, or otherwise include, various machine learning-prepared models. Exemplary machine learning-prepared models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning-prepared models can leverage attention mechanisms such as self-attention. For example, some exemplary machine learning-prepared models may include multi-head self-attention models (e.g., transformer models).

[0033] The user computing device 102 and / or the server computing system 130 can train models 120 and / or 140 by interacting with a training computing system 150, which is connected to it communicatively via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0034] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 154 may include one or more non-temporary computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158, the instructions 158 being executed by the processors 152 to cause the server computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices, or is otherwise implemented by them.

[0035] Generally, it should be noted that "training" a model, as described herein, can also refer to "fine-tuning" or additional training / tuning iterations performed on the model. For example, an initial training session may be performed to train the model, and the model's parameters may be locked. After using the model for a period of time during the inference phase, additional tuning iterations may be performed to "fine-tune" the model to perform additional tasks or to improve the model's performance on the current task.

[0036] The training computing system 150 may include a model trainer 160 that trains machine-learned models 120 and / or 140 stored in the user computing device 102 and / or server computing system 130 using various training or learning techniques, such as backpropagation. For example, a loss function may be backpropagated by the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update parameters over several training iterations. Alternatively, parameters may be updated using methods that do not involve gradients.

[0037] In some embodiments, backpropagation may include performing time-evolved backpropagation. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0038] In particular, the model trainer 160 can train the machine learning-based semantic routing models 120 and / or 140 based on a set of training data 162. The training data 162 may include training data for pre-training and fine-tuning the models 120 and / or 140. For example, the training data may include multiple sets of pre-training data. A pre-training pair may include route segments of routes previously traveled by users of the mapping service, or simulated routes traveled by a simulated user. A pre-training pair may also include route feature information. Route feature information may include, or otherwise include, metadata associated with the route traveled by the user. Route feature information may indicate the method of transport used, total time, traffic information, entities located along or near the route (e.g., POIs, businesses, landmarks, houses, etc.), initial requests made by the user, geolocation information (e.g., coordinates of the starting and ending locations), etc.

[0039] In some embodiments, pre-training pairs can be dynamically generated. In particular, a single route containing N segments can be used to generate N-1 training pairs that can be used in various ways. For example, given 1, 2, ..., K, the training task is to predict the route segments K+1, ..., N (continuation part). Another example is a training task to predict K (masking the segments) given 1, 2, ..., _, K+1, ..., N.

[0040] The model trainer 160 can train the machine learning-prepared semantic routing models 120 and / or 140 by masking specific route segments of a pre-trained pair to obtain model outputs, and then processing the pre-trained pair using the machine learning-prepared semantic routing models 120 and / or 140. The model outputs may include predicted route segments for the masked route segments. The model trainer 160 can then train and evaluate an optimization function that evaluates the difference between the masked route segments and the predicted route segments. Based on the optimization function, the model trainer can adjust the parameter values ​​of the machine learning-prepared semantic routing models 120 and / or 140.

[0041] The training data 162 may further include fine-tuning data for fine-tuning the machine learning-based semantic routing models 120 and / or 140. The type or form of the fine-tuning data included in the training data 162 may vary based on the task(s) to which the machine learning-based semantic routing models 120 and / or 140 are fine-tuned to function. For example, suppose the machine learning-based semantic routing models 120 and / or 140 are pre-trained and fine-tuned to generate semantic text descriptions of routes. The set of fine-tuning data may include routing information that shows scenic routes along coastlines and ground truth text descriptions of routes. The model trainer 160 can process the routing information using the machine learning-based semantic routing models 120 and / or 140 to produce model outputs that include predicted text descriptions of routes. The model trainer 160 can train the machine learning-based semantic routing models 120 and / or 140 based on an optimization function that evaluates the difference between the ground truth text descriptions and the predicted text descriptions.

[0042] In some embodiments, if the user consents, training examples can be provided by the user computing device 102. Thus, in such embodiments, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some examples, this process may be referred to as model personalization.

[0043] The model trainer 160 includes computer logic used to provide a desired function. The model trainer 160 can be implemented in hardware, firmware, and / or software that control a general-purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored in a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer executable instructions stored in RAM, a hard disk, or a tangible computer-readable storage medium such as an optical or magnetic medium.

[0044] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or a combination thereof, and may include any number of wired or wireless links. In general, communications through Network 180 can be transmitted over any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection methods (e.g., VPN, Secure HTTP, SSL).

[0045] The machine learning models described herein can be used in a variety of tasks, applications, and / or use cases.

[0046] In some embodiments, the input to the machine learning model(s) of this disclosure may be image data. The machine learning model(s) may process the image data to generate an output. For example, the machine learning model(s) may process the image data to generate an image recognition output (e.g., recognition of image data, latent embedding of image data, encoded representation of image data, hash of image data, etc.), and further use that image recognition output to generate routing information, map-related information, etc. In another example, the machine learning model(s) may process the image data to generate a prediction output, and further use that prediction output to generate routing information, map-related information, etc.

[0047] In some embodiments, the input to the machine learning model(s) of this disclosure may be text or natural language data. The machine learning model(s) may process the text or natural language data to generate an output. For example, the machine learning model(s) may process the text or natural language data to generate a latent text embedding output, and further use that latent text embedding output to generate routing information, map-related information, etc. Another example is that the machine learning model(s) may process the text or natural language data to generate a translation output, and further use that translation output to generate routing information, map-related information, etc. Yet another example is that the machine learning model(s) may process the text or natural language data to generate a semantic intent output, and further use that semantic intent output to generate routing information, map-related information, etc.

[0048] In some embodiments, the input to the machine learning model(s) of this disclosure may be audio data. The machine learning model(s) may process the audio data to generate an output. For example, the machine learning model(s) may process the audio data to generate a speech recognition output, and further use that speech recognition output to generate routing information, map-related information, etc. Another example is that the machine learning model(s) may process the audio data to generate a speech translation output, and further use that speech translation output to generate routing information, map-related information, etc. Another example is that the machine learning model(s) may process the audio data to generate a latent embedding output. Another example is that the machine learning model(s) may process the audio data to generate a text representation output (e.g., a text representation of the input audio data), and further use that text representation output to generate routing information, map-related information, etc.

[0049] In some embodiments, the input to the machine learning model(s) of this disclosure may be latent encoding data (e.g., a latent spatial representation of the input). The machine learning model(s) may process the latent encoding data to generate an output. For example, the machine learning model(s) may process the latent encoding data to generate a recognition output, and further use that recognition output to generate routing information, map-related information, etc. In another example, the machine learning model(s) may process the latent encoding data to generate a prediction output, and further use that prediction output to generate routing information, map-related information, etc.

[0050] In some embodiments, the input to the machine learning model(s) of this disclosure may be sensor data. The machine learning model(s) may process the sensor data to generate an output. For example, the machine learning model(s) may process the sensor data to generate a recognition output, and further use that recognition output to generate routing information, map-related information, etc. In another example, the machine learning model(s) may process the sensor data to generate a visualization output, and further use that visualization output to generate routing information, map-related information, etc.

[0051] In some cases, the input includes vision data, the task is a computer vision task, and furthermore, routing information, map-related information, etc., can be generated using the computer vision task. In some cases, the input includes pixel data of one or more images, and the task is an image processing task. For example, the image processing task could be image classification, and the output would be a set of scores, each score corresponding to a different object class, representing the likelihood that one or more images depict an object belonging to that object class. The image processing task could be object detection, and the image processing output would identify one or more regions within one or more images, and for each region, the likelihood that the region depicts an object of interest. Another example is image segmentation, where the image processing output defines the likelihood for each category within a given set of categories for each pixel in one or more images. For example, the set of categories could be foreground and background. Another example is that the set of categories could be object classes. Another example is depth estimation, where the image processing output defines the depth value for each pixel in one or more images. As another example, the image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted in the pixels between the images in the network input, for each pixel in the input images.

[0052] In some cases, the input may include audio data representing spoken utterances, and the task may be a speech recognition task that can be further used to generate routing information, map-related information, etc. The output may include text output mapped to the spoken utterances. In some cases, the task may include microprocessor performance tasks such as branch prediction or memory address translation.

[0053] Figure 1A shows one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used in a similar manner. For example, in some embodiments, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such embodiments, the model 120 may be trained and used locally on the user computing device 102. In some such embodiments, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0054] Figure 1B shows a block diagram of an exemplary computing device 10 for training and pre-training a machine learning-based semantic routing model according to several embodiments of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0055] The computing device 10 includes multiple applications (e.g., applications 1 to N). Each application includes its own machine learning library and pre-trained models(s). For example, each application may include a pre-trained model. Exemplary applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0056] As shown in Figure 1B, each application can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0057] Figure 1C shows a block diagram of an exemplary computing device 50 that generates mapping information or mapping-related information using a machine learning-based semantic routing model, according to several embodiments of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0058] The computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Exemplary applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some embodiments, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API across all applications).

[0059] The central intelligence layer includes several machine learning models. For example, as shown in Figure 1C, each machine learning model is provided by each application and can be managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model to all applications. In some embodiments, the central intelligence layer is contained within the operating system of the computing device 50, or otherwise implemented therein.

[0060] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized repository of data for the computing device 50. As shown in Figure 1C, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0061] Figure 2A shows a data flow diagram 200A for pre-training a machine learning-based semantic routing model according to several embodiments of the present disclosure. In particular, the data flow diagram 200 includes training data 202. The training data 202 may be, or otherwise include, data used to pre-train a machine learning-based semantic routing model. The training data 202 may include route information 204. The route information 204 may represent a route from a starting point to a destination. In some embodiments, the route information 204 may represent a route previously traveled by a user, such as a user of a mapping application. In this example, the route information 204 is strictly processed to remove any identifying information in order to protect user privacy. Alternatively, in some embodiments, the route information 204 may represent a simulated route or include media and / or multimedia that generally describe a route. Alternative sources and / or types of route information 204 are described in more detail with respect to Figure 2B.

[0062] In particular, route information 204 may indicate a route comprising multiple route segments 206A, 206B, 206C, and 206D (generally, route segments 206). Each of the route segments 206 may include, or otherwise indicate, information of any other type or form relating to the route segment 206 of the route indicated by route information 204, including the starting point, destination, method of travel, time spent navigating the route segment, and any other type or form relating to the route segment 206 of the route indicated by route information 204. Following the illustrated example, route segment 206A may be the first route segment in the route, and therefore the starting point of route segment 206A may be the same starting point as the route indicated by route information 204. Route segment 206A may have an intermediate destination different from the final destination of the route indicated by route information 204. In some embodiments, the intermediate destination of a route segment may be the intermediate starting point of the next route segment immediately following the current route segment. As illustrated in the example, the intermediate destination location of route segment 206A (e.g., 35.78, -78.81) may be the same as the starting intermediate start location of route segment 206B that immediately follows route segment 206A (e.g., 35.78, -78.81).

[0063] As illustrated, in some embodiments, the root segment 206 may indicate a location (e.g., starting location, destination location, etc.) using latitude / longitude coordinates. Additionally or alternatively, in some embodiments, the root segment 206 may indicate a location in an alternative manner. For example, the root segment 206 may indicate a location based on the name of an entity associated with the location (e.g., a specific landmark, POI, business establishment, residence, etc.). In other examples, the root segment 206 may encode the location as an address.

[0064] Additionally or alternatively, as illustrated, in some embodiments, route segment 206 may indicate a method of travel used to travel through a particular route segment. For example, route segment 206A indicates the use of a private vehicle (e.g., PRIV_VEH) to travel through the route segment, while route segment 206D indicates the use of public transport (e.g., PUB_TRANS) to travel through the route segment.

[0065] Additionally or alternatively, as illustrated, in some embodiments, route segment 206 may indicate the time spent traveling through the route segment. For example, route segment 206A indicates that it took 17 minutes and 35 seconds to travel through the route segment. Note that the time required to travel through a route segment can be calculated so that only the time spent directly traveling through the segment is evaluated. In other words, the time required to travel through a route segment can exclude time spent by the user doing things other than traveling through the route segment (e.g., stopping for meals or refueling, shopping, taking detours, etc.).

[0066] The routes and route segments 206 indicated by route information 204 can utilize any type or form of transportation infrastructure, recreational resources, or infrastructure, etc. In some embodiments, the routes indicated by route information 204 may be routes that travel along roads (e.g., streets, highways, bridges, etc.). Additionally or alternatively, in some embodiments, the routes indicated by route information 204 may travel along additional transportation infrastructure that is inaccessible to conventional ground-based public or private transport vehicles such as cars, trucks, and buses. For example, a route or route segment(s) may travel along sidewalks, bicycle paths, hiking trails, alleys, buildings (e.g., entering a building via an overhead walkway or underpass, moving to another building via an underpass, etc.), tunnels, bodies of water, airspace, etc. Additionally or alternatively, in some embodiments, a route or route segment(s) may travel using transport vehicles other than conventional ground-based transport vehicles. For example, a route or segment(s) of a route can be traveled by train, subway, tram, personal means of transport (e.g., bicycle, scooter, unicycle, electronic device, etc.), boat, ferry, airplane, helicopter, vertical take-off and landing (VTOL) vehicle, etc.

[0067] Additionally or alternatively, in some embodiments, a route or segment(s) of a route may traverse recreational resources or infrastructure. For example, a route or segment(s) of a route may include ski runs, hiking trails, cycling trails, campgrounds, etc. Additionally or alternatively, in some embodiments, a route or segment(s) of a route may traverse virtual resources or infrastructure. For example, a route or segment(s) of a route may traverse virtual spaces such as video games or simulations. In particular, a route or segment(s) of a route may traverse such spaces in the same manner as corresponding non-virtual traffic resources. For example, suppose a virtualized fictional city in a video game includes traffic infrastructure similar to that of a modern city (e.g., roads, public transport, bicycle paths, etc.). A route or segment(s) of a route may traverse the traffic infrastructure of the virtualized fictional city in the same manner as traversing a modern city.

[0068] In addition or alternatively, in some embodiments, a route or segment(s) of a route may leverage, incorporate, or otherwise collaborate with a transport service provider. For example, a segment(s) of a route traveling along urban roads may be traversed using rideshare services, taxis, etc. In another example, a segment(s) of a route from one city to another may be traversed using commercial air transport services. In yet another example, a segment(s) of a route from one city to another may be traversed using public transport (e.g., buses, trains, etc.).

[0069] Additionally or alternatively, in some embodiments, a route may be a multimodal route in which different segments of the route traverse different types of transport infrastructure and / or utilize different methods of travel. For example, a first segment of the route may travel along roads to reach a train station, and then a second segment of the route may travel along railway tracks using a train. In another example, a first segment of the route may use a bicycle to travel along a recreational mountain bike trail to a parking lot where the user's personal vehicle is stored, and then a second segment of the route may use the personal vehicle to return to the user's home.

[0070] The training data 202 may include route feature information 208. Route feature information 208 may be metadata or meta-information that describes various segment-level and / or route-level features of the route indicated by route information 204, or otherwise may include them. Following the illustrated example, route feature information 208 may include weather features that describe the weather conditions at the time each of the route segments 206 was traversed. In another example, route feature information 208 may include the initial request made by the user that resulted in the generation of the route indicated by route information 204 (e.g., "from home to RDU...no highways"). In another example, route feature information 208 may include traffic features that describe the traffic conditions at the time each of the route segments 206 was traversed. Thus, it should be broadly understood that route feature information 208 may include any type or format of information associated with the route or traversal of the route indicated by route information 204, either segment-wise or route-wise.

[0071] In some embodiments, the route feature information 208 may include images associated with the route segment 206. For example, a vehicle used to travel through route segments 206A and 206B may include a camera sensor (for example, used to facilitate the operation of an autonomous vehicle). The route feature information 208 may include images captured using the vehicle's camera sensor at the start and / or destination locations of route segments 206A and 206B. In other examples, the route feature information 208 may include publicly available Street View images, traffic sensor images, satellite images, etc., of the start and / or destination locations of route segment 206.

[0072] In some embodiments, route feature information 208 may include historical user information specific to users who have traveled the route indicated by route information 204. The historical user information may, after being processed to remove any arbitrary identifying information to protect user privacy, indicate various user characteristics that may be relevant to travel through route segment 206. For example, historical user information may indicate the language(s) spoken by the user. In other examples, historical user information may indicate the means of transportation available to the user (e.g., PRIV_VEH, PUB_TRANS, SELF, etc.) and / or whether the user is able to walk or has a means of transportation that can assist walking (e.g., bicycle, electric device, mobile device, etc.). In yet another example, historical user information may include user information embeddings that function as latent representations of user information related to mapping and / or travel through the route. Thus, historical user information can capture detailed information, which may or may not be relevant in a privacy-preserving manner.

[0073] In some embodiments, route feature information 208 can be extended by additional route feature information 209. In particular, the data extender 211 can acquire additional route feature information 209 and extend route feature information 208 with the additional route feature information 209. The additional route feature information 209 may include information associated with the route segment 206, entities located along the route segment 206 (e.g., businesses, POIs, landmarks, etc.), user-generated content associated with the route segment 206, etc. For example, the additional route feature information 209 may include user-submitted reviews about businesses along a particular route segment 206. In some embodiments, the additional route feature information 209 may include user reviews specifically for the route segment 206. For example, if the route segment 206 is for recreation (e.g., hiking trail segment, ski trail segment, scenic road segment, etc.), the additional route feature information 209 may include reviews about the route segment 206 from a web source of user reviews. In some embodiments, the additional route feature information 209 may include information from a different mapping service, application, source, etc., than the mapping service(s) associated with the route information 204. For example, the additional route feature information 209 may include route metadata from other mapping applications or services, different geographic information systems (GIS), review platforms, etc.

[0074] It should be noted that the route feature information 208 is shown separately from the route information 204 in order to more clearly illustrate the various embodiments of this disclosure. Rather, in some embodiments, the route information 204 may include certain features (e.g., segment-specific weather features, segment-specific traffic features, etc.) that are shown as being included in the route feature information 208, or it may include the route feature information 208 entirely. Additionally or alternatively, in some embodiments, the route feature information 208 may include features or information (e.g., method of travel, time spent traversing the segment, starting place / destination place, etc.) that are shown as being included in the route segment 206.

[0075] The pre-trained semantic routing model 210 can be trained and / or fine-tuned using the training data 202. The pre-trained semantic routing model 210 may be any type or style of model that has been at least partially trained to process the route information 204 and route feature information 208 together with any type or style of data contained in the route information 204 and route feature information 208. To train and / or fine-tune the pre-trained semantic routing model 210, one or more route segments 206 can be masked before being processed by the pre-trained semantic routing model 210. In some embodiments, the training data 202 can be acquired after the masking has been applied to it. Alternatively, in some embodiments, the masking can be applied to the training data 202 after it has been acquired.

[0076] As illustrated in the example, a route segment 206C can be masked by obscuring or otherwise removing the information associated with the route segment 206C. In some embodiments, route feature information 208 associated with the masked route segment 206C can also be masked. For example, segment-specific traffic features of route segment 206C can be masked. Alternatively, in some embodiments, the route feature information 208 associated with the masked route segment 206C may remain unmasked or be partially masked. In some embodiments, a single route segment 206 can be masked. Instead, in some embodiments, multiple route segments 206 (e.g., consecutive route segments, discontinuous route segments, etc.) can be masked.

[0077] The training data 202 can be processed by the machine learning-prepared semantic routing model 210 to obtain predicted route segments 212. Predicted route segments 212 may be predictions of masked route segments 206C. The model trainer 214 can evaluate the predicted route segments 212 using an optimization function 216. In particular, the optimization function 216 can evaluate the difference(s) between the predicted route segments 212 and the ground truth route segments 218. The ground truth route segments 218 may be route segments 206C before masking is applied (i.e., the "original" route segments 206C). Based on the optimization function 216, the model trainer 214 can apply adjustments 220 to the parameters(s) (or the values(s)) of the machine learning-prepared semantic routing model 210. In this way, the machine learning-prepared semantic routing model 210 can be trained or pre-trained to efficiently and accurately predict route segments given routes and information associated with the routes.

[0078] Figure 2B is a data flow diagram 200B for performing subsequent pre-training on a machine learning-prepared semantic routing model using route information different from that in Figure 2A, according to some embodiments of the present disclosure. In particular, a second set of training data 222 can be obtained for fine-tuning, training, or other iterations of pre-training on the machine learning-prepared semantic routing model 210.

[0079] Training data 222 may include route information 224. Unlike the route information 204 of training data 202, the route information 224 of training data 222 may be, or otherwise include, content that semantically represents a route. Following the illustrated example, route information 224 may include a file of a book titled "My Travels Through Vietnam" (e.g., MY_TRAVELS_THROUGH_VIETNAM.epub) along with corresponding images extracted from the book (e.g., VIETNAM_IMG_1.png, VIETNAM_IMG_2.png, etc.). Although route information 224 does not explicitly describe a specific route segment, such as route segment 206 of route information 204 in Figure 2A, a machine learning-trained semantic routing model 210 can be trained and / or fine-tuned to identify and segment the routes semantically represented by route information 224.

[0080] Following the illustrated example, suppose the content of a book included in route information 224 describes a vacation in which the author visits four major cities in Vietnam. The four images included in route information 224 could each correspond to one of the four major cities. The machine learning-trained semantic routing model 210 can process the training data 222 to generate four predicted route segments 226. The predicted route segments 226 could correspond to the routes between the four major cities visited by the author. In this way, the machine learning-trained semantic routing model 210 can be trained to identify route features semantically described in media or multimedia. The machine learning-trained semantic routing model 210 can also be trained to divide the identified routes into specific segments.

[0081] The model trainer 214 can evaluate the predicted route segment 226 using the optimization function 216. The optimization function 216 can evaluate the difference(s) between the predicted route segment 226 and the ground truth route segment 228. The ground truth route segment 228 may be a route segment that precisely corresponds to a route described by the author in the route information 224. For example, the ground truth route segment 228 may be created by a user who has read the books included in the route information 224. In another example, the ground truth route segment 228 may be generated as the output of an LLM that processes at least a portion of the books included in the route information 224, or based on that output. Based on the optimization function 216, the model trainer 214 can apply adjustments 230 to the values(s) of the parameters(s) of the machine learning-prepared semantic routing model 210.

[0082] Figure 2B shows books and corresponding images as route information 224, but any type or format of media or multimedia can be used as route information 224 to train and / or fine-tune the machine learning-trained semantic routing model 210. For example, any type or format of written material can be used as route information 224, such as books, articles, blog posts, social media posts, scraped or aggregated text content, or text content generated by a machine learning-trained model. In other examples, any type or format of image or video data, such as images, movies, or television programs, can be used as route information 224. In yet another example, any type or format of simulated information, such as 3D representations or simulated information, can be used to train the machine learning-trained semantic routing model 210, such as a simulated map or space.

[0083] Figure 3 is a block diagram of an exemplary machine learning-trained semantic routing model 300 according to several embodiments of the present disclosure. Note that any part(s) of the machine learning-trained semantic routing model 300 described herein with respect to Figure 3 may or may not be a “component,” and may or may not be integrated with the machine learning-trained semantic routing model 300. In some embodiments, the machine learning-trained semantic routing model 300 may generally refer to a single model that includes multiple model parts (e.g., encoders, decoders, tokenizers, etc.). Alternatively, in some embodiments, the machine learning-trained semantic routing model 300 may refer to only a portion of the model parts described herein. For example, a computing system may instantiate an embedding layer 302 to generate an intermediate representation 306, and then instantiate a decoder layer 318 to process the intermediate representation 306. In other examples, the computing system may provide a model input 304 to another computing system that has instantiated an embedding layer 302 for processing, and in response, receive an intermediate representation 306 from the other computing system.

[0084] The machine learning-based semantic routing model 300 may include one or more embedding layers 302. The embedding layer(s) 302 can process the model input 304 to generate an intermediate representation 306 of the model input 304. In some embodiments, the embedding layer(s) 302 may include a pre-trained language encoder 308A (e.g., the decoder part of the LLM). The pre-trained language encoder 308A can be trained to generate an intermediate representation of text content. For example, the model input 304 may include request information 310 and route feature information 312. The request information 310 may include text content describing a user request for a specific route, a specific route feature, specific mapping-related information, etc. The pre-trained language encoder 308A can process the text content to obtain an intermediate representation of the text content. The intermediate representation 306 may include, or be based on, the intermediate representation of the text content.

[0085] In some embodiments, the model input 304 may include previously generated routes or route segments for the user. For example, suppose the user provides feedback indicating that two routes are particularly highly rated. Route information indicating the previous routes may be included in the model input to provide the machine-trained semantic routing model 300 with a "few-shot" prompt. In other words, the previously rated routes highly by the user may serve as context for the routing request, prompting the machine-trained semantic routing model 300 to generate routes similar to those indicated by the model input 304.

[0086] In addition or alternatively, in some embodiments, the embedding layer(s) 302 may include a pre-trained audio encoder 308B (e.g., the encoder portion of a basic audio model). The pre-trained audio encoder 308B can be trained to generate an intermediate representation of the audio data. For example, request information 310 may include audio data containing a record of user-spoken utterances describing a user request, such as a specific route or specific route features. The pre-trained audio encoder 308B can process the audio data to obtain an intermediate representation of the audio data. The intermediate representation 306 may include, or be based on, the intermediate representation of the audio data.

[0087] Additionally or alternatively, in some embodiments, the embedding layer(s) 302 may include a pre-trained vision encoder 308C (e.g., the encoder portion of a base vision model). The pre-trained audio encoder 308C can be trained to generate an intermediate representation of image data (e.g., images, video data, etc.). For example, the request information 310 may include image data provided by a user describing a specific target location. The pre-trained vision encoder 308C can process the image data to obtain an intermediate representation of the image data. The intermediate representation 306 may include, or be based on, the intermediate representation of the image data.

[0088] Additionally or alternatively, in some embodiments, the embedding layer(s) 302 may include a mapping encoder 308D. The mapping encoder 308D can be at least partially trained with pre-training data, as described with respect to training data 202 in Figure 2A and training data 222 in Figure 2B. In particular, the mapping encoder 308D can be trained to generate intermediate representations of map information and / or route information. For example, request information 310 and / or route feature information 312 may include map information and / or route information. The mapping encoder 308D can process the map information and / or route information to obtain an intermediate representation of the map information and / or route information. The intermediate representation 306 may include, or be based on, the intermediate representation of map information and / or route information.

[0089] It should be noted that in some embodiments, encoders 308A to 308D may be encoder / decoder architectures, decoders, etc. More specifically, encoders 308A to 308D may refer to several types of machine-learned models or layers that can process the model input to generate an intermediate representation 306.

[0090] In some embodiments, the machine learning-prepared semantic routing model 300 may include a tokenizer 309, or otherwise have access to it. The tokenizer 309 may be a machine learning-prepared model, or part of a model, capable of generating a stream of tokens based on its inputs. In some embodiments, the tokenizer 309 may process the model input(s) 304 to obtain an intermediate representation 306, which may be tokens generated by the tokenizer(s) 309, or otherwise include them. Alternatively, in some embodiments, the tokenizer 309 may process the model input(s) 304 to generate a stream of tokens that can be processed by the embedding layer(s) 302, in order to obtain an intermediate representation 306. Alternatively, in some embodiments, the tokenizer 309 may process the intermediate representation 306 to generate a stream of tokens that can be processed by the attention layer(s) 314 and / or the graph layer(s) 316.

[0091] In some embodiments, the tokenizer 309 can generate multiple token streams, each specific to a particular type of information. For example, the tokenizer 309 may include an audio tokenizer, which processes the model input 304 to generate a stream of audio tokens (e.g., by discretizing raw audio into a series of tokens at a constant rate). In other examples, the tokenizer 309 may include a vision tokenizer, which processes the model input 304 to generate a stream of image tokens (e.g., by chunking an image using 16x16 chunk slices).

[0092] In some embodiments, the embedding layer(s) 302 may include a single encoder capable of processing tokens stacked from multiple modalities. For example, suppose the model input(s) 304 includes image data and audio data. The tokenizer 309 can process the model input(s) 304 to obtain an audio token stream and an image token stream. The audio token stream and the image token stream are stacked to generate an intermediate representation 306, which can be processed by a single encoder.

[0093] In some embodiments, the machine learning-prepared semantic routing model 300 may include an attention layer(s) 314. The attention layer(s) 314 can apply weights to specific parts of the input that indicate the relative importance of the parts of the input. For example, the attention layer(s) 314 may be a transformer-based self-attention mechanism, or otherwise include one. Additionally or alternatively, in some embodiments, the machine learning-prepared semantic routing model 300 may include a graph layer(s) 316. The graph layer(s) 316 may be a layer(s) of a graph neural network model. The graph layer(s) 316 may represent routes and segments of routes as a graph of nodes and edges, where nodes represent start and end locations, and edges represent route segments that navigate between the start and end locations. Furthermore, in some embodiments, the graph layer(s) 316 may include additional relationships between nodes in a journey involving multiple routes (e.g., from home to the airport, from the airport to a second airport, from the second airport to a hotel, etc.).

[0094] A machine learning-based semantic routing model 300 may include a decoder layer(s) 318. The decoder layer(s) 318 can process several intermediate outputs (e.g., intermediate representations 306, outputs from attention layers(s) 314, or graph layers(s) 316) to obtain a model output 320. The model output 320 may include routing information 322 and / or semantic mapping information 324. The routing information 322 may indicate routes and / or route segments. For example, a model input(s) 304 may indicate a request for an alternative route segment for a planned route segment. The routing information 322 may include information indicating a proposed route segment in response to the request.

[0095] Semantic mapping information 324 may include information describing a route, route segment, or specific route features. For example, model input 304 may indicate a request to describe a specific route. Semantic mapping information 324 may include text content describing the specific route in response to the request. In another example, model input 304 may indicate a request to identify the most popular businesses located along a route. Semantic mapping information 324 may include content indicating the most popular businesses on the specific route in response to that request (e.g., images of the businesses, text content describing the businesses, mapping and / or routing information associated with the businesses, user reviews corresponding to the businesses, etc.).

[0096] In some embodiments, the decoder layer(s) 318 may include a pre-trained language decoder 326A. For example, the pre-trained language decoder 326A may be a decoder in an LLM model that includes a pre-trained language encoder 308A. The pre-trained language decoder 326A can be trained to generate a language output based on several intermediate outputs (e.g., intermediate representation 306). For example, suppose request information 310 includes a request describing a particular route. The pre-trained language decoder 326A can process several intermediate outputs to generate text content describing the particular route indicated by the model input(s) 304. The text content may be contained in semantic mapping information 324.

[0097] In some embodiments, the decoder layer(s) 318 may include a pre-trained audio decoder 326B. For example, the pre-trained audio decoder 326B may be a decoder in an underlying audio model that includes a pre-trained audio encoder 308B. The pre-trained audio decoder 326B can be trained to generate an audio output based on several intermediate outputs (e.g., intermediate representation 306). For example, suppose request information 310 includes a request to generate audio describing a particular route. The pre-trained audio decoder 326B can process several intermediate outputs to generate audio describing a particular route indicated by the model input(s) 304. The audio may be contained in semantic mapping information 324.

[0098] In some embodiments, the decoder layer(s) 318 may include a pre-trained vision decoder 326C. For example, the pre-trained vision decoder 326C may be a decoder in an underlying vision model that includes a pre-trained vision encoder 308C. The pre-trained vision decoder 326C can be trained to generate an image output based on several intermediate outputs (e.g., intermediate representation 306). For example, suppose request information 310 includes a request to depict a particular route in a stylized manner (e.g., "Draw this route in the same style as the ski map"). The pre-trained vision decoder 326C can process several intermediate outputs to generate image data describing the particular route indicated by the model input(s) 304. The image data may be included in semantic mapping information 324.

[0099] In some embodiments, the decoder layer(s) 318 may include a mapping decoder 326D. For example, the mapping decoder 326D may be a decoder trained in conjunction with the mapping encoder 308D. The mapping decoder 326D can be trained to generate mapping and / or routing information based on several intermediate outputs (e.g., intermediate representation 306). For example, suppose request information 310 includes a request to generate an alternative route segment for a particular route segment of a route. The mapping decoder 326D can process several intermediate outputs to generate mapping and / or routing information that indicates a proposed route segment as an alternative to the route segment. The mapping and / or routing information may be included in routing information 322.

[0100] Additionally or alternatively, in some embodiments, the decoder layer(s) 318 may generate a multimodal model output 320. For example, suppose the request information 310 includes (a) text content describing a request to generate an alternative route segment for a particular route segment within a route, and (b) a request describing the alternative route segment and the difference between the alternative route segment and the particular route segment. The mapping decoder 326D may process several intermediate outputs to generate a proposed route segment as an alternative to the particular route segment. The pre-trained language decoder 326A may process several intermediate outputs to generate a text description of the proposed route segment. The proposed route segment may be contained in route information 322, and the text description may be contained in semantic mapping information 324. In this way, the decoder layer(s) may generate a multimodal output containing multiple types of data.

[0101] Exemplary Method Figure 4 shows a flowchart of an exemplary method 400 for training and fine-tuning a machine learning-trained semantic routing model according to several embodiments of the present disclosure. While Figure 4 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or arrangement shown. Various steps of method 400 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0102] In 402, the computing system can acquire training data for training (or “pre-training”) a machine learning-based semantic routing model. The training data may include route feature information. Route information may describe a route from a starting point to a destination. A route may include multiple route segments, including a subset of a first route segment (e.g., a set of “unmasked” segments) and a subset of a second route segment (a set of “masked” route segments). Route information may also include route feature information. Route feature information may describe one or more route features. A route feature may include at least one of the following: the type of route (e.g., scenic quality, efficiency, speed, safety, etc.), entities located along the route (e.g., businesses, residences, landmarks, POIs, etc.), average traffic conditions for the route or a segment of the route, average weather conditions for the route or a segment of the route, and information associated with users who have traveled the route.

[0103] In 404, the computing system may process at least a subset of the first route segments and a portion of the route feature information associated with the subset of the first route segments using a machine learning-based semantic routing model to obtain one or more predicted route segments for a subset of the second route segments. More specifically, the subset of the first route segments may be unmasked route segments. The subset of the second route segments may be route segments that are masked or otherwise not processed by the machine learning-based semantic routing model so that the model can predict the segments.

[0104] In some embodiments, to process a subset of a first route segment, the computing system may process at least the subset of the first route segment and a portion of the route feature information associated with the subset of the first route segment using a first part of a machine learning-based semantic routing model to obtain a latent representation of the subset of the first route segment. In some embodiments, the first part of the machine learning-based semantic routing model may be, or otherwise include, the encoder portion of a pre-trained LLM. For example, the route information may represent a second route from a second starting location to a second destination location, and prompts indicating a request to describe the second route. The model output may include text content describing the second route.

[0105] In 406, the computing system can train and / or fine-tune a machine learning-based semantic routing model based on an optimization function that evaluates the difference between one or more predicted route segments and a subset of second route segments.

[0106] In some embodiments, a computing system can process input data using a machine-learned semantic routing model to obtain a model output. The input data may include route information indicating one or more first route segments of an incomplete route, and prompts indicating a request to generate one or more second route segments with requested route features for the incomplete route. The model output may include one or more second route segments with the requested route features.

[0107] In some embodiments, route information may include one or more exemplary routes, a second route comprising a plurality of second route segments, and prompts indicating a request to generate one or more alternative route segments for one or more second route segments of the second route based on the one or more exemplary routes. Model output may include one or more alternative route segments of the second route.

[0108] In some embodiments, a computing system can process input data using a machine learning-prepared semantic routing model to obtain model output. The input data may include route information describing a second route that includes multiple second route segments. The model output may include classification information classifying the route as a first route type among multiple route types (e.g., scenic route type, efficient route type, non-highway route type, toll road route type, low-traffic route type, etc.). In some embodiments, the classification information further classifies the second route segments among multiple second route segments as a first route segment type among multiple route segment types.

[0109] In some embodiments, a computing system can process input data using a machine learning-prepared semantic routing model to obtain model output. The input data may include the current route segment, which contains intermediate destinations of existing routes. The output data may include one or more candidate route segments. The starting location of each of the one or more candidate route segments may correspond to an intermediate destination of the current route segment.

[0110] In some embodiments, a computing system can process input data using a machine-learned semantic routing model to obtain model output. The input data may include route request information describing a route from a requested starting point to a requested destination, and preferred route feature information describing preferred route features for the route. The model output may include route information showing a route from the requested starting point to the requested destination, including the preferred route features.

[0111] In some embodiments, a computing system can process input data using a machine learning-prepared semantic routing model to obtain model output. The input data may include one or more images of one or more locations. The output data may include itinerary information showing a proposed route that includes at least one of the one or more locations.

[0112] Figure 5 shows a flowchart of an exemplary method 500 for performing map-related tasks using a machine learning-prepared semantic routing model, according to several embodiments of the present disclosure. While Figure 5 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or arrangement shown. Various steps of method 500 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0113] In 502, the computing system may obtain one or more inputs from a client computing device to a machine-trained semantic routing model. The machine-trained semantic routing model may be a model trained to process mapping information to produce a model output that includes a proposed route segment and / or information associated with the route segment. One or more inputs may include at least one of request information indicating a request for the requested route segment and / or mapping-related information, or route feature information indicating one or more route features.

[0114] In 504, the computing system may process one or more inputs to obtain a model output. The model output may include at least one of the following: (a) routing information indicating a route containing one or more proposed route segments, or (b) semantic mapping information associated with the route. In particular, the routing information may be data for a mapping application, or information describing a particular route or route segment, or otherwise may include such information. For example, the routing information may be data provided to an application programming interface (API) for a mapping application, which then causes the application to generate a route based on the model output or modify an existing route. In other examples, the routing information may be text content describing a route or route segment, or a modification of a route. In other examples, the routing information may be an image depicting a route.

[0115] Semantic mapping information can generally refer to information describing several aspects or features of routes or mapping-related entities included in a map, such as businesses, landmarks, houses, points of interest (POIs), etc. Semantic mapping information can include text content that responds to queries. For example, model inputs may include queries related to a particular route or entity located on the map. Semantic mapping information can also include general descriptions of routes or entities located on the map. For example, semantic mapping information may include a semantic description of a particular route (e.g., "This route is a scenic coastal route with a low probability of encountering either traffic congestion or bad weather conditions"). In another example, semantic mapping information may include a semantic description of a route segment for a particular road or route segment (e.g., "The road on which this business is located is visited by business travelers almost every day during normal working hours"). In yet another example, semantic mapping information may be, or otherwise include, a classification output that categorizes a particular route or segment (e.g., traffic volume: low, collision probability: low, probability of bad weather: low, etc.).

[0116] In some embodiments, in use cases where a machine learning-prepared semantic routing model is used to predict route segments or suggest alternative route segments, the computing system can restrict such predictions or suggestions to known route segments that have been verified to exist. In other words, the model output can select known route segments rather than attempting to generate route segments that may or may not be passable. Thus, the computing system can ensure that the route segments provided to the user are passable.

[0117] In 506, the computing system may provide model output to a client computing device. A client computing device may generally refer to a device that receives routing or mapping-related services from a computing system, such as a user computing device (e.g., a smartphone, laptop, wearable device, etc.) or a second computing system (e.g., a virtualization computing device, compute node, or network node, etc.). In some embodiments, the computing system may provide model output in response to receiving model input. For example, the computing system may receive a routing or mapping-related request from a client computing device that includes model input. In response, the computing system may provide the client computing device with model output, or provide information describing the model output or information based on the model output.

[0118] Additionally or alternatively, in some embodiments, the computing system can iteratively provide model outputs to client computing devices. For example, over several iterations, the computing system may receive model inputs that include a request to classify the type of root segment associated with a particular root segment. After classifying a certain number of root segments, the computing system can update the client computing devices by providing information indicating the classified root segments. More generally, if the model output generated by the computing system is applicable to multiple client computing devices (e.g., classifying root segments, classifying entities, etc.), the computing system can update some (or all) of the client devices based on the model output. For example, the computing system can push such information to the client devices by updating a mapping application associated with the computing system.

[0119] In addition or alternatively, in some embodiments, the computing system may perform other actions on the model output. For example, if the model output classifies a particular route segment or entities present in the mapping data (e.g., businesses, houses, landmarks, points of interest, etc.), the computing system may store the classifications in a data store that stores similar classifications to enable mapping services.

[0120] In some embodiments, obtaining one or more inputs to a machine-learned semantic routing model may include obtaining from a client computing device exemplary route information indicating one or more exemplary routes and a second route comprising a plurality of second route segments, and prompts indicating a request to generate one or more alternative route segments for one or more each of the second route segments of the second route based on one or more exemplary routes. Processing one or more inputs to obtain a model output may include processing exemplary route information and prompts to obtain a model output. The model output may include routing information indicating one or more alternative route segments for one or more each of the one or more second route segments.

[0121] In some embodiments, obtaining one or more inputs to a machine-learned semantic routing model may include obtaining information from a client computing device that indicates the current route segment, including intermediate destinations. Processing one or more inputs to obtain a model output may include processing information that indicates the current route segment to obtain a model output. The model output may include one or more candidate route segments. The starting location of each of the one or more candidate route segments may correspond to an intermediate destination of the current route segment.

[0122] In some embodiments, obtaining one or more inputs to a machine-learned semantic routing model may include obtaining from a client computing device (a) route request information describing a route from a requested starting location to a requested destination, and (b) preferred route feature information describing preferred route features of the route. Processing one or more inputs to obtain a model output may include processing information indicating the route request information and the preferred route feature information in order to obtain a model output. The model output may include information indicating a route from a requested starting location to a requested destination, including the preferred route features.

[0123] In some embodiments, obtaining one or more inputs to a machine-learned semantic routing model may include obtaining information from a client computing device that indicates one or more requested route segments. This information may include one or more images of one or more locations. Processing one or more inputs to obtain a model output may include processing information that indicates the current route segment to obtain a model output. The model output may include journey information that indicates a proposed route that includes at least one of one or more locations.

[0124] Additional disclosures The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed on such systems and information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of feasible configurations, combinations, and divisions of tasks and functions across multiple components and across groups of components. For example, the processes described herein can be performed using a single device or component, or multiple devices or components working together. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0125] While the subject matter of this disclosure has been described in detail with respect to various specific exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art will readily be able to create modifications, variations, and equivalents to such embodiments, having understood the foregoing. Therefore, this disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter of this disclosure, which will be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used in conjunction with other embodiments to give rise to further embodiments. Therefore, this disclosure is intended to include such modifications, variations, and equivalents.

Claims

1. A method implemented in a computer, The acquisition of training data by a computing system comprising one or more computing devices, wherein the training data is (a) Route information indicating a route from a starting place to a destination place, wherein the route includes a plurality of route segments, each including a subset of a first route segment and a subset of a second route segment, (b) Acquisition of route feature information that describes one or more route features, In order to obtain one or more predicted route segments for a subset of the second route segment, the computing system processes at least a subset of the first route segment and a portion of the route feature information associated with the subset of the first route segment using a machine learning-based semantic routing model. The computing system adjusts one or more parameters of the machine-learned semantic routing model based on an optimization function that evaluates the difference between one or more predicted route segments and a subset of the second route segments. A method of implementation in a computer, including

2. Processing at least a subset of the first route segments and a portion of the route feature information associated with the subset of the first route segments using the machine learning-trained semantic routing model is: To obtain a latent representation of the subset of the first route segment, the computing system processes at least the subset of the first route segment and the portion of the route feature information associated with the subset of the first route segment using a first part of the machine learning-based semantic routing model. A method implemented in a computer according to claim 1, including the method described in claim 1.

3. The computer implementation method according to claim 2, wherein the first portion of the machine learning-prepared semantic routing model includes an encoder and / or decoder of a pre-trained large language model (LLM).

4. The above method further, To obtain model output, the computing system processes the input data using the machine learning-prepared semantic routing model, and the input data is Route information showing a second route from a second starting point to a second destination, A prompt indicating a request to describe the second route, The computer-implemented method according to claim 3, wherein the model output includes text content describing the second route.

5. The above method further, A computer-implemented method according to claim 3, comprising processing input data using the machine-learned semantic routing model by the computing system in order to obtain a model output, wherein the input data includes text content describing a requested route, and the model output includes route information indicating the requested route.

6. Before processing the input data using the machine-trained semantic routing model to obtain the model output, the method: To obtain a text description of the training route, the computing system processes second training data, including the training route, using the machine learning-prepared semantic routing model. The computing system adjusts one or more parameters of the machine-learned semantic routing model based on a loss function that evaluates the text description of the training route and the corresponding ground truth text description of the training route. A method implemented in a computer according to claim 5, including the following:

7. Processing at least a subset of the first route segments and the portion of the route feature information associated with the subset of the first route segments using the first part of the machine learning-trained semantic routing model further means that To obtain a graph output, the computing system includes processing the latent representation using the graph-based portion of the machine learning-prepared semantic routing model, and the graph output is A plurality of nodes representing the starting location, the destination location, and intermediate locations between the starting location and the destination location, Multiple edges representing multiple root segments, A method implemented in a computer according to claim 2, including the above.

8. The above method further, To obtain model output, the computing system processes the input data using the machine learning-prepared semantic routing model, and the input data is Route information indicating one or more first route segments of an incomplete route, Includes a prompt indicating a request to generate one or more second route segments with the requested route features for the incomplete route, The computer-implemented method according to claim 1, wherein the model output includes the one or more second route segments having the requested route features.

9. The above method further, To obtain model output, the computing system processes the input data using the machine learning-prepared semantic routing model, and the input data is Route information indicating one or more exemplary routes and a second route including multiple second route segments, Includes a prompt indicating a request to generate one or more alternative route segments for one or more second route segments of the second route based on the one or more exemplary routes, The computer-implemented method according to claim 1, wherein the model output includes one or more alternative route segments of the second route.

10. The above method further, A computer-implemented method according to claim 1, comprising processing input data using the machine-learned semantic routing model by the computing system in order to obtain a model output, wherein the input data includes route information describing a second route comprising a plurality of second route segments, and the model output includes classification information classifying the route as a first route type among a plurality of route types.

11. The method implemented in a computer according to claim 10, wherein the classification information further classifies one of the plurality of second root segments as a first root segment type among a plurality of root segment types.

12. The above method further, To obtain model output, the computing system includes processing the input data using the machine-learned semantic routing model, wherein the input data includes the current route segment, which includes an intermediate destination of an existing route. The method implemented in a computer according to claim 1, wherein the model output includes one or more candidate route segments, and the starting location of each of the one or more candidate route segments corresponds to the intermediate destination location of the current route segment.

13. The above method further, To obtain model output, the computing system processes the input data using the machine learning-prepared semantic routing model, and the input data is Route request information that describes the route from the requested starting location to the requested destination location, Preferred route feature information describing the preferred route features of the aforementioned route, Includes, The computer-implemented method according to claim 12, wherein the model output includes route information indicating a route from the requested starting location to the requested destination location including the preferred route features.

14. The above method further, To obtain model output, the computing system processes the input data using the machine-learned semantic routing model, wherein the input data includes one or more images of one or more locations. The computer-implemented method according to claim 1, wherein the model output includes itinerary information indicating a proposed route that includes at least one of the one or more locations.

15. A computing system, One or more processor devices, The memory comprises, A machine learning-trained semantic routing model, wherein the machine learning-trained semantic routing model is trained to process mapping information to generate a model output that includes proposed route segments and / or information associated with route segments, The system comprises one or more tangible, non-temporary computer-readable media that store computer-readable instructions that cause the one or more processors to perform an operation, and the operation is, Obtaining one or more inputs from a client computing device to the machine learning-trained semantic routing model, wherein the one or more inputs are: Request information indicating the request for the requested route segment and / or mapping-related information, or Route feature information that shows one or more route features, To obtain, which includes at least one of the following: Processing one or more inputs in order to obtain a model output, wherein the model output is (a) Routing information indicating a route that includes one or more proposed route segments, (b) Semantic mapping information associated with the route, Processing that includes at least one of the following, The model output is provided to the client computing device, A computing system that includes this.

16. Obtaining the one or more inputs to the machine-trained semantic routing model is The process includes obtaining from the client computing device exemplary route information indicating a second route comprising one or more exemplary routes and a plurality of second route segments, and prompts indicating a request to generate one or more alternative route segments for one or more second route segments of the second route based on the one or more exemplary routes, Processing one or more inputs in order to obtain the model output is, The computing system according to claim 15, comprising processing the exemplary route information and the prompts in order to obtain the model output, wherein the model output includes routing information indicating the one or more alternative route segments for each of the one or more second route segments.

17. Obtaining the one or more inputs to the machine-trained semantic routing model is This includes obtaining information from the client computing device indicating the current route segment, including an intermediate destination location. Processing one or more inputs in order to obtain the model output is, The computing system according to claim 15, comprising processing the information indicating the current route segment in order to obtain the model output, wherein the model output includes routing information indicating one or more candidate route segments, and the starting location of each of the one or more candidate route segments corresponds to the intermediate destination location of the current route segment.

18. Obtaining the one or more inputs to the machine-trained semantic routing model is From the aforementioned client computing device, Route request information that describes the route from the requested starting location to the requested destination location, This includes obtaining preferred route feature information that describes the preferred route features of the aforementioned route, Processing one or more inputs in order to obtain the model output is, The computing system according to claim 15, comprising processing information indicating the route request information and the preferred route feature information in order to obtain the model output, wherein the model output includes routing information indicating a route from the requested starting location to the requested destination location including the preferred route features.

19. Obtaining the one or more inputs to the machine-trained semantic routing model is This includes obtaining information from the client computing device indicating the requested route segment, the information including one or more images of one or more locations, Processing one or more inputs in order to obtain the model output is, The computing system according to claim 15, comprising processing one or more images of one or more locations in order to obtain the model output, wherein the model output includes routing information indicating a proposed route that includes at least one of the one or more locations.

20. One or more tangible, non-temporary, computer-readable media that store computer-readable instructions that cause the one or more processors to perform an operation, wherein the operation is The acquisition of training data, said training data is (a) Route information indicating a route from a starting place to a destination place, wherein the route includes a plurality of route segments, each including a subset of a first route segment and a subset of a second route segment, (b) Acquisition of route feature information that describes one or more route features, To obtain one or more predicted route segments for a subset of the second route segment, at least a subset of the first route segment and a portion of the route feature information associated with the subset of the first route segment are processed using a machine learning-based semantic routing model. Adjusting one or more parameters of the machine learning-trained semantic routing model based on an optimization function that evaluates the difference between the one or more predicted route segments and a subset of the second route segments, Non-temporary computer-readable media, including [specific examples of such media].

Images