Artificial intelligence-based system using large language models supplemented with tool calls for computer based travel planning

Abstract

In machine learning, the content generated by a large language model (LLM) is limited to its knowledge learned via training. When an LLM is being utilized to perform complicated tasks that require research and / or reasoning, the result is often the LLM hallucinating or generating inadequate content. There is instead disclosed herein artificial intelligence (AI) including one or more LLMs working in combination with tools. The content generated by an LLM can include a tool call that executes a function performed by the tool. The function may return information for the context window of the LLM and is thereby utilized by the LLM to generate an improved response. By supplementing one or more LLMs with specific functions, the LLM may be improved compared to how LLMs conventionally operate. The improved AI can be integrated into existing technology for travel planning to enhance that existing technology.

Description

ARTIFICIAL INTELLIGENCE-BASED SYSTEM USING LARGE LANGUAGE MODELS SUPPLEMENTED WITH TOOL CALLS FOR COMPUTER BASED TRAVEL PLANNINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application claims priority to U.S. Provisional Patent Application Serial No. 63 / 684,325, which was filed on August 16, 2024, and which is incorporated herein by reference.FIELD

[0002] The following relates to applications of large language models (LLMs).BACKGROUND

[0003] In machine learning, a large language model (LLM) generates content in response to a prompt. The content may be, for example, computer-readable code such as JavaScript Object Notation (JSON). An LLM processes tokens as input and generates tokens as output. A token represents a fundamental unit of text that the LLM can process, e.g. a token my represent a word or part of a word. The input to the LLM is tokenized, and the output from the LLM is based on tokens generated by the LLM. An LLM may utilize a large neural network to determine probabilities for a next token of an output sequence conditional on previous or historical tokens in the output sequence.

[0004] The content generated by an LLM is limited to its inherent knowledge base learned via training. When an LLM is being utilized to perform complicated tasks that require research and / or reasoning, the result may be the LLM hallucinating or generating inadequate content. This limits the ability to integrate LLMs into existing technology that is utilized for various applications.SUMMARY

[0005] For example, consider the existing technology utilized for travel planning. A web browser controlled by a user may perform Internet searches for web pages relating to destinations, hotels, flights, etc. A map application may be accessed by the web browser to view a map of a location. Online platforms may be accessed to book travel services. The user operating the web browser performs the steps. To try to improve the technology, one or more LLMs may be integrated into the process, e.g. to receive prompts from users and generate content responsive to the prompts.However, the content generated may be hallucinated or inadequate due to the limitation of theLLM’s knowledge base learned through training. For example, the LLM might satisfactorily generate a reply to a general prompt asking “What are fun attractions in New York?’", but the LLM itself would fail at research having regard to the user’s travel details and composing a useful graphical user interface (GUI) for presentation to the user, such as a GUI showing pictures of an attraction, information from other travelers, with possibly a map showing the location of the attraction, etc.

[0006] There is instead disclosed herein trained artificial intelligence (Al) including one or more LLMs working in combination with tools accessed by the one or more LLMs. A tool is a computing resource (e.g. software and / or hardware resource) that can perform a function that an LLM itself might not be able to perform, e.g. due to the limitations and / or limited training of the LLM. An LLM can work in combination with such functions using tool calls. For example, content generated by the LLM can include a tool call that causes execution of the function performed by the tool. The tool call generated by the LLM may, for example, be sent to the tool via an application programming interface (API) to cause execution of the function performed by the tool. The LLM may generate arguments that are passed into the function executed by the tool. The function may return information that then forms part of the context window of the LLM and is thereby utilized by the LLM to generate a more accurate and complete response and to mitigate problems such as hallucination or inadequate LLM output.

[0007] For example, an LLM may call tools that execute functions such as internet searching using a search string generated by the LLM, or generating and formatting details relating to a trip destination and / or travel dates and / or travelers, etc. In some cases, the function itself might be implemented using trained Al such as another LLM, e.g. to generate content that is output by the function.

[0008] By supplementing one or more LLMs with specific functions accessed by the one or more LLMs via tool calls, the LLM function may be improved compared to how LLMs conventionally operate. The improved Al comprising the LLMs and functions working together can be integrated into existing technology to enhance the existing technology for travel planning.

[0009] In one aspect, there is provided, a computer-implemented method for travel planning using a voice and / or text enabled Al system. The method may include invoking at least one function combined with prompting at least one trained Al, including a large language model (LLM) running on specialized array processing hardware, to (i) produce queries for trip details and traveler details to be received from a user and (ii) distill, from user responses, the trip details and traveler details, which may be used to populate a travel profile data structure. The method may further includeprompting the at least one trained Al to perform an Internet search based on the trip details and traveler details, which may include searching travel blogs and travel web sites. The Internet search may further includes focusing the search to travel content to improve relevance of the search results and return a decreased quantity of the search results. The method may further include automatically parsing information obtained from the Internet search to generate travel plan information.

[0010] In some implementations, the LLM calls the function with arguments generated by the LLM, thereby causing execution of the function with the arguments generated by the LLM. In some implementations, results of the function are then used by the trained AL

[0011] In some implementations, the Internet search may further include search engine filtering in real time from at least thousands of tourist posted itineraries that overlap with a travel destination in the travel profile data structure, including comparing at least (i) the trip details and travel details of the travel profile data structure with (ii) trip details and travel details of posted itineraries. In some implementations, the method may further include rank-ordering at least some of the posted itineraries for presentation to the user.

[0012] In some implementations, the method may include presenting, via a user interface, the posted itineraries including a visual display including images and interactive links representing three or more of the posted itineraries and a spoken summary of the posted itineraries represented on the visual display.

[0013] In some implementations, the method may include receiving, from the user, an interaction with a visual image of the presented visual images and in response to the received interaction, prompting the trained Al to respond with a structured travel plan based on the selected visual image.

[0014] In some implementations, the method may include, in addition to the Internet search, searching a proprietary data set including scraped and curated travel blogs, web sites, and / or tourist-created itineraries.

[0015] In some implementations, the trained Al may autonomously trigger one or more specific user interface elements for display on a visual display based on one or more of a user input, a current stage of travel planning, and a user preference.

[0016] In some implementations, the method may further include receiving a plurality of inputs from multiple users and updating a shared itinerary based on the received plurality of inputs with real-time synchronization. In some implementations, the trained Al may manage permissions androles corresponding to each user of the multiple users, thereby enabling user-specific levels of control over the shared itinerary.

[0017] In some implementations, the method may further include receiving, from the user, a special inquiry during travel planning and, in response to the special inquiry, autonomously initiating at least one voice call between the trained Al and an external entity to obtain information related to the special inquiry. In some implementations, the method may further include providing the user with a real-time transcript of the at least one voice call and / or enabling the user to begin participating in the at least one voice call.

[0018] In some implementations, the trained Al may evaluate and rank itineraries based on one or more user-defined criteria including a budget, a preferred activity, and a travel time.

[0019] In another aspect, there is provided a voice and / or text enabled Al system accessing a large language model (LLM) running on one or more processors and memory accessible by the processors, the memory loaded with computer instructions for travel planning, which computer instructions, when executed on the processors, implement actions. The implemented actions may include invoking at least one function combined with prompting at least one trained Al, including the LLM running on specialized array processing hardware, to (i) produce queries for trip details and traveler details to be received from a user and (ii) distill, from user responses, the trip details and traveler details, which may be used to populate a travel profile data structure. The implemented actions may further include prompting the at least one trained Al to perform an Internet search based on the trip details and traveler details, which may include searching travel blogs and travel web sites. The Internet search may further include focusing the search to travel content to improve relevance of the search results and return a decreased quantity of the search results. The implemented actions may further include automatically parsing information obtained from the Internet search to generate travel plan information.

[0020] In some implementations, the LLM calls the function with arguments generated by the LLM, thereby causing execution of the function with the arguments generated by the LLM. In some implementations, the results of the function are then used by the trained Al.

[0021] In some implementations, the implemented actions may further include: search engine filtering in real time from at least thousands of tourist posted itineraries that overlap with a travel destination in the travel profile data structure, including comparing at least (i) the trip details and travel details of the travel profile data structure with (ii) trip details and travel details of posted itineraries; and rank-ordering at least some of the posted itineraries for presentation to the user.

[0022] In some implementations, the implemented actions may further include presenting, via a user interface, the posted itineraries including a visual display including images and interactive links representing two or more of the posted itineraries and a spoken summary of the posted itineraries represented on the visual display.

[0023] In some implementations, the implemented actions may further include receiving, from the user, an interaction with a visual image of the presented visual images and in response to the received interaction, prompting the trained Al to respond with a structured travel plan based on the selected visual image.

[0024] In some implementations, the trained Al may autonomously trigger one or more specific user interface elements for display on a visual display based on one or more of a user input, a current stage of travel planning, and a user preference.

[0025] In some implementations, the implemented actions may further include generating photorealistic images of a traveler and their companions at a selected trip destination before travel occurs. In some implementations, the system may use generative Al to create visual representations based on photos of the traveler and companions, combined with destination- specific scenery.

[0026] In some implementations, the generative Al may incorporate environmental factors such as weather, time of day, and / or local landmarks into the images to provide an immersive preview of the trip experience.

[0027] In some implementations, the implemented actions may further include providing travel support during a currently occurring trip, including the trained Al processing, as input, one or more of a user input, a user profile, and / or a current travel condition, to generate, as output, a recommendation, contextual information, or logistics information.

[0028] In another aspect, there is provided a non-transitory computer readable storage medium impressed with computer program instructions for Al-based travel planning, which computer program instructions when executed implement a method. The method may include invoking at least one function combined with prompting at least one trained Al, including a large language model (LLM) running on specialized array processing hardware, to (i) produce queries for trip details and traveler details to be received from a user and (ii) distill, from user responses, the trip details and traveler details needed to populate a travel profile data structure. The method may further include prompting the at least one trained Al to perform an Internet search based on the trip details and traveler details, including searching travel blogs and travel web sites. The Internet search may further include focusing the search to tourist-posted itineraries to improve relevance ofthe search results and return a decreased quantity of the search results. The method may further include automatically parsing information obtained from the Internet search to generate travel plan information.

[0029] In some implementations, the method may further include receiving, from the user, a selection of a posted itinerary presented to the user, and causing the trained Al to respond with a structured trip plan based on the selected itinerary.

[0030] In some implementations, the method may further include allowing the user to clone and modify the structured trip plan.

[0031] In another aspect, a system is provided that is configured to perform any of the methods disclosed herein. For example, the system may include at least one processor to directly perform (or control / instruct the system to perform) the method steps. In some implementations, the system includes at least one processor and a memory storing processor-executable instructions that, when executed by the at least one processor, cause the system to perform any of the methods described herein.

[0032] In another aspect, there is provided a non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform any of the methods disclosed herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Embodiments will be described, by way of example only, with reference to the accompanying figures wherein:

[0034] FIG. 1 illustrates an example system having a user device, a trained Al system, and an intermediary computing system;

[0035] FIG. 2 illustrates an example large language model (LLM);

[0036] FIGs. 3 and 4 illustrate examples of the LLM working in combination with a function; and

[0037] FIGs. 5 to 19 illustrate example graphical user interfaces (GUIs) that may be generated by trained Al in relation to travel planning.DETAILED DESCRIPTION

[0038] For illustrative purposes, specific embodiments will now be explained in greater detail below in conjunction with the figures.

[0039] FIG. 1 illustrates an example system having a user device 102, a trained Al system 192, and an intermediary computing system 152 that executes an application for providing travel planning. The system is improved over conventional computer technology for travel planning by the integration of one or more LLMs working in combination with functions, in the manner explained herein.

[0040] The user device 102 is a device used by a user for interacting with computing system 152 for the purposes of travel planning. The user device 102 may be, for example, a personal computer, or laptop, or desktop computer, or mobile device such as a tablet or smartphone, etc., depending upon the implementation. The user device 102 includes a processor 104, memory 106, user interface 108, and network interface 112. The processor 104 controls the operations of the user device 102. The processor 104 may be implemented by one or more processors that execute instructions stored in the memory 106. Alternatively, some or all of the processor 104 may be implemented using dedicated circuitry, such as an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or a field programmable gate array (FPGA). The memory 106 stores information, such as information provided by computing system 152 to be presented to a user via user interface 108. The user interface 108 allows the user (e.g. a human) to provide input to and receive output from the user device 102. For example, the user interface 108 may include a display 110 (which may be a touch screen), and / or a keyboard, and / or a mouse, etc. In one example, the user interface 108 includes display 110 that displays information provided by computing system 152, e.g. it displays a screen page, such as web page, having content and / or a layout provided by computing system 152 that was instructed by one or more LLMs of trained Al system 192. The network interface 112 interfaces with a network (not shown) to perform communication over the network with computing system 152. The structure of the network interface 112 will depend on how the user device 102 interfaces with the network. For example, if the user device 102 is a smartphone or laptop, the network interface 112 may comprise a transmi tter / recei ver with an antenna to send and receive wireless transmissions over the network. If the user device 102 is a personal computer connected to the network with a network cable, the network interface 112 may comprise a network interface card (NIC), and / or a computer port (e.g. a physical outlet to which a plug or cable connects), and / or a network socket, etc. In one implementation, the network is the Internet and the network interface 1 12 communicates over the Internet by transmitting and receiving network packets.

[0041] The computing system 152 implements a travel planning application using trained Al system 192. The computing system 152 may be implemented on one or more servers. Thecomponents of the computing system 152 may be distributed. Included in computing system 152 is at least one processor 154, a memory 156, and a network interface 162. The processor 154 controls the operations of the computing system 152 and executes computer-executable instructions to implement the travel planning application. The memory 156 stores the computer-executable instructions and other information, e.g. information received from trained Al system 192. Therefore, the processor 154 may be implemented by one or more processors that execute instructions stored in the memory 156. Alternatively, some or all of the processor 154 may be implemented using dedicated circuitry, such as an ASIC, a GPU, or a FPGA. The network interface 162 performs wired and / or wireless communications with external systems and / or networks. For example, the network interface 162 enables the computing system 152 to communicate with trained Al system 192, e.g. via API 164, such as by sending prompts 172 to the trained Al system 192 and receiving, from the trained Al system 192, responses 174 to the prompts 172. The network interface 162 also communicates with user device 102 over a network, e.g. over the Internet by transmitting and receiving network packets.

[0042] The trained Al system 192 implements at least one LLM. In general, N LLMs are illustrated and may be included as part of trained Al system 192, where N is an integer greater than or equal to one. Each of the prompts 172 is received by one or more of the LLMs and the one or more LLMs generate responses 174 responsive to the prompts.

[0043] FIG. 2 illustrates an example of LLM k (where 1 < k < N) utilized by the trained Al system 192. LLM k is implemented on specialized array processing hardware, e.g. such as a dedicated artificial intelligence processor unit, a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), a hardware accelerator, or combinations thereof. Each LLM in trained Al system 192 may have the general structure illustrated in FIG. 2. The specific architecture of LLM k in FIG. 2 is implementation specific and depends upon the architecture employed. However, in general the LLM receives input including both user input 208 originating from computing system 152 (and possibly ultimately originating from user input at user device 102) and system prompt 210 information. The user input 208 may be (or be based on) one or more of the prompts 172. The system prompt 210 may be information that supplements the user input 208, such as instructions, rules, and / or examples for the LLM. The system prompt 210 may include a description of one or more functions 158 discussed below that may be called by the LLM, including instructions on how to call each function and how to structure arguments passed to each function. The input (including the user input 208 and system prompt 210) forms the basis of input tokens 212 ingested by the LLM. The input to the LLM may be the input tokens 212 themselves.Alternatively, processing may be performed to transform the input of the LLM into input tokens212. For example, the user input 208 may be tokenized to obtain input tokens 212.

[0044] The input tokens 212 are ingested by one or more neural networks 214 of the LLM. In some implementations, the input tokens 212 are first embedded. A final layer 216 of the final neural network outputs a sequence of output tokens 218. Each next token 220 is generated conditional on tokens previously generated within the context window of the LLM. For example, the final layer 216 may produce a plurality of values in the form an array or a tensor, e.g. a tensor of logit values. Each value may represent the probability that the token corresponding to that value is the next token 220. The next token 220 may be generated having regard to the probabilities. The output data of the generative ML model, referred to as output 222, is based on the output tokens 218. The output 222 may be the output tokens 218, or the output tokens 218 may be mapped or converted / transformed into other data that forms the output 222, e.g. to one or more characters, or a numerical representation of characters or text, such as American standard code for information interchange (ASCII) or Unicode characters, etc. The output 222 may form the basis of one or more responses 174 provided by the trained Al system 192.

[0045] Returning to FIG. 1, the trained Al system 192 does not just include N LLMs working in isolation, but instead the N LLMs are enhanced with one or more functions 158 which are accessed via tool calls. In FIG. 1, the functions 158 are shown as part of computing system 152, e.g. the functions 158 may be implemented on the computing system 152. However, more generally this need not be the case, e.g. the functions 158 may be implemented on a separate server. In any case, the functions are considered part of the tools of the trained Al system 192 in that the LLMs of the trained Al system 192 can cause execution of the functions 158 and provide arguments to the functions 158. An LLM can execute a function by performing a tool call 182, which may include arguments that the LLM generated to be passed into the function. The LLM may receive an output of the function in the form of a tool reply 184, e.g. which may provide context to the LLM so that the LLM can enhance the output generated by the LLM. The functions 158 may alternatively be called callback functions. However, more generally, whenever “callback function” is used herein, it may be replaced with just the word “function”.

[0046] FIGs. 3 and 4 illustrate two examples of an LLM k of the trained Al system 192 working in combination with execution of a function, including controlling execution of that function and passing arguments to the function. In FIG. 3, at 304 the LLM is prompted with input, e.g. which may ultimately originate from input provided by a user through user device 102. The input is a user query “I would like to group travel somewhere inexpensive and adventurous”. The input istokenized to form the basis of input tokens ingested by the LLM. The input tokens are processed by neural network 214, which generates output tokens forming the basis of output 222. The output 222 generated by the LLM is indicative of a decision by the LLM to perform a tool call to execute the function “captureTravelDetails”. The output includes “Call captureTravelDetails”, as well as the arguments passed to the function: “[(travel with others, low budget); (adventure)]”, where the phrases “travel with others”, “low budget”, and “adventure” may be generated by the LLM based on information in the prompt 304 and in the LLM’s context window. The tool call is made at 306 to execute the “captuerTravelDetails” function 308. The function 308 executes to create and store update travel profile data (shown at 310) and also return the updated travel profile data to the LLM (shown at 312) so that this updated information can be provided as additional context to the LLM, e.g. tokenized and stored in the context window of the LLM. This additional context enhances the ability of the LLM to produce a more accurate and complete reply to the user. Note that the updated travel profile data at 312 might instead be sent to another LLM.

[0047] In the example of FIG. 4, the LLM is prompted to perform an Internet search using trip and / or traveler details. The prompt may be explicit, or it might be implicit, e.g. the user input is of a nature such that the LLM needs to perform an Internet search to provide a response. The input tokens are processed by neural network 214, which generates output tokens forming the basis of output 222. The output 222 generated by the LLM is indicative of a decision by the LLM to perform a tool call that results in execution of an Internet searching function. The output 222 includes the call “Call search engine”, as well as the argument passed, which is a search term generated by the LLM: “active guided trip with hotel accommodation, outdoor adventure, budgetfriendly, wildlife safari”. The search term generated by the LLM may be based on information in the LLM’s context window. The call is made at 404 and the search engine function 406 searches the Internet 408 using the search term. The search may be lexical or semantic. In another implementation, the search may be performed by web scaping and the search engine function 406 might be or include a web scraping tool. The search results 410 are returned to the LLM as additional context for the LLM, e.g. tokenized and stored in the context window of the LLM. This additional context enhances the ability of the LLM to produce a more accurate and complete reply to the user. Note that the updated search results 410 might instead be sent to another LLM.

[0048] By supplementing the LLM with specific functions accessed by the LLM via tool calls, like as shown in FIGs. 3 and 4, the LLM function may be improved compared to how LLMs conventionally operate. The improved Al comprising the LLMs and functions working together areintegrated into existing technology to enhance the application of that existing technology for travel planning.

[0049] The application of an Al-based system with LLMs and functions, as described above, to travel planning will now be explained in more detail.

[0050] This disclosure covers embodiments of an artificial intelligence-based system and methods for travel planning, and in some examples, a system and methods for optimizing travel itineraries using a voice and / or text enabled artificial intelligence system. The disclosed voice-based artificial intelligence system may gather trip details provided by a user, search sources including the Internet and databases of curated travel content for travel information related to the provided trip details including available itineraries, and automatically parse the information obtained from the Internet to generate a structured travel plan. This may be achieved using at least one LLM combined with functions executed using tool calls by the LLM.

[0051] In some examples, using at least one LLM combined with functions executed using tool calls, the technology is an improved Al based system and methods for travel planning, and in some examples, a system and methods for optimizing travel planning using a voice and chat enabled Al system, including travel itineraries, identifying potential options for and / or booking flights, hotels, attractions, and so on. The disclosed Al system may gather trip details provided by a user, search the Internet for travel information related to the provided trip details including available itineraries, and automatically parse the information obtained from the Internet to generate a structured travel plan.

[0052] In some implementations, the technology disclosed provides a solution to the problem of composing a useful graphical user interface (GUI), responsive to a user input including a travel request or trip details, from tens of thousands of possible combinations of travel activities. A method implementation of one example of the technology disclosed includes invoking at least one callback function combined with prompting at least one trained Al, including an LLM running on specialized array processing hardware, to (i) produce spoken or textual queries for trip details and traveler details received from the user and (ii) distill, from user responses, the trip details and traveler details needed to populate a travel profile data structure. The queries may be dynamically adjusted to request trip details and traveler details not yet received in the user responses. Other implementations of the technology disclosed leverage text-based interaction in addition to, or in place of, voice-based communication. For the sake of clarity and conciseness, the example implementations described will primarily refer to voice-based communication, but it is understoodthat text-based communication can be implemented as an alternative to spoken queries and responses.

[0053] Some examples of disclosed methods may include further prompting the trained Al to search the Internet based on the trip details and traveler details, including searching travel blogs and travel web sites. In addition to the Internet, private libraries and other proprietary collections can be searched. The search may further include focusing the search to tourist-posted itineraries and other travel planning content like photos and videos of travel destinations, amenities, and attractions and travel logistics information like available flights and hotels to improve relevance of the search results and return a decreased quantity of the search results. The Internet search may further include search engine filtering in real time from at least thousands of posted itineraries that overlap with a travel destination in the travel profile data structure. The filtering may include comparing at least (i) the trip details and travel details of the travel profile data structure with (ii) trip details and travel details of posted itineraries, and rank-ordering at least some of the posted itineraries for presentation to the user. Limiting the search space in combination with using the specialized array processing hardware for the filtering reduces response times corresponding to the spoken queries, and the filtering and rank-ordering improves relevance of the search results. The resulting reduced response times improve user accessibility (e.g., attention span of a user and usefulness of the filtered and rank-ordered results).

[0054] Some examples of disclosed methods may further includes presenting, via a GUI, the filtered and rank-ordered results including a spoken summary of the filtered and rank-ordered results and a visual display including visual images and interactive links representing three or more of the posted itineraries of the filtered and rank-ordered results. Some implementations may further include receiving, from the user, an interaction with a visual image of the presented visual images and in response to the received interaction, prompting the trained Al to respond with a structured travel plan based on the selected visual image. In some implementations, the method may include, in addition to the Internet search, searching a proprietary data set including scraped and curated travel blogs, web sites, and tourist-created itineraries.

[0055] Some implementations of the technology disclosed include a callback function combined with prompting the trained Al, via an application programming interface (API), to (i) produce spoken queries for trip details and traveler details received from the user and (ii) distill, from user responses, the trip details and traveler details needed to populate a travel profile data structure. The trip details can further include a destination and one or more dates of travel. The traveler details can include a quantity of travelers and ages corresponding to respective travelers. The method furtherincludes prompting the trained Al, via the API, to use the trip details and traveler details in the travel profile data structure to search a constrained space of travel blogs and web sites that include tourist- posted itineraries, instead of searching the Internet as a whole. The method may further include selecting, using the trained Al, tourist-posted itineraries, from the constrained space of travel blogs and web sites, including computer-accessible trip details and traveler details.

[0056] Some implementations of the technology disclosed provides an innovative solution to the problem of curating travel itinerary data from tens of thousands of possible combinations of travel activities that includes directing a trained Al-based system running on specialized array processing hardware to carry out a spoken interaction with a user and to use a different search strategy than humans use, and producing a GUI that improves upon the state of the art by including novel features such as providing a spoken narration of the selected search results and travel itinerary options, interactive photographs and / or videos with links facilitating drill down into particular itineraries, receiving spoken user responses and processing the spoken user responses to provide further details about the selected search options and travel itinerary options or modify the selected search options and travel itinerary options based on the spoken user responses. In one example, the GUI provides further spoken details about a particular posted itinerary in response to receiving a user activation of a visual image, video, or interactive link displayed by the GUI and related to the particular posted itinerary. The trained Al system can further generate a structured trip plan based on one or more user interactions with the GUI related to at least one selected travel itinerary. The method can further include allowing the user to clone or modify the structured trip plan.

[0057] Various implementations of the disclosed method may include dynamically updating the GUI based on voice commands from the user without requiring manual inputs or visual cues. The dynamic updating of the GUI further includes the trained Al system processing the voice commands to trigger a creation, deletion, or modification of one or more UI elements in real-time based on the voice commands and the context of the conversation. The trained Al system may autonomously determine whether to trigger a creation, deletion, or modification of a specific UI element in dependence upon user input, a current stage of travel planning, and user preferences, wherein the user preferences have been obtained implicitly (i.e., inference) or explicitly (e.g., user settings).

[0058] Other implementations of the technology disclosed may relate to a method of collaborative travel planning using an Al system, wherein multiple users participate in the planning process. Collaborative travel planning implementations can include allowing two or more users to contribute respective voice commands and other GUI interactions, thereby triggering updates to a shareditinerary with real-time synchronization. The Al system can manage permissions and roles for each participant to enable specific control over different aspects of the planning process.

[0059] Another implementation of the technology disclosed may relate to a method for managing special inquiries during travel planning, such as disability accommodation planning, wherein an Al system automatically initiates calls to external entities such as hotels or restaurants and provides the user with a real-time transcript of the conversation. The user may also have access to real-time audio of the conversation. The user has the option to take over the conversation at any point. The real-time transcript is displayed on the GUI and the system allows seamless transition between Al control of the voice call and direct user control of the voice call. FIG. 19, provided below within the “GUI Examples” section, shows an example layout of a GUI for an implementation including an Al system automatically initiating calls to external entities such as hotels or restaurants and providing the user with a real-time transcript of the conversation. During the conversation, the Al system may also obtain availability information relating to booking hotels, tours, and so on, as well as booking such amenities.

[0060] In some implementations of the technology disclosed, an Al system assists in the optimization of a travel itinerary, wherein the Al system gathers trip details provided by the user, searches the Internet for available itineraries, and automatically parses the information into a structured trip plan. The Al system evaluates and ranks itineraries based on user-defined criteria such as budget, preferred activities, and travel time, and presents the most suitable options to the user.

[0061] In some implementations, an Al travel planning system collaborative planning module can be extended to enable multiple users to contribute to a shared itinerary using voice commands and provides real-time updates to the multiple users, including conflict resolution features. Other implementations include a computer-readable medium storing instructions that, when executed by a processor, cause an Al travel planning system to autonomously initiate calls to external entities, display a real-time transcript to the user, and allow the user to take over the conversation and update the travel itinerary accordingly.

[0062] Some implementations of the Al system for travel planning may include an intelligent assistant that interacts with users through natural language conversations to help them plan their trips. The system may be designed to gather relevant details about the user, including their travel preferences, family members, and their ages, as well as specific details about their upcoming trips, such as the travel destination, travel dates, and other preferences. The information persistencefunctionality of the Al system also can carry over across multiple trips, such that information collected during the planning of previous trips can be used for the planning of a future trip.

[0063] Features of some implementations may include:

[0064] 1. User Information Collection: During conversations, the system collects and stores information about the user, such as travel preferences (e.g., preferred types of destinations, accommodation preferences), the number of family members, and their ages. This information is stored in a user profile that is continuously updated as the user provides new details or refines existing preferences. The system uses this information to tailor trip suggestions, making recommendations that align with the user’s past choices and preferences.

[0065] 2. Trip Details Gathering: The system also gathers detailed information about specific trips the user is planning. This includes the travel destination, travel dates, activities of interest, and any other relevant details. Each trip’s details are categorized and stored separately, allowing the system to easily reference this information when the user needs to make changes or continue planning the trip in subsequent sessions.

[0066] 3. Information Persistence: All collected information, whether it pertains to the user’s general preferences or specific trip details, is stored securely and persists across sessions. This persistence allows the system to retrieve previously gathered information during future interactions, enabling the user to pick up where they left off without needing to repeat details. The system also uses this stored information to provide context-aware suggestions and actions. For example, if a user starts planning a new trip, the system can suggest destinations based on past preferences or remind the user of their typical travel dates.

[0067] 4. Session Continuity: The system is designed to maintain continuity between sessions. If a user starts planning a trip in one session and continues in a later session, the Al can retrieve the previously stored trip details and continue the conversation without needing the user to repeat information. This feature is particularly useful for users who may plan their trips over several interactions, allowing them to review and modify details incrementally.

[0068] 5. Personalized Recommendations: By analyzing the stored user information and trip details, the system can offer personalized recommendations for destinations, activities, and accommodations that match the user's preferences. The system can also alert users to special offers or travel opportunities that align with their profile, making it easier for them to discover new experiences that fit their preferences.

[0069] The following is an example interaction flow according to some implementations:1. Initial Interaction:• User: "I’m planning a family trip next summer."• Al: "Great! How many family members will be traveling with you, and what are their ages?"• User: "It’s me, my spouse, and our two kids, aged 8 and 10."• Al: "Got it. Do you have any preferences for the destination?"2. Continuing the Plan:• User, in a later session: "Let’s continue planning that family trip."• Al, after retrieving the stored information: “Last time, you mentioned you’re planning a trip with your spouse and two kids, aged 8 and 10. Are you still considering the same group?”• The user confirms or updates the details, and the Al continues the conversation from where it left off.3. Personalized Suggestions:• Based on the stored preferences and past trips, the Al suggests: "Since you enjoyed your last beach vacation, how about exploring the Caribbean this summer? I can find familyfriendly resorts that cater to kids aged 8 and 10.”4. Finalizing the Trip:• When the user is ready to book, the Al summarizes all the details: "You’re planning a trip to the Caribbean in July with your family. Shall I look for flights and accommodation options now?"

[0070] The following are some benefits of the system of some implementations:• Efficiency: By persisting information across sessions, the system reduces redundancy, allowing users to focus on refining their plans rather than re-entering details.• Personalization: Tailored suggestions based on stored preferences enhance user satisfaction and help create memorable travel experiences.Convenience: Users can plan their trips at their own pace, picking up where they left off in previous sessions, making the planning process more flexible and user-friendly.

[0071] As discussed herein, the one or more LLMs work in combination with functions to enhance the ability of the LLM to provide replies, e.g. by the LLM generating output based on output tokens that call the functions and provide arguments to the functions. At least some output of some of the functions may be provided to the LLM (and / or another LLM) as additional context. Some example functions in some implementations include:

[0072] information: The information function is designed to present detailed information about a specific topic to the user. It takes an object as its argument, with properties including title (a string representing the title of the information), subtitle (an optional string for additional context), description (a markdown-formatted string with detailed information), and priority (an array of strings that indicate the type of content to prioritize, such as photos, videos, text, or maps). Optionally, it can also include a map object, which may contain details like zoom_l evel and points of interest (POIs). The server processes this information and responds with the requested content, which could include media like photos or videos, depending on the priority set in the function's arguments.

[0073] getTripInspiration: The getTripInspiration function is aimed at providing users with ideas and inspiration for their trips. It accepts an object that includes a search_query (a string that represents the user's search intent related to trip ideas) and trip_details (an array of objects that specify trip-specific details such as dates and locations). The server responds by gathering and returning a set of trip ideas based on the user's input. It searches various sources for relevant content, processes the data, and compiles a list of trip inspirations. The server may also fetch related media like photos to visually enhance the ideas presented. If the user provides more specific details in the search query, the server tailors the inspiration options more closely to match the user's preferences. This process ensures that users receive a curated list of potential trips that align with their expressed interests and details.

[0074] userOnboarding: The userOnboarding function manages the onboarding process for new users, helping them clarify and define their travel preferences. It takes an object with a step property that outlines the current phase of the onboarding process. This step can include a type (such as question, priorities, or images), a question to be answered by the user, instructions for the next steps, and potentially options or image labels. The server responds by guiding the user through the next step of the onboarding flow, ensuring that the process is intuitive and tailored to the user's input, ultimately helping users define their trip details and preferences more accurately.

[0075] tripDestination: The tripDestination function is designed to manage and confirm the destination details of a user's trip. It takes an object with several properties, including confidence (a number between 0 and 1 indicating how confident the system is about the location), location (a string with the destination name), latjong (an array of latitude and longitude coordinates), and details_confirmed (a string indicating whether the details have been confirmed, either 'false' or 'true'). It may also include funfacts (an array of interesting facts about the destination). The server uses this information to confirm or update the trip destination details, ensuring that the planning process accurately reflects the user's intentions and preferences.

[0076] travelDates: The travelDates function handles the confirmation or adjustment of travel dates for a user's trip. The function accepts an object with properties such as departure_date (a string in ISO format representing the start of the trip), return_date (a string in ISO format for the end of the trip), details_confirmed (a string indicating if the dates are confirmed), and details_inferred (a string indicating whether the dates were inferred by the system). The server processes these details, confirming or updating the travel dates as necessary to ensure that the trip is planned according to the user’s schedule.

[0077] travelers: The travelers function is responsible for managing the information about the people who will be part of the trip. It takes an object that includes a travelers array (each element of the array is an object containing details about an individual traveler) and a details_confirmed string (indicating whether the traveler details have been confirmed). The server processes this information to ensure that all travelers are accounted for in the trip planning, providing a more personalized and comprehensive travel experience.

[0078] messagePredictions: The messagePredictions function retrieves message predictions based on the conversation history and stores them in the results.

[0079] Search Queries: Many implementations of the technology disclosed include triggering the LLM to generate a search query containing all the known information about the trip (e.g. winter family vacation with young children to Oslo). A search engine is then employed for this query to find the relevant web pages. In one example, the html of the top 3 results are parsed and modified into a format for display by the client. Examples of queries are as follows: active family trip to Ottawa with hiking and biking; active guided trip with hotel accommodation, outdoor adventure, budget-friendly, wildlife safari; active guided trip with hotel accommodation, skiing, and beach activities; active vacation with guided tours, hotels, skiing, beach, cultural experiences, budgetfriendly; active vacation with guided tours, staying in hotels, enjoying skiing, beach, and hiking;Christmas and new year in Ottawa for a couple; Christmas trip to Ottawa for a family of four; culturaland relaxing family adventure in Sydney; cultural family adventure a week in paris; dune buggy tours in Tunisia; events and outdoor activities in Ottawa in late august; family adventure in morocco august; family adventure in Ottawa in September; family adventure in paris September 2024; family fun a relaxing week in new york city and paris; family fun a week in Ottawa; family trip to paris, france in august; hiking, beach, skiing, sipping coffee; italy summer trip for couple; mississauga, Ontario, Canada weekend trip; montreal, quebec, Canada; morocco family trip august 2024; Ottawa hotels for 2 adults; Ottawa travel September 2024; outdoor adventure budget hiking; paris, france travel ideas; paris, france travel itinerary for 5 nights; relaxing beach vacation with family; relaxing guided tours staying in hotels flying; relaxing guided tours with hotel stays; romantic february getaway to paris; romantic getaway a february escape to paris; romantic getaway a September week in paris; romantic getaway in Ottawa, Ontario, Canada; romantic getaway three nights in Ottawa; solo adventure exploring paris in September; solo adventure in Ottawa in august; solo senior adventure two weeks at dark lake, Michigan; Spain trip july 1 to july 8 with friends; summer adventure exploring Ottawa in august; summer adventure exploring Ottawa in august; summer adventures a week in Ottawa; tokyo family-friendly cultural budget travel; tokyo, paris, Sydney outdoor adventure, family-friendly, destination; trip ideas in Ottawa, Canada; winter wonders a festive Christmas in Ottawa.

[0080] Some GUI examples will now be provided.

[0081] FIGs. 5 - 13 show an example GUI interaction with a user seeking travel planning assistance for a trip to Ottawa, in accordance with one implementation of the technology disclosed.

[0082] FIG. 5 shows an initial GUI 500 prepared to assist a user in their travel planning, including receiving spoken user responses to be processed by a trained Al system. In addition to the rendering of visual text on the user display, a text-to-speech functionality can be used to produce audio of spoken queries to the user simultaneously with (or instead of) the visual text display. The spoken queries presented to the user can match the displayed text (“Hi I’m your Al travel agent. How can I help you plan your trip? Just talk to me.”) or deviate from the displayed text, such as a shortened summary, an expanded explanation, or a personalized spoken query in response to a user that has already begun speaking to the GUI.

[0083] FIG. 6 shows an example GUI 600 responding to a verbal input from a user stating that the user is requesting assistance with planning a trip to Ottawa. In response to the user stating, within a verbal command, “Hi. I’d like to plan a trip to Ottawa.,” the received verbal input is converted via speech-to-text into a transcript that is presented back to the user in real-time. Theuser is able to read the transcript to confirm that the verbal command has been accurately processed, and if the system has erroneously misunderstood the verbal input, the user is able to quickly identify the error and provide a correction in additional verbal commands. On the right side of the display, a UI element is dynamically updated in response to the receiving and processing of verbal commands from the user as important trip details are received, such as the travel destination. For example, at the present trip planning stage of the process represented in FIG. 6, the “Trip details” 602 visual element has been updated to include a section titled “Where,” and the “Where” section has been populated with “Ottawa, Ontario, Canada.” The “Where” section further includes an interactive button resembling a pencil. The user can activate the button to initiate modification of the travel destination. Alternatively, the user can also directly initiate modification of the travel destination with a verbal command.

[0084] The GUI 600 further displays a visual element for receiving travel dates from the user after receiving the trip destination information from the user. The travel date visual element indicates that the dates being considered include September 2024. A query from the system is displayed in text (“Does traveling from Sunday, September 1stto Saturday September 7thsound good to you?”) and the query is also spoken to the user. The user has the option of responding to the question (i.e., confirming the date range or changing to a different date range) via voice command without additional manual input. Alternatively, interactive buttons are presented within the travel date visual element, one to “Confirm” and one to “Change” the date range, if the user prefers.

[0085] FIG. 7 shows another example GUI 700 responding to a verbal input from a user stating that the user is requesting assistance with planning a trip to Ottawa. In contrast to the confirmation of travel dates in FIG. 6, FIG. 7 confirms the intended destination prior to the system proceeding to the search stage. A destination visual element, similar to the travel date visual element of FIG. 6, states “Ottawa, Ontario, Canada” and further includes a rendered map region with Ottawa located via a pin point icon. A query from the system is displayed in text (“Great choice! Ottawa, Ontario, Canada is a beautiful destination. Is this the correct location for your trip?”) and the query is also spoken to the user. The user has the option of responding to the question (i.e., confirming the destination or changing to a different destination) via voice command without additional manual input. Alternatively, interactive buttons are presented within the destination visual element, one to “Confirm” and one to “Change” the date range, if the user prefers.

[0086] FIG. 8 shows another example GUI 800 in the process of assisting a user plan a trip to Ottawa, displaying both (a) a visual indicator 802 communicating to the user that search is running and (b) a query 804 presented to the user for obtaining further traveler information. As indicated by (a) the visual indicator 802 communicating to the user that search is running (“I’m finding some magical trips for you!”) the search operations, as described above, have begun to select travel itineraries to display. While the search is running, the GUI continues to collect useful information from the user. For example, the system has inferred from a previous voice command that the user will be traveling with their wife, and the query 804 is displayed at the bottom of the page confirming the traveler information (“And just to confirm, it will be you and your wife traveling, correct?”). The query is also spoken out loud to the user to continue voice-based communication.

[0087] FIG. 9 shows another example GUI 900 in the process of assisting a user plan a trip to Ottawa, confirming traveler information parsed from voice commands received by the user. A traveler visual element, similar to the visual elements of FIGs. 6 and 7, displays icons and text indicating that 3 adults and one 12 year old child will be traveling. At the stage displayed in FIG. 9, the system further confirms that the traveler information parsed from the user voice command is accurate. A query from the system is displayed in text (“So, it will be you, your wife, your twelve-year-old son, and your twenty-four-year-old daughter. Is that correct?”) and the query is also spoken to the user. The user has the option of responding to the question (i.e., confirming or changing the traveler information) via voice command without additional manual input. Alternatively, interactive buttons are presented within the traveler visual element, one to “Confirm” and one to “Change” the traveler information, if the user prefers.

[0088] FIG. 10 shows an example visual element of a GUI 1000 that includes important details of the user’s trip planning process. The displayed “My Trip” element is dynamically updated in response to the receiving and processing of verbal commands from the user as important trip details are received, such as the travel destination. For example, at the present trip planning stage of the process represented in FIG. 10, the “My Trip” visual element has been updated to include an Al-generated title (“Family Adventure: Exploring Ottawa in September”), a “Where” section populated with “Ottawa, Ontario, Canada,” a “When” section populated with “September 1 to September 7,” and a “Who” section populated with “3 adults and 1 child,” Similarly to the trip details visual element of FIG. 6, the sections included within the “My Trip” visual element each further include an interactive button resembling a pencil. The user canactivate the button to initiate modification of the corresponding section. Alternatively, the user can also directly initiate modification of the trip details with a verbal command.

[0089] FIG. 11 shows another example GUI 1100 in the process of assisting a user plan a trip to Ottawa, presenting two Al-selected itineraries for visiting Ottawa. As described above, at least one trained Al is used to search the Internet (and optional additional resources, such as a proprietary dataset), and select the most relevant results from a filtered and rank-ordered list of itineraries. The GUI display of FIG. 11 displays two itineraries to the user, accompanied by the statement “The 2 best trips I found for you!” at the top of the display. A spoken summary of the two itineraries is provided to the user. Each respective itinerary is represented by a photograph, a title, selected details from the itinerary, an interactive “Explore” button, and an interactive “Clone this trip” button. When one of the interactive “Explore” buttons is activated by the user (manually or verbally), additional information about the corresponding itinerary is displayed, as shown in FIG. 1 below. When one of the interactive “Clone this trip” buttons is displayed, a copy of the itinerary is saved for the user. Optionally, the user can continue interacting with the GUI to further modify the cloned trip. Alternatively, the user may provide additional verbal commands to modify the displayed results. Examples displayed on the screen include the questions “What inspiration do you have?”, “Can you recommend activities?”, or “What about the weather?”, but the user is not limited to the suggested examples. The user is additionally provided with an interactive link to switch from talking to typing, as expanded upon further with reference to FIG. 13.

[0090] FIG. 12 shows another example GUI 1200 in the process of assisting a user plan a trip to Ottawa, presenting one itinerary selected from the interface of FIG. 11 for the user in response to a request to explore the itinerary further. Additional detail for the selected itinerary is provided, including more text details and extra photographs. As indicated by the scrolling text partially visible at the bottom of the screen (“You can explore the festive charm...”), a narrative summary for the itinerary being explored is provided in a spoken and a text format. As with the interface of FIG. 11 , the user may provide additional verbal commands to receive additional information about the itinerary. Examples displayed on the screen include the questions “What are the ideas?”, “Suggest activities for kids”, or “Include shopping options”, but the user is not limited to the suggested examples. The presented examples may dynamically change over time, as shown by the different examples in FIG. 13 below (“What about food options?”, “Can you list events?”, “Suggestions for gifts?”).

[0091] FIG. 13 shows a portion of an example GUI 1300 in the process of assisting a user plan a trip to Ottawa, representing the system transitioning to receiving text inputs from the user rather than speech. As shown in the interface of FIG. 13, the user is able to switch to manual inputs (e.g., typing) instead of providing voice commands. In response to the user activating an interactive link, the system has switched to receiving typed inputs instead of verbal commands, as indicated by the displayed text (“I see you’ve switched to typing.”). In one implementation, the user is able to start typing at any time without performing additional steps first, like activating an interactive link, and the system will automatically switch to processing typed inputs. In another implementation, the system will switch to responding with text-only without verbal queries when the user begins to provide typed inputs, as indicated by the displayed text (“I’ll respond with text from now on.”). In other implementations, the system can be switched to responding with text only independently of whether the user is providing verbal or typed inputs.

[0092] FIGs. 14 to 18 show another example GUI in the process of assisting a user plan a trip to Tunisia.

[0093] FIG. 14 shows an example GUI 1400 in the process of assisting a user plan a trip to Tunisia, representing the system responding to a follow-up question from the user. As indicated by the transcribed text at the top of the screen, the user has requested to see photos of Tunisia. In response, a variety of photos of Tunisia are presented to the user. FIG. 15 shows a text transcription 1500 of the user’s follow up question in response to the presented photos, “That’s great. Can you show me on a map the main sites to see where in the country the main tourist destinations are?”. FIG. 16 shows the GUI display 1600 in response to the question of FIG. 15, including a map of Tunisia annotated with pinpoints indicating where popular tourist destinations are located. The pinpoints are both numbered and color-coded, as shown in a key below the map. A narrative can be presented to the user describing the indicated tourist destinations on the map, verbally and / or via text, as indicated by the text at the bottom of the screen (“Tunisia is home to several key tourist destinations...”).

[0094] FIGs. 17 and 18 show another progression of the trip planning from FIGs. 14-16, in response to the user asking, “Can you show me a video of Tunisia?”. In FIG. 17, the text response 1700 displayed states “I found a video about Tunisia that covers food, transport, and more.” The displayed text can also be spoken aloud to the user. FIG. 18 shows the video 1800 displayed to the user in response to the request for a video of Tunisia.

[0095] FIG. 19 is a wireframe representation 1900 of a trip planning GUI according to an implementation that the voice-based Al system autonomously initiating phone calls to externalentities, such as hotels. As indicated by the text 1902 at the bottom of the display, that may also be spoken aloud, the system is currently calling a variety of hotels to verify that wheelchair- accessible rooms can be accommodated for the user’s son. Additionally, the system instructs the user that in addition to listening in to the phone calls, the user is able to take over direct control of the conversation at any time. A first hotel is currently being presented to the user, including two photos of the hotel. On the right side of the display, a transcript 1904 of the voice conversation between the Al-based system and the hotel is presented to the user along with an interactive button to initiate user control of the conversation and an interactive button to begin providing a real-time audio feed of the call so that the user can listen in to the conversation rather than merely reading a transcript. The display has two additional tabs for second and third hotels, indicating that a similar voice conversation process will be performed for all three hotels, thereby enabling the user to compare the three respective options.

[0096] The examples described with reference to FIGs. 1 - 19 are provided for illustrative purposes, and other various implementations of the technology disclosed will be apparent to a user skilled in the art. Some particular implementations and features for the disclosed technologies are described in the following discussion. The method described in the following section and other sections of the description can include one or more of the following features and / or features described in connection with additional methods disclosed. In the interest of conciseness, the combinations of features disclosed in this application are not individually enumerated and are not repeated with each base set of features. The reader will understand how features identified in this method can readily be combined with sets of base features identified as implementations.

[0097] A method implementation of the technology disclosed according to one example includes optimizing travel information including travel itineraries using an Al system, which can use spoken queries and responses, typed queries and responses, or a choice between spoke and typed. The Al gathers trip details provided by the user, searches the Internet and / or curated databases for available travel information and itineraries, and automatically parses the information into a structured trip plan. The example method further includes invoking at least one callback function combined with prompting at least one trained Al, including a large language model (LLM) running on specialized array processing hardware, to (i) produce spoken queries for trip details and traveler details received from the user and (ii) distill, from user responses, the trip details and traveler details needed to populate a travel profile data structure.The spoken queries are dynamically adjusted to request trip details and traveler details not yet received in the user responses.

[0098] The method may also include further prompting the trained Al to search the Internet based on the trip details and traveler details, including searching travel blogs and travel web sites. The Internet search may further include focusing the search to tourist-posted itineraries to improve relevance of the search results and return a decreased quantity of the search results. The Internet search may further include search engine filtering in real time from at least thousands of posted itineraries that overlap with a travel destination in the travel profde data structure. The filtering may include comparing at least (i) the trip details and travel details of the travel profile data structure with (ii) trip details and travel details of posted itineraries, and rank-ordering at least some of the posted itineraries for presentation to the user. Limiting the search space in combination with using the specialized array processing hardware for the filtering reduces response times corresponding to the spoken queries, and the filtering and rank-ordering improves relevance of the search results. The resulting reduced response times improve user accessibility (e.g., attention span of a user and usefulness of the filtered and rank-ordered results).

[0099] The method may further include presenting, via a GUI, the filtered and rank-ordered results including a spoken summary of the filtered and rank-ordered results and a visual display including visual images and interactive links representing three or more of the posted itineraries of the filtered and rank-ordered results. Some implementations of the method may include receiving, from the user, an interaction with a visual image of the presented visual images and in response to the received interaction, prompting the trained Al to respond with a structured travel plan based on the selected visual image. In some implementations, the method may include, in addition to the Internet search, searching a proprietary data set including scraped and curated travel blogs, web sites, and tourist-created itineraries.

[0100] In some implementations, the Al system autonomously determines the necessity of triggering specific UI elements based on user input, the current stage of travel planning, and inferred user preferences. Another implementation includes a method for collaborative travel planning using an Al system, wherein multiple users can be invited to participate in the planning process, allowing each participant to contribute voice commands and updates to a shared itinerary with real-time synchronization. Yet another method includes the Al system managing permissions and roles for each participant, enabling specific control over the planning process. Most trips are taken with more than one person, and accordingly, many implementations of the disclosed system include functionality allowing a user to add more people to the trip planningprocess to simplify planning trips (e.g., friends or a spouse). For example, you could add your friends or spouse. When the other travelers join, they can be prompted to suggest changes to the trip. The suggestions may then be added to the trip or changes would be made directly or suggestions could be sent to the other travelers included in the trip planning process.

[0101] Other implementations include a method for managing special inquiries during travel planning, wherein an Al system autonomously initiates voice calls to external entities such as hotels, providing the user with a real-time transcript of the conversation and the option to take over the conversation at any point. In some implementations, the real-time transcript is displayed on the user interface, and the system allows seamless transition from Al control to direct user control of the voice conversation.

[0102] In one disclosed implementation, the Al system evaluates and ranks itineraries based on user-defined criteria such as budget, preferred activities, and travel time, and presents the most suitable options to the user. Various implementations may include an Al travel planning system, comprising a collaborative planning module that enables multiple users to contribute to a shared itinerary using voice commands, with real-time updates and conflict resolution features.

[0103] Other implementations may include a non-transitory computer readable storage medium (CRM) storing instructions executable by a processor to perform the method described above. One implementation includes a computer-readable medium storing instructions that, when executed by a processor, cause an Al travel planning system to autonomously initiate calls to external entities, display a real-time transcript to the user, and allow the user to take over the conversation and update the travel itinerary accordingly.

[0104] Yet another implementation may include a system including memory and one or more processors operable to execute instructions, stored in the memory, to perform the method described above.

[0105] Each of the features discussed for a method implementation apply equally to the CRM implementation. All the method features are not repeated here and should be considered repeated by reference.

[0106] Some implementations may include a method for building traveler context using an Al system, wherein the system connects to external data sources including email, online travel agencies (OTAs), and social media platforms to gather and integrate information about the user’s travel history, preferences, and social connections, creating a comprehensive traveler profile to personalize the trip planning process. In one implementation, the Al systemcontinuously updates the traveler profile by monitoring incoming data from the connected sources, allowing the system to adapt travel recommendations in real-time based on changes in the user’s travel preferences or social circumstances.

[0107] Another implementation includes a method for providing travel support during the trip planning process and actual travel, wherein a human travel agent collaborates with the user, supported by an Al system that provides real-time context, recommendations, and logistics management based on the traveler's profile and current travel conditions. The human travel agent is expected to provide support while on the road. The agent can review and comment on the itinerary, which doubles as training data. A travel agent in the local destination can help with local fare, such as to connect a tour guide. The agent also provides support if something isn’t working during a trip. In one implementation, the Al system assists the human travel agent by generating actionable insights, automating routine tasks, and facilitating seamless communication between the traveler and the agent during both the planning and travel phases.

[0108] One implementation of the technology disclosed includes a system for generating photorealistic images of a traveler and their companions at a selected trip destination before travel occurs, wherein the system uses generative Al to create visual representations based on photos of the traveler and companions, combined with destination-specific scenery. The system may further include the generative Al incorporating environmental factors such as weather, time of day, and local landmarks into the images to provide an immersive preview of the trip experience.

[0109] The system may also further include an embodiment of the entity that is responding. The entity can be an animated avatar, whether human, comic character or animal. It can be a visual entity that morphs, such as becoming an airplane when planning flights, a building when planning accommodations, or a vehicle when planning ground transport. It can be a sphere or map section representing locations.

[0110] Each of the features discussed above in relation to a method implementation apply equally to the system implementation. All the method features are not repeated here and should be considered repeated by reference.

[0111] The following provides example prompting and callback functions used by certain implementations of the technology disclosed. In particular, the following is an example LLM prompt according to one implementation, showing the LLM being configured to call tools that execute functions, thereby enhancing the functionality of the LLM. In the example prompt, thefollowing functions can be executed by way of the tool calls: “tripDestination()”, “travel Dates()’", “travelersQ”, “getTripInspirationQ’-, “goBackQ”, “tripTitleQ’-, “endCall()’-, “closeModuleQ”, “captureTravelDetailsO”, “informationO”, and “getTripInspiration’'. Instructions are provided to the LLM regarding when to make the tool call to execute the function, what arguments to pass, etc.

[0112] Example prompt: export const travelAssistant: any = { name: ’Travel Fox’, model: { provider: ’openai’, model: ’gpt-4o’, temperature: 0, emotionRecognitionEnabled: true, messages: [{ role: ’system’, content: 'You are an expert travel planner with extensive knowledge of destinations worldwide. A user has asked for your help planning the perfect trip. Your goal is to provide concise, personalized recommendations based on the user's preferences, interests, and specific requirements. Offer advice on the best times to visit, transportation options, safety tips, local customs, packing suggestions, and necessary preparations (e.g., visas, vaccinations). Ask questions to better understand their preferences, such as budget, travel dates, interests, and any special requirements (e.g., accessibility needs, dietary restrictions). Tailor your recommendations based on their responses.Today's date is ${new Date().toISOString().split('T')[0] }.## Primary Mode of Interaction: ##Interactions occur mainly through voice audio. Please ensure that your responses are clear and easy to understand when spoken aloud. For example, unless instructed otherwise, responses should only be in concise paragraph format using complete sentences, no headings and no lists.## Flexible Conversation Handling ##- Adapt to non-linear conversations, allowing users to change details at any point.- Gather trip information in any order, tracking known details and identifying missing information.- Allow easy modification of previously provided information.- Ask for missing details opportunistically based on the conversation flow.- Handle and incorporate user interruptions or tangents gracefully.- Process multiple intents in a single user message when necessary.- Provide a way for users to revisit or modify earlier parts of the conversation.- Offer partial recommendations based on incomplete trip details.- Suggest alternatives if user preferences can’t be met.- Support open-ended travel exploration without requiring specific details.## Function Instructions: ##- tripDestinationQ: The first time this function is called, the details_confirmed parameter must be set to "false". Once the user has confirmed their destination details, this function should be called again with the details_confirmed parameter set to "true". When a user mentions any place name, geographic feature, or location that could be a travel destination, call the tripDestination function. This includes countries, cities, regions, landmarks, or natural features. If the mention is ambiguous or could refer to multiple locations, ask the user to clarify. Always call this function even if you're not certain about the location - use the confidence field to express your level of certainty. Examples of when to call tripDestination:- "I'm thinking about going to Paris" (clear destination)- "I love the beaches in Thailand" (country as destination)- "Maybe somewhere in Europe" (region as destination)- "I want to see the Eiffel Tower" (landmark implying destination)- "Springfield sounds nice" (ambiguous location)- "I live in Palo Alto and plan on traveling to Ottawa"- travelDates(): The first time this function is called, the details_confirmed parameter must be set to "false". Once the user has confirmed their travel dates, this function should be called again with the details_confirmed parameter set to "true". If the user has mentioned a time period for their trip without specifying exact dates, try to estimate their dates and set the "details_inferred" parameter to true. When estimating dates, if no length of stay has been specified, assume the trip will be 7 days. The departure_date and retum_date parameters must be later than today’s date and the return_date parameter must not be earlier than the departure_date. If the user has not specified a year, select the first date that is later than today's date. Also call travelDates if the user indirectly implies travel dates, such as mentioning a festival or event that occurs at a specific time. For example:- if the user sets a departure_date of February 2, and the current date is 2024-07-09, then the date should be 2025-02-02.- if the user specifies a travel date of September 2 and the current date is 2024-07-09, then the date should be 2024-09-02.- travelers(): The first time this function is called, the details_confirmed parameter must be set to "false". Once the user has confirmed the number of travelers, this function should be called again with the details_confirmed parameter set to "true". If the user has indicated that others will be traveling with them, but wasn't specific, set the "details_inferred" parameter to "true".- getTripInspiration(): Call this function automatically and immediately after all of the trip details (including destination and travel dates) have been confirmed by the user. For example, trigger this function after the user has specified and confirmed their travel dates, the number of travelers and the location of their trip.- goBackQ: Users will be shown different modules during the conversation. Call this function when the user indicates they want to navigate back to a previous module or screen.- tripTitle(): Call this function when new trip details such as the dates, destination or number of travelers have been mentioned by the user. For example, if the user says they are going to Ottawa in February for a long weekend, a trip title could be "Winter Wonders: A Frosty February Weekend in Ottawa"- endCallQ: If the user has indicated that they would like to stop talking, or they are done planning their trip, use the "endCall" function.- closeModuleQ: Call this function when the user has asked that a module be closed, hidden or dismissed.- captureTravelDetailsQ: Call this function when the user mentions details about their trip or themselves that would be important to consider when making travel recommendations. For example: dietary restrictions, activity preference, accommodation type preference, preferred method of transportation, interests, budget, must see attractions.- information(): This function should be called when the user has asked for information about a topic not covered by the following functions: "tripDestination", "travelers", "travelDates", "getTripInspiration". It is mandatory that the output of the description parameter be markdown. Examples of when to call the "information" function:- "Tell me about india" (text)- "Show me photos of Ottawa" (photo)- "Where is Tokyo" (map)- "Can I see a video of the Eiffel Tower" (video)- userOnboarding_v2(): It is mandatory that this function be called when the user indicates that they need help planning their trip, they aren't certain of details, what priorities to consider, expressesuncertainty or indecision about their travel plans. Examples include, but are not limited to, phrases like "I need some ideas", "I don't know what important details I should consider", "I don't know where to go", "I don't know what kind of trip I'm looking for", "I'm not sure where to go.", "I don't know what to do", "I'm not sure what to do", "I'm not sure where to go", "Can you help me decide?", "I'm overwhelmed with choices", "I need some suggestions", "I'm confused about my options", "Can you help me plan?", "I need travel ideas", "I'm uncertain about my trip". The information should use what is known about the user to determine the contents of the step. For example:- type: "question"- question: "What did you have in mind for your trip?"- suggested_questions: ["Are you looking for a relaxing or active vacation?", "Do you prefer guided tours or exploring on your own?", "What type of accommodation do you prefer?"]## Rules: ##- Provide information, answer questions, and make recommendations about travel only. Travel related requests such as photos, videos or maps are allowed.- Do not fabricate information. It is acceptable to say you don't know an answer.- Present dates in a clear format (e.g., January Twenty Fourth) and Do not mention years in dates. Present time in a clear format (e.g. Four Thirty PM) like: 11 pm can be spelled: eleven pee em. Speak dates gently using English words instead of numbers.- If the user repeats the same request, it is fine to repeat the same answer.Always adhere to the rules in the rules above when responding to the user's messages. ## Task: ##1. Flexibly gather trip information, confirming each detail as it's provided. If the user is uncertain about where to go or what to do, use the userOnboarding_v2 function.2. Offer trip inspiration based on known details, even if incomplete.3. Adapt to the user's conversation style, allowing for changes and exploration throughout the planning process.',},], tools: [{ async: true, type: 'function', function: {name: 'goBack', description: 'Call this function when the user has indicated that they want to navigate to the previous screen.', }, }, { async: true, type: 'function', function: { name: 'captureTravelDetails', description:'Captures important details about the user or their planned trip. If the detail is specific to travel dates, destination or number of travelers, use those functions instead.', parameters: { type: 'object', properties: { details: { type: 'array', description: 'An array of captured details’, items: { type: 'object', properties: { detail_type: { type: 'string', enum: ['userjnfo', 'trip_info'], description: 'Indicates whether the detail is about the user or the trip. A value of "user_info" should be set when the detail is specific to the user. For example, the following preferences should be considered "user_info": accommodation, dietary restrictions, activity.',}, category: { type: 'string', enum: [ ’personal_preference' , 'budget','accommodation','transportation','activities','dietary_requirements','accessibility_needs','other',], description: 'The category of the captured detail',}, other_category: { type: 'string', description: 'If category is other, create a category',}, detail: { type: 'string', description: 'The specific detail mentioned by the user',}, confidence: { type: 'number', minimum: 0, maximum: 1, description: 'Confidence level of the captured detail (0 to 1)', },}, required: ['detail_type', 'category', 'detail', 'confidence'],}, minltems: 1,},}, required: ['details'],},},},{async: true, type: 'function', function: { name: 'closeModule', description: 'Name of the UI module that should be closed', / / strict: true, parameters: { type: 'object', properties: { module: { type: 'string', enum: ['travelers', 'information', 'getTripInspiration', 'tripDestination', 'travelDates'], }, }, required: I'module'],}, }, }, { async: true, type: 'function', function: { name: 'tripTitle', description: 'This function should be triggered when trip details have been specified by the user.', parameters: { type: 'object', properties: { title: { type: 'string', description: "Creative title for the user's trip based on their travel details.", }, }, },}, }, { async: true, type: 'function', function: { name: 'endCall', description: 'Call this function when the user has indicated that they are done planning or would like end the call.', }, }, { async: false, type: 'function', 270 server: { url: 'https : / / northamerica-northeastl -jane- 425814.cloudfunctions.net / travelfox-api', }, function: { name: 'userOnboarding_v2', description: 'It is mandatory that this function be called when the user indicates that they need help planning their trip, they aren't certain of details, what priorities to consider, expresses uncertainty or indecision about their travel plans.', parameters: { type: 'object', properties: { step: { type: 'object', properties: { type: { type: 'string', enum: ['question'], }, question: { type: 'string',description: ’The question to be displayed.',}, category: { type: 'string', description: 'The category of the step. For example: destination, activities, travel preferences...', }, suggested_questions: { type: 'array', description: "A list of 3 questions for the 'question' type step. The questions should inform the user on the types of information the user can provide.", items: { type: 'string', description: 'A suggested question.',},},}, required: ['type', 'question', 'category', 'suggested_questions'],},}, required: ['step'],},},},{ async: false, type: 'function', server: { url: 'https: / / northamerica-northeastl-jane-425814.cloudfunctions.net / travelfox-api',}, messages: [{ type: 'request-start', content: ",},], function: { name: 'travelDates', description: 'Use this function when the user’s departure and return dates have been specified. Ensure that these dates are later than and relative to the current date of ${newDate().toISOString().split(T)[0]}. The "detail s_confirmed" parameter should be set to "false" until the user has confirmed the details. The "details_inferred" parameter should be set to "true" if the dates have been guessed (e.g. the user says they will be traveling in a particular season or month without specifying the exact date)', parameters: { type: 'object', properties: { departure_date: { type: 'string', description: 'The start date of the trip in ISO 8601 format (YYYY-MM-DD).', }, return_date: { type: ’string’, description: 'The end date of the trip in ISO 8601 format (YYYY-MM- DD).', }, details_confirmed: { type: ’string’, enum: [’false’, ’true’], description: ’This defines whether the departure_date and return_date have been confirmed by the user.’, }, details_inferred: { type: ’string’, enum: [’false1, ’true’], description: ’This defines whether the departure_date or retum_date have been inferred based on imprecise information from the user.’,},}, required: ['departure_date', ’return_date'], }, }, }, { async: false, type: 'function', server: { url: 'https: / / northamerica-northeastl-jane-425814.cloudfunctions.net / travelfox-api', }, messages: [ { type: 'request-start', content: ”, }, ], function: { name: 'tripDestination', description: 'This function should be called when the user has mentioned the destination of their trip and when the user has confirmed their destination. The "details_confirmed" parameter should be set to "false" until the user has confirmed the location.', parameters: { type: ’object1, properties: { confidence: { type: ’number’, minimum: 0, maximum: 1, description: 'Confidence level in the identified location, from 0 to 1.', }, location: { type: 'string', description: 'The destination of the trip, including city, state / province, and country.',}, latjong: { type: 'array', items: { type: 'number', }, description: 'The latitude and longitude of the trip destination as an array of numbers. Example: [45.424721 , -75.695000]',}, details_confirmed: { type: 'string', enum: ['false', 'true'], description: 'This defines whether the location has been confirmed by the user.', }, funfacts: { type: 'array', items: { type: 'string', }, description: 'List of up to 5 interesting facts about the trip location.', }, }, required: ['location', 'lat_long', 'details_confirmed'], },}, }, { async: false, messages: [ { type: 'request-start', content: 'One moment while I gather trip ideas for you.', }, {type: 'reques t-response-delayed’ , content: "I'm almost done finding amazing trip ideas. This shouldn't be much longer.", timingMilliseconds: 10000, }, ], server: { url: 'https: / / northamerica-northeastl-jane-425814.cloudfunctions.net / travelfox-api',}, type: 'function', function: { name: 'getTripInspiration', description: 'This function fetches trip ideas to be presented to the user. It should be triggered immediately after the trip details (including destination and travel dates) have been confirmed by the user.' , / / strict: true, parameters: { type: 'object', properties: { search_query: { type: 'string', description: 'Query appropriate for a google search to find high quality trips on the web based on important details about the user and their travel preferences.', }, trip_details: { type: 'array', items: { type: 'object', properties: { key: { type: 'string', description: 'This is the key or detail for which the values have been set. For example, key can be departure_date, retum_date, travelers, location, or other trip details the user mentioned.’, }, value: {type: 'string', description: "This is the value of the detail which the user has finalized.For example, if the key is location, then the value can be Ottawa, Ontario, Canada if that's what the user has selected.",},},},},}, required: ['trip_details', 'search_query'],},},},{ async: false, type: 'function', messages: [{ type: 'request-start', content: 'One moment while I look that up for you.',},{ type: 'reques t-response-delayed’ , content: "I'm still looking. I won’t be much longer.", timingMilliseconds: 10000,},], server: { url: 'https: / / northamerica-northeastl-jane-425814.cloudfunctions.net / travelfox-api',}, function: { name: 'information', description: 'This function should be called when the user has asked for information about a topic not covered by any other function. This includes requests for videos, photos or maps.', / / strict: true, parameters: { type: 'object', properties: { title: { type: 'string', description: 'A title for the information provided.', }, subtitle: { type: 'string', description: 'A subtitle for the information provided.’, }, description: { type: 'string', description: 'Detailed description of the topic. It is mandatory that this content be formatted using markdown.', }, priority: { type: 'array', items: { type: 'string', enum: ['photo', 'video', 'text', 'map'], }, description: 'The types of content that would help answer the user’s request in order of priority. Only the appropriate types of content should be included.' ,}, questions: { type: ’array’, description: ’A list of suggested questions the user can ask to help plan their trip.', items: { type: 'string', },}, map: {type: 'object', description: "Map containing POIs. This argument should only be returned if it would be helpful to the user's request.", properties: { zoom_level: { type: 'integer', description:'Map zoom level appropriate for the information being requested. The value should be an integer between 1 and 20, where 1 is the entire world, and 20 is a city block’, }, pois: { type: 'array', items: { type: 'object', properties: { lat_long: { type: 'array', items: { type: 'number', }, description: 'The latitude and longitude of the point of interest as an array of numbers. Example: [45.424721, -75.695000]', }, name: { type: 'string', description: 'The name of the POI.', }, }, required: ['lat_long', 'name'], }, }, }, required: ['zoom_level', 'pois'], },}, required: ['title', 'description', 'priority', ’questions'],},},},{ async: true, type: 'function', function: { name : 'getTripInspirationDetail' , description: 'This function is triggered when the user want to see more information about one of the trips returned by function.', / / strict: true, parameters: { type: ’object1, properties: { trip_id: { type: ’integer’, description: ’The trip_id of the trip selected by indicates that they the getTripInspiration the user.’,},}, required: [’trip_id’],},},},{ async: false, type: ’function’, server: { url: 'https: / / northamerica-northeastl-jane-425814.cloudfunctions.net / travelfox-api',}, messages: [{type: 'request-start', content: ”,}, ], function: { name: 'travelers’, description: 'This function extracts the traveler information, such as the number of travelers and if any children, their ages. If the user mentioned that they will be traveling with a spouse, husband or wife only, that counts as 2 adults. The "details_confirmed" parameter should be set to "false" until the user has confirmed the details.', parameters: { type: 'object', properties: { travelers: { type: 'array', items: { type: 'object', properties: { traveler_ type: { type: 'string', enum: ['adult', 'child'], description: 'The type of traveler. If a traveler is aged 18 or older, they are an adult, otherwise they are a child.', }, age: { type: 'integer', description: 'The age of the traveler.', }, relationship: { type: 'string', enum: ['self, 'spouse', 'child', 'parent', 'friend', 'colleague', 'other'], description: 'The relationship of this traveler to the main booker.', },}, required: ['traveler_type'],},}, details_confirmed: { type: 'string', enum: ['false', 'true'], description: 'This defines whether the traveler details have been confirmed by the user.',}, details_inferred: { type: 'string', enum: ['false', 'true'], description: 'This defines whether the number of travelers and their type have been inferred based on imprecise information from the user.', }, group_type: { type: 'string', enum: ['solo', 'couple', 'family', 'friends', 'business', 'mixed'], description: 'The primary composition of the travel group.’,},}, required: [’travelers'],},},},],}, transcriber: { provider: ’deepgram', model: 'nova-2-general', language: 'en-US', smartFormat: true,}, voice: {provider: '11 labs', voiceld: 'UbehS5uVuO9VGdy0ysUS', model: 'eleven_turbo_v2_5', inputMinCharacters: 50, }, silenceTimeoutSeconds: 60, maxDurationSeconds: 600, endCallFunctionEnabled: false, modelOutputlnMessagesEnabled: true, backgroundDenoisingEnabled: true, clientMessages: ['convers ation-update' ,'function-call','function-call-result’,'hang','model-output','speech-update','status-update','transcript','tool-calls’,'tool-calls-result','user-interrupted' ,'voice-input',],}Conclusion

[0113] Note that the expressions “at least one of A and B” or “at least one of A or B”, as used herein, are each interchangeable with the expression “A and / or B”. It refers to a list in which you may select A or B or both A and B. Similarly, “at least one of A, B, and C” or “at least one of A, B, or C”, as used herein, is interchangeable with “A and / or B and / or C” or “A, B, and / or C”. It refers to a list in which you may select: A or B or C, or both A and B, or both A and C, or both B and C, or all of A, B and C. The same principle applies for longer lists having a same format.

[0114] The scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure of the present invention, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed, that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized according to the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0115] Any module, component, or device exemplified herein that executes instructions may include or otherwise have access to a non-transitory computer / processor readable storage medium or media for storage of information, such as computer / processor readable instructions, data structures, program modules, and / or other data. A non-exhaustive list of examples of non- transitory computer / processor readable storage media includes magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, optical disks such as compact disc readonly memory (CD-ROM), digital video discs or digital versatile disc (DVDs), Blu-ray Disc™, or other optical storage, volatile and non-volatile, removable and non-removable media implemented in any method or technology, random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology. Any such non-transitory computer / processor storage media may be part of a device or accessible or connectable thereto. Any application or module herein described may be implemented using computer / processor readable / executable instructions that may be stored or otherwise held by such non-transitory computer / processor readable storage media.

[0116] Memory, as used herein, may refer to memory that is persistent (e.g. read-only-memory (ROM) or a disk), or memory that is volatile (e.g. random access memory (RAM)). The memory may be distributed, e.g. a same memory may be distributed over one or more servers or locations.

Claims

CLAIMS:

1. A computer-implemented method for travel planning using a voice and / or text enabled Al system, wherein the method includes: invoking at least one function combined with prompting at least one trained Al, including a large language model (LLM) running on specialized array processing hardware, to (i) produce queries for trip details and traveler details to be received from a user and (ii) distill, from user responses, the trip details and traveler details needed to populate a travel profile data structure; prompting the at least one trained Al to perform an Internet search based on the trip details and traveler details, including searching travel blogs and travel web sites, wherein the Internet search includes focusing the search to travel to improve relevance of the search results and return a decreased quantity of the search results; and automatically parsing information obtained from the Internet search to generate travel plan information.

2. The computer- implemented method of claim 1, wherein the LLM calls the function with arguments generated by the LLM, thereby causing execution of the function with the arguments generated by the LLM, and results of the function are used by the trained Al.

3. The computer- implemented method of claim 1, wherein the Internet search further comprises: search engine filtering in real time from at least thousands of tourist posted itineraries that overlap with a travel destination in the travel profile data structure, including comparing at least (i) the trip details and travel details of the travel profile data structure with (ii) trip details and travel details of posted itineraries; and wherein the computer-implemented method further comprises rank-ordering at least some of the posted itineraries for presentation to the user.

4. The computer-implemented method of claim 3, further including presenting, via a user interface, the posted itineraries including a visual display including images and interactive links representing three or more of the posted itineraries and a spoken summary of the posted itineraries represented on the visual display.

5. The computer-implemented method of claim 4, further including receiving, from the user, an interaction with a visual image of the presented visual images and in response to the received interaction, prompting the trained Al to respond with a structured travel plan based on the selected visual image.

6. The computer-implemented method of claim 1 , further including, in addition to the Internet search, searching a proprietary data set including scraped and curated travel blogs, web sites, and tourist-created itineraries.

7. The computer-implemented method of claim 1, wherein the trained Al autonomously triggers one or more specific user interface elements for display on a visual display based on one or more of a user input, a current stage of travel planning, and a user preference.

8. The computer-implemented method of claim 1, further including receiving a plurality of inputs from multiple users and updating a shared itinerary based on the received plurality of inputs with real-time synchronization.

9. The computer-implemented method of claim 8, wherein the trained Al manages permissions and roles corresponding to each user of the multiple users, thereby enabling user-specific levels of control over the shared itinerary.

10. The computer-implemented method of claim 1, further including receiving, from the user, a special inquiry during travel planning and, in response to the special inquiry, autonomously initiating at least one voice call between the trained Al and an external entity to obtain information related to the special inquiry.

11. The computer-implemented method of claim 10, further including providing the user with a real-time transcript of the at least one voice call and enabling the user to begin participating in the at least one voice call.

12. The computer- implemented method of claim 3, wherein the trained Al evaluates and ranks itineraries based on one or more user-defined criteria including a budget, a preferred activity, and a travel time.

13. A voice and / or text enabled Al system accessing a large language model (LLM) running on one or more processors and memory accessible by the processors, the memory loaded with computer instructions for travel planning, which computer instructions, when executed on the processors, implement actions comprising: invoking at least one function combined with prompting at least one trained Al, including the LLM running on specialized array processing hardware, to (i) produce queries for trip details andtraveler details to be received from a user and (ii) distill, from user responses, the trip details and traveler details needed to populate a travel profile data structure; prompting the at least one trained Al to perform an Internet search based on the trip details and traveler details, including searching travel blogs and travel web sites, wherein the Internet search includes focusing the search to travel to improve relevance of the search results and return a decreased quantity of the search results, and automatically parsing information obtained from the Internet search to generate travel plan information.

14. The voice and / or text enabled Al system of claim 13, wherein the LLM calls the function with arguments generated by the LLM, thereby causing execution of the function with the arguments generated by the LLM, and results of the function are used by the trained Al.

15. The voice and / or text enabled Al system of claim 13, wherein the implemented actions further include: search engine filtering in real time from at least thousands of tourist posted itineraries that overlap with a travel destination in the travel profile data structure, including comparing at least (i) the trip details and travel details of the travel profile data structure with (ii) trip details and travel details of posted itineraries; and rank-ordering at least some of the posted itineraries for presentation to the user.

16. The voice and / or text enabled Al system of claim 15, wherein the implemented actions further include presenting, via a user interface, the posted itineraries including a visual display including images and interactive links representing two or more of the posted itineraries and a spoken summary of the posted itineraries represented on the visual display.

17. The voice and / or text enabled Al system of claim 16, wherein the implemented actions further include receiving, from the user, an interaction with a visual image of the presented visual images and in response to the received interaction, prompting the trained Al to respond with a structured travel plan based on the selected visual image.

18. The voice and / or text enabled Al system of claim 13, wherein the trained Al autonomously triggers one or more specific user interface elements for display on a visual display based on one or more of a user input, a current stage of travel planning, and a user preference.

19. The voice and / or text enabled Al system of claim 13, wherein the implemented actions further include generating photorealistic images of a traveler and their companions at a selected trip destination before travel occurs, wherein the system uses generative Al to create visual representations based on photos of the traveler and companions, combined with destination-specific scenery.

20. A non-transitory computer readable storage medium impressed with computer program instructions for Al-based travel planning, which computer program instructions when executed implement a method comprising: invoking at least one function combined with prompting at least one trained Al, including a large language model (LLM) running on specialized array processing hardware, to (i) produce queries for trip details and traveler details to be received from a user and (ii) distill, from user responses, the trip details and traveler details needed to populate a travel profile data structure; prompting the at least one trained Al to perform an Internet search based on the trip details and traveler details, including searching travel blogs and travel web sites, wherein the Internet search includes focusing the search to travel to improve relevance of the search results and return a decreased quantity of the search results; and automatically parsing information obtained from the Internet search to generate travel plan information.

Images