Route planning method and device based on large model, intelligent agent and storage medium

Abstract

The invention provides an itinerary planning method and device based on a large model, an intelligent agent and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, large models, big data, intelligent e-commerce and the like. The travel route planning method based on the large model comprises the steps of obtaining demand information of a target object; an oral playing video and a picture element matched with the demand information are displayed, the oral playing video represents that the virtual object indicates the picture element through a specified action in the oral playing process, and the picture element represents extended travel demand information having a semantic difference with the demand information; and processing at least one of the extended journey demand information and the session information by using a large model to obtain journey planning information, the session information being input data of the target object for the oral video, and the extended journey demand information being determined based on the interaction behavior for the picture element.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, large models, big data, and smart e-commerce. Background Art

[0002] With the rapid development of Internet technology, users can input relevant text for inquiries through smart terminals such as smartphones. Relevant Internet platforms can use large models to perform semantic understanding of the text input by users and output feedback information. Summary of the Invention

[0003] The present disclosure provides a large-scale model-based itinerary planning method, device, intelligent agent, electronic device, and storage medium.

[0004] According to one aspect of the present disclosure, a large model-based itinerary planning method is provided, comprising: obtaining demand information of a target object;

[0005] Display the spoken video and screen elements that match the demand information. The spoken video represents the virtual object instructing the screen elements through specified actions during the spoken process. The screen elements represent the extended trip demand information that has semantic differences from the demand information.

[0006] The large model is used to process at least one of the extended itinerary demand information and the conversation information to obtain itinerary planning information. The conversation information is the input data of the target object for the spoken video, and the extended itinerary demand information is determined based on the interactive behavior for the screen elements.

[0007] According to another aspect of the present disclosure, a trip planning device based on a large model is provided, including: an acquisition module for acquiring demand information of a target object; a display module for displaying a spoken video and screen elements matching the demand information, wherein the spoken video represents that a virtual object indicates screen elements through specified actions during the spoken video process, and the screen elements represent extended trip demand information that has semantic differences from the demand information; and a trip planning information acquisition module for processing at least one of the extended trip demand information and session information using the large model to obtain trip planning information, wherein the session information is input data of the target object for the spoken video, and the extended trip demand information is determined based on the interactive behavior for the screen elements.

[0008] According to another aspect of the present disclosure, an artificial intelligence agent is provided, including: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large model based on the target task, and executing the large model-based itinerary planning method provided in an embodiment of the present disclosure by calling the large model to obtain output information; and an output module for outputting the output information obtained by the processing module.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the large model-based itinerary planning method provided in an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the large model-based itinerary planning method provided by an embodiment of the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the large model-based trip planning method provided in an embodiment of the present disclosure is implemented.

[0012] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0014] Figure 1 Schematically illustrates an exemplary system architecture to which a large model-based trip planning method and apparatus can be applied according to an embodiment of the present disclosure;

[0015] Figure 2 The flowchart of the trip planning method based on the large model according to the embodiment of the present disclosure is schematically shown;

[0016] Figure 3 The following schematically illustrates an application scenario of a large model-based itinerary planning method according to an embodiment of the present disclosure;

[0017] Figure 4 A schematic diagram of a service system for executing a large model-based itinerary planning method according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 5 A schematic diagram of a system architecture suitable for executing a large model-based trip planning method provided according to an embodiment of the present disclosure is shown;

[0019] Figure 6 A block diagram of a trip planning device based on a large model according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 7A block diagram schematically illustrates a structure of an artificial intelligence agent according to an embodiment of the present disclosure; and

[0021] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement the model-based trip planning method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0024] The inventors discovered that before embarking on travel, such as tourism or business trips, users often search online for information on attractions, accommodations, and transportation options to plan their itineraries. However, this process is often complex and can easily overlook important travel needs, resulting in inefficient itinerary planning and a failure to meet actual travel needs.

[0025] Embodiments of the present disclosure provide a large-scale model-based itinerary planning method, apparatus, intelligent agent, electronic device, and storage medium. The large-scale model-based itinerary planning method includes: obtaining target object demand information; displaying a spoken video and screen elements matching the demand information, wherein the spoken video represents a virtual object instructing screen elements through specified actions during the spoken video, and the screen elements represent extended itinerary demand information that is semantically different from the demand information; and using the large-scale model to process at least one of the extended itinerary demand information and conversation information to obtain itinerary planning information, wherein the conversation information is input data from the target object to the spoken video, and the extended itinerary demand information is determined based on interactive behavior with the screen elements.

[0026] According to an embodiment of the present disclosure, by displaying a spoken video that matches the demand information, the target object can naturally pay attention to the displayed screen elements by watching the instructions of the designated actions of the virtual object during the spoken video, so that the screen elements representing the extended itinerary demand information that has semantic differences with the demand information can be utilized. In this way, while watching the spoken video, the target object can be prompted by the immersive expression of the virtual object for the extended itinerary demand information to quickly determine the extended demand information by interacting with the screen elements, and to prompt the target object for the subsequent demand information required for the itinerary planning, so that the target object can express the subsequent demand for the itinerary planning by inputting conversation information, thereby improving the accuracy and efficiency of exploring the travel needs of the target object, avoiding the omission of the important travel needs of the target object, improving the accuracy and timeliness of the itinerary planning, and improving the user experience by providing the target object with an immersive interactive environment.

[0027] Figure 1 An exemplary system architecture to which a large model-based trip planning method and apparatus can be applied according to an embodiment of the present disclosure is schematically shown.

[0028] It should be noted that Figure 1 The examples shown are merely examples of system architectures applicable to embodiments of the present disclosure, to help those skilled in the art understand the technical content of the present disclosure. This does not imply that embodiments of the present disclosure cannot be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture applicable to the large-scale model-based itinerary planning method and apparatus may include a terminal device, but the terminal device may implement the large-scale model-based itinerary planning method and apparatus provided in the embodiments of the present disclosure without interacting with a server.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0031] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0032] Server 105 may be a server that provides various services, such as a background management server (for example only) that supports content browsed by users using terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device.

[0033] Server 105 can be a cloud server, also known as a cloud computing server or cloud host. This is a host product within a cloud computing service system that addresses the management difficulties and limited scalability of traditional physical hosts and VPS (Virtual Private Server) services. Server 105 can also be a server for a distributed system or a server integrated with blockchain.

[0034] It should be noted that the large model-based itinerary planning method provided in the embodiments of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the large model-based itinerary planning device provided in the embodiments of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0035] Alternatively, the large model-based itinerary planning method provided in the embodiments of the present disclosure may also generally be executed by the server 105. Accordingly, the large model-based itinerary planning apparatus provided in the embodiments of the present disclosure may generally be provided in the server 105. The large model-based itinerary planning method provided in the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and that is capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the large model-based itinerary planning apparatus provided in the embodiments of the present disclosure may also be provided in a server or server cluster that is different from the server 105 and that is capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0036] For example, after a user enters a request, terminal devices 101, 102, and 103 may obtain the request and then send the obtained request to server 105. Server 105 then displays a spoken video and visual elements matching the request. The server then invokes a large model to process at least one of the user's session information and extended itinerary request information entered via any of terminal devices 101, 102, and 103 to obtain itinerary planning information. Alternatively, a server or server cluster capable of communicating with terminal devices 101, 102, 103 and / or server 105 may display a spoken video and visual elements matching the request. The server then invokes a large model to process at least one of the user's session information and extended itinerary request information to obtain itinerary planning information.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0038] Figure 2 The flowchart of the large model-based trip planning method according to an embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the method includes operations S210 to S230.

[0040] In operation S210 , demand information of a target object is acquired.

[0041] In operation S220 , the spoken video and screen elements matching the demand information are displayed.

[0042] In operation S230 , at least one of the extended travel demand information and the session information is processed using the large model to obtain travel planning information.

[0043] According to an embodiment of the present disclosure, demand information may represent the target object's demand for travel planning, and the demand information may include any demand information related to travel planning, such as travel destination, travel transportation mode, etc.

[0044] It should be noted that the requirement information can be determined based on the requirement text input by the target object, but is not limited to this. The requirement information can also be determined based on other interactive behaviors of the target object, such as voice or image input by the target object. Alternatively, the requirement information can be determined based on the corresponding requirement information option determined by the target object's gesture interaction behavior.

[0045] According to an embodiment of the present disclosure, a spoken video represents a virtual object indicating screen elements through specified actions during the spoken process. The spoken video can be a video of a virtual object speaking about content related to the required information. For example, the spoken video is a video of a virtual object explaining information such as attractions and special videos of the destination city A. The virtual object in the spoken video can enhance the naturalness of the spoken explanation by performing actions during the spoken explanation, wherein the specified actions performed by the virtual object can indicate the screen elements displayed in the playback screen of the spoken video.

[0046] It should be noted that the virtual object can be a fictional character object such as an animated character, but is not limited thereto. The virtual object can also be a real object under the condition of obtaining authorization from the relevant real person. The embodiments of the present disclosure do not limit the specific type of virtual objects.

[0047] According to the embodiments of the present disclosure, the spoken video can be determined from a preset video library, or the spoken video can also be generated by driving a spoken video script corresponding to the demand information. The embodiments of the present disclosure do not limit the specific method of determining the spoken video, as long as it can represent the virtual object's explanation of travel information related to the demand information.

[0048] According to an embodiment of the present disclosure, the screen elements represent extended itinerary demand information that has semantic differences from the demand information. The extended itinerary demand information may be, for example, demand information that is missing from the demand information. For example, if the demand information is "travel to Province A", the extended itinerary demand information may include, for example, "1-day tour", "2 to 3-day tour" and other travel duration demand information that can represent the missing demand information. However, this is not limited to the above. The extended itinerary demand information may also be information that limits the semantic scope of the demand information. For example, if the demand information is "travel to Province A", the extended itinerary demand information may include, for example, demand information representing scenic spots and destinations in Province A, such as "City A" and "Town B".

[0049] It should be noted that the embodiments of this disclosure do not limit the specific semantic type used to represent the extended trip demand information, as long as it can be semantically distinct from the demand information. Screen elements can be represented by any form that can be displayed on the screen, such as text or icons, and the embodiments of this disclosure do not limit the display method of screen elements.

[0050] In one embodiment, a spoken video can be played on a display interface such as a display screen, and while a virtual object is explaining the extended travel demand information represented by the screen elements, multiple screen elements can be indicated by a gesture of pointing a finger to the bottom of the screen, so that the target object can naturally pay attention to the extended travel demand information represented by the screen elements through the designated actions of the virtual object while watching the spoken video, thereby enabling the target object to pay attention to potential travel demands in a timely manner while immersed in watching the video.

[0051] According to an embodiment of the present disclosure, the session information is input data of the target object for the spoken video, and the extended itinerary demand information is determined based on the interactive behavior for the screen elements.

[0052] In one embodiment, the target object can determine the required extended itinerary requirement information by performing an interactive behavior on a screen element. For example, the target object can perform a click interactive behavior on a target screen element representing "2 to 3 day tour" to determine the extended itinerary requirement information required for the target object's travel.

[0053] In one embodiment, the target object can also input conversation information that is the same as or different from the extended travel demand information represented by the picture elements based on the spoken explanation content and picture elements of the virtual object while watching the spoken video, so that the target object can input actual travel needs through flexible interaction.

[0054] According to an embodiment of the present disclosure, using a large model to process at least one of the extended trip demand information and the session information can be understood as using the large model to perform trip planning based on at least one of the extended trip demand information and the input session information selected by the target object's interactive behavior, so that the resulting trip planning information can match the target object's travel demand intention. The travel demand intention may include at least one of the first travel demand intention represented by the extended trip demand information, the second travel demand intention represented by the session information, and part or all of the demand intention represented by the demand information. In this way, the large model's more powerful semantic understanding capabilities can be utilized to output trip planning information that can accurately match the target object's travel demand, thereby improving the target object's travel planning efficiency.

[0055] It should be noted that the itinerary planning information can be represented by any type of data such as text, images, maps, videos, etc. The embodiments of the present disclosure do not limit the specific data types contained in the itinerary planning information.

[0056] In one embodiment, the itinerary planning information may be structured text information. For example, the structured text information may include text themes, travel destination sequence, transportation description information, ticket purchase link information, and other multi-type data represented by structured hierarchical relationships, so that the target object can prepare for the trip by browsing the text information and performing interactive actions on the link information in the text information, thereby improving the user experience.

[0057] It should be understood that the display screen for displaying spoken video and picture elements can be a display device of any type of terminal equipment such as a mobile phone, tablet computer, smart glasses, etc. The embodiments of the present disclosure do not limit the type of display device for displaying spoken video and picture elements.

[0058] According to an embodiment of the present disclosure, the extended trip demand information represents at least one of the following demand attributes: a trip duration attribute, a trip budget attribute, a waypoint demand attribute, a vehicle demand attribute, and a traveler attribute.

[0059] The travel duration attribute may represent the travel duration of the target object. The extended travel demand information representing the travel duration attribute may be, for example, “1-day tour”, “2 to 3-day tour”, etc.

[0060] The trip budget attribute can represent the budgeted amount of travel expenses. For example, the extended trip requirement information representing the trip budget attribute can be "1,000 to 2,000 yuan," "10,000 to 20,000 yuan," etc. It should be noted that the trip budget attribute can represent the specific amount of travel expenses, or can also represent the amount range that the target person can spend, or can also represent the target person's travel consumption level. For example, the extended trip requirement information can be "luxury travel" or "economy travel." The embodiments of this disclosure do not limit the amount attribute represented by the extended trip requirement information.

[0061] The waypoint requirement attribute represents the cities, villages, scenic spots, and other locations that the target person needs to pass through or stop at during their trip. It should be understood that a waypoint represents a location that the target person needs to reach, and can be any of the starting, intermediate, or final locations in the itinerary planning information.

[0062] The transportation demand attribute can represent any type of travel mode, such as high-speed rail, airplane, car or subway.

[0063] Traveler attributes can represent the number of people traveling, or they can also represent other traveler-related attributes such as the relationship between the traveler and the target object. For example, extended itinerary demand information representing traveler attributes can be "family travel" or "travel with parents".

[0064] It should be noted that the extended itinerary demand information can also represent other types of demand attributes, such as whether to carry pets, dining preferences, and other demand attributes. The embodiments of the present disclosure do not limit the specific types of demand attributes represented by the extended itinerary demand information.

[0065] In one embodiment, the extended itinerary demand information may represent multiple types of demand attributes. For example, the extended itinerary demand information may be "family tour for three persons" or "luxury tour for two persons." The extended itinerary demand information represented by the screen elements may also represent one or more types of demand attributes.

[0066] To facilitate explanation of the methods provided by the embodiments of the present disclosure, the following embodiments denote extended trip requirement information represented by screen elements as candidate extended trip requirement information, and extended trip requirement information determined by a target object's interaction with the screen element as target extended trip requirement information. It should be understood that, once a target object selects and confirms a target screen element, the target extended trip requirement information represented by the target screen element can be processed using the macro model to perform trip planning for the target object and output trip planning information.

[0067] According to an embodiment of the present disclosure, the interactive behavior for the screen elements includes at least one of the following: voice interactive behavior, gesture interactive behavior, eye movement interactive behavior, and touch operation behavior.

[0068] The voice interaction behavior can represent the voice and audio data output by the target object while watching the spoken video. The voice and audio data can be used to indicate the selection of the target screen element. For example, the voice interaction behavior can be the voice and audio data representing "3-person family tour" input by the target object.

[0069] In one embodiment, the voice audio data used for voice interaction behavior is processed by a voice recognition algorithm, and the obtained voice recognition information such as voice recognition text and voice recognition vector can be used to determine the target screen elements required by the target object from the screen elements, and then determine the target extended travel requirement information for the large model.

[0070] A touch operation can be understood as the interactive behavior of selecting a target screen element by operating a touch-sensitive component such as a touchscreen display or physical buttons on a smart terminal. Touch operations can include any touch-related operation, such as double-clicking, single-clicking, and selecting a frame. The embodiments of this disclosure do not limit the specific types of touch operations.

[0071] In one embodiment, the target object can determine the target extended itinerary demand information required by the target object representing the luxury travel demand attribute by performing a single-click operation on the target screen element representing "luxury travel" pointed to by the virtual object with a finger on the touch screen.

[0072] Gesture interaction behavior can be generated by touchable components without contacting the smart terminal. For example, the front camera of the smart terminal device can detect the finger movement trajectory of the target object to select and determine the target screen element that the target object needs to select.

[0073] Eye movement interaction can be understood as an interactive operation determined by detecting the target subject's eye focus. For example, a front-facing camera can be used to detect the target subject's eye movement towards the display screen to determine the target screen element that the target subject needs to select.

[0074] In one embodiment, the target screen element selected and confirmed by the target object can be determined by detecting any one or more of the target object's voice interaction behavior, gesture interaction behavior, eye movement interaction behavior, and touch operation behavior. As a result, while the target object is watching the spoken video, the virtual object's designated action instructions can be used to indicate the displayed screen element, allowing the target object to perform interactive operations based on the virtual object's designated action prompts for the required extended itinerary demand information, thereby achieving an immersive interactive experience of "watching and selecting at the same time," thereby improving the target object's interactive efficiency in selecting travel demand information and improving the accuracy of generated itinerary planning information.

[0075] In one embodiment, if no interaction with an on-screen element is detected while the target subject is watching the spoken video, it can be understood that the extended itinerary requirement information displayed by the on-screen element does not meet the target subject's travel needs. The target subject can also input text representing the travel needs as conversation information based on the spoken explanation of the virtual object or the extended itinerary requirement information prompted by the on-screen element. This allows the target subject to more accurately and quickly think through their actual travel needs through the spoken video and on-screen prompts, enabling in-depth exploration of the target subject's further travel intentions. By processing the conversation information through a large model to understand the target subject's needs, it can obtain itinerary planning information that accurately meets the target subject's actual needs.

[0076] In one embodiment, the large model can perform itinerary planning by targeting the extended itinerary demand information indicated by the conversation information and the target object's interactive behavior. Thus, under the multiple prompt conditions of the virtual object's oral explanation content and the screen elements of the specified action instructions, the target object's potential travel needs can be excavated based on the target object's diversified interactive behavior, thereby accurately and timely planning the itinerary planning information to quickly and accurately meet the user's travel planning needs.

[0077] It should be noted that the acquisition of data involved in any embodiment of the present disclosure, including but not limited to session information, extended itinerary request information, and demand information, is only authorized by the relevant user or organization, and the purpose of the data acquisition is clearly stated beforehand, namely, to push itinerary planning information to the target user. Necessary confidentiality measures are also implemented to prevent information leakage, comply with relevant laws and regulations, and do not violate public order and good morals.

[0078] According to an embodiment of the present disclosure, the designated action includes at least one of the following actions of the virtual object: eye action, head action, and gesture action.

[0079] In one embodiment, eye movements represent the indicated direction or indicated position of the virtual object's eyes. For example, the eye movement can be the virtual object's eyes looking at the screen elements below the spoken video. By using eye movements to indicate the screen elements displayed in the spoken video, the virtual object can prompt the target object to pay attention to the screen elements through relatively natural eye movements during the spoken explanation process, thereby enabling the target object to naturally and smoothly pay attention to the optional extended itinerary demand information during the immersive viewing of the virtual object's spoken explanation, so as to quickly determine the target extended itinerary demand information representing travel needs through interactive behavior. This can improve the accuracy and planning efficiency of the itinerary planning information output by the large model.

[0080] In one embodiment, head movements can be understood as turning or moving the virtual object's head, and can be used to indicate displayed screen elements. For example, a head-down gesture can be used to indicate screen elements below a spoken video, allowing the target user to promptly focus on available extended itinerary requirements. This provides an immersive interactive experience, improves the efficiency of exploring potential requirements, and enhances the efficiency and accuracy of generating itinerary planning information.

[0081] In one embodiment, gesture actions can be understood as actions of a virtual object to indicate screen elements. For example, a gesture action can be an action of sliding one hand to the left to point to the screen element display area on the left side of the spoken video, so as to achieve a larger movement amplitude to prompt the target object to pay attention to the optional extended itinerary demand information in a timely manner, thereby providing an immersive interactive experience, improving the efficiency of exploring potential demand information and the efficiency and accuracy of generating itinerary planning information.

[0082] In one embodiment, the designated action may include at least two of the virtual object's eye movements, head movements, and hand gestures, thereby combining the diverse actions displayed by the virtual object during the oral expression to enhance the prompting of the target object to promptly pay attention to the optional extended itinerary demand information, thereby providing an immersive interactive experience, and improving the efficiency of exploring potential demand information and the efficiency and accuracy of generating itinerary planning information.

[0083] Figure 3 The following schematically illustrates an application scenario diagram of a large model-based itinerary planning method according to an embodiment of the present disclosure.

[0084] like Figure 3 As shown, after a user enters their desired information, a voice-over video and screen elements matching the desired information are displayed on the terminal device's display interface 300. The voice-over video can represent a virtual object providing a spoken explanation of the tourist attractions in City A. During the virtual object's spoken explanation, the virtual object can use designated gestures to indicate the first screen element 311 and the second screen element 312 displayed on the left side of the display interface 300. The first screen element 311 and the second screen element 312 represent extended itinerary requirements, respectively, such as "In-Depth Experience" and "Scenic Spot Check-In." The user can interact with the first screen element 311 to confirm their selected extended itinerary requirement. They can also enter their actual travel requirements by entering conversation information in the input box 331 of the display interface 300. Furthermore, the display interface 300 can display a scenic spot video window 321, which plays video content introducing scenic spots in City A. This prompts the user to interact with the video while immersively viewing the voice-over video and to enter conversation information in the input box 331. The trained large model deployed on the server can perform itinerary planning by processing session information, target-expanded itinerary demand information, and demand information to obtain itinerary planning information that meets the user's actual travel needs.

[0085] In one embodiment, the screen elements are obtained by using a large language model to perform travel demand detection on demand information and historical preference attributes of the target object.

[0086] The historical preference information can be determined based on the target object's historical interaction behaviors, for example, the target object's historical preference attributes can be determined based on the target object's historical interaction behaviors such as likes, reposts, and comments. In one example, the target object's historical preference attributes can be determined based on the target object's historical satisfactory consumption resources.

[0087] By utilizing the large language model to process demand information and historical preference attributes, the large language model can more comprehensively and accurately understand the preferences and needs of the target user. This allows the output of the extended trip demand information corresponding to the screen elements to more accurately represent the target user's potential travel needs. This allows the target user to more accurately identify their potential needs by displaying screen elements representing the extended trip demand information. This allows the target user to quickly and easily provide feedback on their potential travel needs through interactive behaviors targeting these screen elements. By understanding the target user's extended trip demand information and demand information, the large model can generate more accurate travel planning information to meet the target user's actual needs.

[0088] In one embodiment, screen elements can be semantically similar to the demand information by performing a semantic similarity check against pre-set extended trip demand information. Pre-set extended trip demand information whose semantic similarity meets a predicted similarity interval threshold is selected as the extended trip demand information to be displayed. This allows for the rapid identification of extended trip demand information that exhibits semantic differences from the demand information, but not significantly so, enabling the rapid and timely display of corresponding screen elements for selection by the target user. This improves the efficiency of generating itinerary planning information while reducing computational overhead.

[0089] In one embodiment, the target subject inputs a demand information for "travel to city A." During the spoken video playback, multiple first screen elements representing similar extended itinerary demand information are displayed. For example, two first screen elements representing "1-day tour" and "2- to 3-day tour" may be displayed. During the spoken video, the virtual subject can indicate the location of the first screen elements in the display interface through gestures and eye movements. After the target subject performs a touch operation on the first screen element representing "2- to 3-day tour," it can be determined that "2- to 3-day tour" is the target extended itinerary demand information that meets the target subject's travel needs. By processing the demand information "travel to city A," the currently determined target extended itinerary demand information "2- to 3-day tour," and the target subject's historical preference attributes, the extended itinerary demand information represented by the new second screen elements is determined to be "budget 2,000-3,000 yuan" and "budget 5,000-8,000 yuan," respectively. The target subject can perform interactive operations on the second screen elements to determine the new target extended itinerary demand information.

[0090] It should be understood that during the playback of the spoken video, multiple rounds of picture elements can be continuously displayed to help the target object explore potential travel needs in the process of immersively watching the spoken video through the extended itinerary demand information represented by multiple rounds of picture elements, and then efficiently and accurately generate itinerary planning information by processing the demand information and multiple rounds of target extended itinerary demand information through a large model.

[0091] In another embodiment, a large model can be used to process demand information, conversational information input by the target user while watching the spoken video, and multiple rounds of target-expanded travel demand information to obtain itinerary planning information. This can be used to explore potential travel needs by providing the target user with diverse interaction methods, improving the accuracy and timeliness of travel demand exploration and enhancing itinerary planning efficiency.

[0092] According to an embodiment of the present disclosure, displaying the spoken video and screen elements that match the demand information may include: specifying a target area for action instructions in the spoken video and displaying the screen elements.

[0093] In one example, a spoken video clip related to a specified action is used to explain the extended information required for the representation of screen elements. For example, during the playback of the spoken video, a virtual object performs a specified gesture to indicate that the target area in the screen displays screen elements representing "1-day tour" and "2- to 3-day tour." The virtual object can also explain the following content while performing the gesture: "City A has many scattered attractions. You can choose a 1-day tour to fully experience attraction A, or choose a 2- to 3-day tour to experience multiple other attractions around attraction A."

[0094] By having a virtual object perform designated actions in a spoken video to indicate screen elements, and explaining the extended travel demand information represented by the screen elements during the execution of the designated actions, the target object's immersion in watching the spoken video can be further enhanced. While effectively prompting the target object, it stimulates the target object's subconscious mind to provide target extended travel demand information and conversation information that meet actual travel needs through interactive behavior, thereby improving the efficiency and accuracy of mining potential travel needs, and thereby improving the degree of match between itinerary planning information and the target object's actual travel needs.

[0095] In one embodiment, the spoken video is determined based on the following operations: using a large model to perform script editing based on demand information to obtain a spoken script; driving a virtual object based on the spoken text segment and action script data to obtain a spoken video.

[0096] Exemplarily, the spoken script includes a spoken text segment that matches the demand information and action script data that represents a specified action. The action script data that represents the specified action can drive the virtual object to perform the specified action to indicate the target area where the screen elements are displayed, and drive the virtual object to explain the screen elements indicated by the specified action through the spoken text segment, so as to improve the naturalness of the expression of the virtual object in the spoken video, and then stimulate the subconscious of the target object to propose travel needs, so that the large model can output more accurate itinerary planning information based on a full understanding of the conversation information representing potential actual needs and the expanded itinerary demand information.

[0097] In one embodiment, a multimodal large model can be used to understand the spoken text segments and action script data in the spoken script to drive the virtual object through the voice attributes, expression attributes, and action attributes in the spoken script, so that the virtual object in the spoken video can give a spoken explanation of the demand information while prompting the extended travel demand information represented by the screen elements through specified actions, expressions, and spoken expression content, thereby achieving dynamic collaboration between the graphical user interface (GUI) and the spoken video in the process of itinerary planning for the target object, improving the immersive experience effect of the target object interaction process, and thus enabling the conversation information and target extended itinerary demand information determined based on the interactive behavior to more accurately and efficiently represent the potential travel demand of the target object, so that the large model can be used to process the conversation information and target extended itinerary demand information representing the potential travel demand of the target object to accurately plan the itinerary and improve the interactive experience of the target object.

[0098] In one embodiment, the large-scale model-based itinerary planning method can also perform semantic conflict detection on the currently acquired session information and target extended travel demand information by utilizing the large-scale model to process the currently acquired session information and target extended travel demand information. If the semantic conflict detection result indicates a semantic logical conflict between multiple session information, multiple target extended travel demand information, or between the session information and the target extended travel demand information, a spoken text segment representing the semantic conflict detection result can be generated. The spoken video can then depict a virtual object explaining the spoken text segment representing the semantic conflict detection result, prompting the target subject to promptly modify and confirm the entered session information and target extended travel demand information. This allows the target subject to promptly identify and correct semantic conflicts between the provided travel demand information. The semantic detection capabilities of the large-scale model can then be used to assist the target subject in verifying their actual travel needs, thereby improving the efficiency of travel demand exploration and the logical coherence and accuracy of the generated itinerary planning information.

[0099] According to an embodiment of the present disclosure, the itinerary planning method based on a large model may further include: displaying itinerary planning information during the playback of the spoken video.

[0100] For example, the itinerary planning information output by the current large model can be displayed on the screen by scrolling during the playback of the spoken video. However, this is not limited to the above. The itinerary planning information output by the current large model can also be displayed through a floating window set on the spoken video screen during the playback of the spoken video, so that the target object can understand the content of the currently planned itinerary in a timely and clear manner. At the same time, by displaying the itinerary planning information output by the current large model, the target object can determine new travel demand information through interactive behaviors on screen elements or input operations of inputting session information. In this way, the large model can be used to process new session information and target extended itinerary demand information to update the current itinerary planning information, realize timely modification and proofreading of itinerary planning information, and improve the target object's satisfaction with travel needs.

[0101] According to an embodiment of the present disclosure, the itinerary planning method based on the large model may further include: displaying itinerary planning information when the playback of the spoken video is terminated.

[0102] In one example, the itinerary planning information output by the large model can be displayed after the spoken video is played, so that the target object can modify and confirm the actual travel needs according to the prompts of the spoken video and screen elements, and the large model can output more accurate itinerary planning information to improve the accuracy of the displayed itinerary planning information.

[0103] In one example, the currently generated itinerary planning information can also be displayed during the pause period of the spoken video playback, so that the target object can think deeply about the actual needs by carefully reading the currently generated itinerary planning information, and control the large model by inputting conversation information and interacting with subsequently generated screen elements to process at least one of the subsequently input conversation information and the target expanded itinerary demand information, so as to achieve a more accurate update of the current itinerary planning information and obtain itinerary planning information that meets the actual needs of the target object.

[0104] Figure 4 A schematic diagram of a service system for executing a large model-based itinerary planning method provided according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 4As shown, the service system includes a smart terminal, a server, and a strategy planning terminal. The smart terminal can be a terminal device used by the target object to watch the voice-over video and perform interactive behaviors on the voice-over video. The strategy planning terminal can deploy a large model for generating itinerary planning information. The target object initiates a planning request to the server by operating the smart terminal. The planning request carries the target object's demand information "family travel guide for City A". The server obtains the intent understanding result by calling the large language model to understand the demand information. The intent understanding result can represent that the target object's basic travel needs are to travel to City A, request a travel guide, and travel with family. Based on the intent understanding result, the extended itinerary demand information with semantic differences from the intent understanding result and the voice-over video that matches the intent understanding result are determined. By returning the voice-over video and the screen elements representing the extended itinerary demand information to the smart terminal, an immersive interactive experience is provided to the target object.

[0106] While watching the voice-over video, the target user can select continuously updated screen elements to determine target extended demand information and provide session information representing travel needs. This session information and target extended trip demand information are transmitted to the server as travel demand information for the target user. The server can integrate the demand information and travel demand information to supplement data for trip planning. Planning strategies are requested by sending a message containing the demand information and travel demand information to the strategy planning end.

[0107] The strategic planning end uses the trained large model to process demand and travel demand information to perform itinerary planning and obtain itinerary information. The strategic planning end returns the itinerary information to the server, which then sends a page containing the itinerary information to the smart terminal. The target user can browse the itinerary information on the smart terminal to meet their actual travel planning needs.

[0108] According to an embodiment of the present disclosure, the large model-based itinerary planning method further includes: displaying an extended video clip.

[0109] According to an embodiment of the present disclosure, displaying the extended video segment may include inserting the displayed extended video segment during the playback of the spoken video, or may also include displaying the extended video segment after the playback of the spoken video is completed.

[0110] According to an embodiment of the present disclosure, the extended video clip represents a virtual object giving a spoken explanation of content related to the extended itinerary demand information or the session information. For example, the spoken explanation content of the virtual object in the extended video clip represents the scenic spot introduction content, transportation methods and other scenic spot related data related to the session information or the target extended itinerary demand information.

[0111] Exemplarily, in the extended video clip, the virtual object can give a verbal explanation of at least one of the target extended itinerary demand information and conversation information determined by the target object, so as to prompt the target object through the verbal explanation content whether the conversation information and target extended itinerary demand information input through the interactive behavior actually meet the target object's needs, thereby achieving timely feedback on the interactive behavior that represents the target object's travel needs, enhancing the target object's immersive experience in the process of thinking about potential travel needs, and improving the accuracy and generation efficiency of the itinerary planning information output by the large model.

[0112] In one embodiment, the extended video segment may be a pre-generated preset video segment, and the preset video segment may be generated by driving a virtual object to perform oral explanation through a preset video segment script.

[0113] In one embodiment, the extended video segment is determined from a preset video segment set based on at least one of the extended trip requirement information and the conversation information. For example, the extended video segment that matches the target subject's current travel requirement can be obtained by matching the extended trip requirement information and at least one of the conversation information selected by the target subject with the respective themes of the segments in the preset video segment library.

[0114] In one example, the extended video segment can also be generated by processing at least one of the following: demand information, conversation information, and target extended trip demand information through a large model to output an extended video segment script. By processing the extended video segment script using a multimodal large model with video generation capabilities, a virtual object can be driven online to verbally explain the target extended trip demand information currently selected by the target. This allows the target to understand the relevant travel information in real time and consider whether to modify the target extended trip demand information. Alternatively, the target can further propose deeper travel demands through other interactive behaviors such as inputting conversation information, thereby providing an immersive interactive experience, improving the efficiency of exploring the target's potential travel demands for their partner, and enhancing the match between trip planning information and the target's actual travel needs.

[0115] According to an embodiment of the present disclosure, the displayed extended video clip can be used as a part of the spoken video, so that the currently displayed spoken video can be updated by displaying the extended video clip, and the corresponding screen elements can be displayed in the process of displaying the extended video clip to prompt the target object to continue to provide subsequent extended travel demand information and conversation information in an immersive interactive environment while watching the updated spoken video, so as to continuously explore the potential travel needs of the target object by displaying the extended video clip, and then obtain travel demand information that more accurately matches the actual travel needs of the target object based on the target object's interactive behavior and conversation information input behavior with respect to the screen elements displayed in the extended video clip, so that the large model can output travel planning information that more accurately matches the travel needs of the target object.

[0116] According to an embodiment of the present disclosure, the extended video segment is determined based on the following operations: according to at least one of the extended trip demand information and the session information, a script is edited using a large model to obtain an extended segment script; and a virtual object is driven based on the extended segment script to obtain an extended video segment.

[0117] In one embodiment, the macro model can process demand information, the target subject's selected extended itinerary demand information, and the input conversation information. This allows the macro model to fully understand the target subject's current actual travel needs and edit the extended segment script used to control the virtual object's spoken explanation. This allows the extended video segment to continuously explore the target subject's potential travel needs, improving the match between itinerary planning information and the target subject's actual needs.

[0118] In one embodiment, the demand information could be "City A Tour," the conversation information could be "Try to save as much as possible and visit several famous attractions in City A," and the target extended demand information could be "Family Tour for 3 people." By using the large model to process the demand information "City A Tour," the conversation information "Try to save as much as possible and visit several famous attractions in City A," and the target extended demand information "Family Tour for 3 people," an extended segment script is generated that can explain the diverse travel needs currently entered by the target subject. This script then drives the virtual subject to deliver oral explanations based on the extended segment script, thereby continuously exploring the target subject's potential travel needs.

[0119] In one embodiment, the extended segment script may also include extended specified action script data corresponding to the screen elements displayed during the extended video segment. This extended segment script can then be used to drive a virtual object to perform specified actions to indicate the screen elements displayed during the extended video segment, thereby providing a continuous immersive interactive experience. This allows the target object to continuously reflect on and provide feedback on potential travel needs, improving the accuracy and timeliness of the trip planning information output by the large model.

[0120] According to an embodiment of the present disclosure, there is a semantic conflict between at least two of the following: the extended video segment representation request information, the extended itinerary request information, and the first session information. The first session information is determined based on the target subject's first request. For example, the first request may be generated based on the target subject's interactive behavior of inputting session information, and the first request may carry the first session information.

[0121] Semantic contradictions can be understood as semantic conflicts between the travel needs of the target user through interactive input. For example, the demand information is "City A Tour," the first conversation information is "Budget 2000 yuan," and the target expanded demand information is "5-day luxury tour." It is difficult for the large model to output itinerary planning information that satisfies both the "Budget 2000 yuan" and "5-day luxury tour" requirements. Therefore, there is a semantic contradiction between "Budget 2000 yuan" and "5-day luxury tour."

[0122] For example, the trained large model can be used to process the demand information, the target extended trip demand information, and the first conversation information, so that the resulting semantic detection results determine that there is a semantic contradiction between at least two of the demand information, the extended trip demand information, and the first conversation information. If the semantic detection results indicate a semantic contradiction, the large language model can be used to perform script editing based on the semantic detection results to generate an extended script segment. This extended script segment can then be used to drive a virtual object to produce an extended video segment explaining the semantic contradiction.

[0123] In one embodiment, the trip planning method based on the large model further includes: responding to a second request of the target object, updating at least one of the demand information, the extended trip demand information, and the first session information according to the second request to obtain target trip demand information.

[0124] By extending the oral explanation content of the virtual object in the video clip, it is prompted that there are semantic conflicts between the multiple travel demands currently input by the target object, making it difficult for the itinerary planning information to simultaneously meet the multiple travel demands with semantic contradictions. The target object is helped to generate a second request through subsequent interactive behaviors on the screen elements or interactive operations of inputting session information, and the second session information carried in the second request and the new target extended itinerary demand information are used to update at least one of the current demand information, the extended itinerary demand information and the first session information, so as to correct the semantic conflicts between the multiple travel demand information and improve

[0125] For example, the first session information "budget 2,000 yuan" can be updated by the second session information "budget 20,000 yuan" carried in the second request, so that the target itinerary demand information obtained includes "5-day luxury tour", "city A tour" and "budget 20,000 yuan". In this way, the large model can generate itinerary planning information that meets the diverse travel needs of the target object without semantic conflicts by processing the target itinerary demand information including "5-day luxury tour", "city A tour" and "budget 20,000 yuan".

[0126] In one embodiment, the second request can be determined based on the target object's interaction with a screen element. For example, during playback of an extended video clip, if the target object interacts with a second screen element displayed during the extended video clip, a second request carrying extended itinerary requirement information represented by the second screen element can be determined. This allows the target object to select screen elements to generate extended itinerary requirement information representing new requirements during the process of further expanding the already selected extended itinerary requirement information with the virtual object, thereby improving the naturalness and fluency of the target object's requests and enhancing the efficiency of the large model in outputting itinerary planning information.

[0127] In one embodiment, the second request may be determined based on the second session information input by the target user. The target user may input the session information to propose new travel requirements based on a thought process, thereby providing the target user with diverse interaction methods to improve the efficiency of exploring the target user's travel needs and improving trip planning efficiency.

[0128] It should be noted that the target object may generate multiple first requests and second requests during the process of watching the spoken video, and the embodiments of the present disclosure do not limit the number of first requests or second requests.

[0129] According to an embodiment of the present disclosure, the itinerary planning method based on a large model may further include: using a large language model to perform demand detection on the target object based on at least one of the first extended information and the conversation information to obtain updated second extended information; and determining a screen element representing the second extended information.

[0130] According to an embodiment of the present disclosure, the extended travel demand information includes first extended information and second extended information.

[0131] In one example, the first extended information and the second extended information may have a semantic conflict. By using a large language model to process at least one of the first extended information and the conversation information to detect the actual travel needs of the target object, the updated second extended information can be made to represent the target object's revised demand intention, so that the target object can be guided to accurately select the extended travel demand information by targeting the updated screen elements by displaying the screen elements representing the second extended information, and new screen elements can be continuously displayed to help users explore travel needs. In turn, the target object can continue to correct the travel needs represented by the already determined extended travel demand information and conversation information through input operations or interactive behaviors with screen elements, thereby improving the accuracy of the travel planning information output by the large model.

[0132] In one example, there is a semantic hierarchical relationship between the first extended information and the second extended information. For example, the second extended information may represent the travel requirements at the next semantic level below the first extended information. For example, the first extended information may be "luxury travel," and the second extended information may be "budget 10,000 to 20,000 yuan" or "budget 20,000 to 50,000 yuan." By using the big model to process at least one of the first extended information and the session information to detect the needs of the target object, the second extended information at the next semantic level can be generated based on the travel needs currently reported by the target object. This allows the target object to more accurately explore their needs, improve the accuracy of travel demand intention feedback, and thereby improve the efficiency and accuracy of the itinerary planning information generated by the big model.

[0133] In one embodiment, while displaying the visual elements for the second extended information, the virtual object can also be driven to adjust the spoken content in real time within the extended video segment to explain the reason for the updated visual elements. For example, if the target object changes their budget requirement from "1,000 yuan" to "10,000 yuan," the virtual object will adjust the spoken content in real time to "Based on your budget, we recommend hotels with better accommodation experiences."

[0134] According to an embodiment of the present disclosure, processing at least one of the extended trip demand information and the session information using a large model includes: processing the demand information, the first extended information, and the second extended information using a large model.

[0135] By using a large model to process demand information, first extended information and second extended information with a semantic hierarchical relationship, it is possible to efficiently generate accurate itinerary planning information while more accurately exploring the travel needs of the target object, thereby improving itinerary planning efficiency.

[0136] According to an embodiment of the present disclosure, the itinerary planning method based on the big model further includes: using the big model to perform semantic understanding on the demand information to obtain basic itinerary planning information; and displaying the basic itinerary planning information.

[0137] According to the embodiments of the present disclosure, by utilizing a large model to perform semantic understanding of the needs input by the target object, and to generate and display basic itinerary planning information that matches the demand information, the target object can have a preliminary understanding of the preliminary itinerary planning strategy, and the target object can be helped to provide subsequent interactive behaviors for screen elements based on the planned basic itinerary planning information, as well as provide conversation information for characterizing travel needs, so as to improve the efficiency of exploring the travel intentions of the target object, and thereby improve the efficiency of the large model in generating itinerary planning information.

[0138] Figure 5 The following schematically shows a system architecture diagram suitable for executing the large model-based itinerary planning method provided according to an embodiment of the present disclosure.

[0139] like Figure 5 As shown, the system architecture includes an intent detection module 501, a demand exploration module 502, and an itinerary planning module 503. The intent detection module 501 includes an intent detection component, a prompt engineering component, and an intelligent retrieval component. The intent detection component can perform intent detection on the demand information "I want to travel in City A for 3 days with a budget of 3,000 yuan" and obtain an intent detection result. The prompt engineering component determines the corresponding prompt information based on the intent detection result and sends the prompt information to the itinerary planning module 503. The intelligent retrieval component can retrieve attraction data and transportation data related to City A based on the intent detection result. The attraction data may include attraction-related information such as opening hours and attraction reservation links. Transportation data may include real-time traffic congestion conditions and transportation types. The attraction data, transportation data, and other travel demand-related information retrieved by the intelligent retrieval component can also be sent to the itinerary planning module 503.

[0140] The demand exploration module 502 includes a loading component, n screen element components, and a demand integration component. The loading component can determine the spoken video and n screen element components that match the demand information, and load the spoken video in the display interface. During the playback of the spoken video, the screen elements displayed in the display interface can be dynamically adjusted in the order of the first screen element component to the nth screen element component according to the display order of the N screen element components in the display strategy. The target object can perform interactive behaviors on the screen elements while watching the spoken video, or input conversation information to transmit travel demand information to the demand exploration module 502. The demand integration component obtains travel demand information and calls a large language model to perform semantic contradiction detection on multiple travel demand information to generate new screen elements based on the semantic contradiction detection results, prompting the target object to modify the travel demand information by performing interactive behaviors on the screen elements or inputting corrected conversation information. The demand integration component sends the target object's target itinerary demand information to the itinerary planning module 503.

[0141] The itinerary planning module 503 includes a itinerary planning component and a itinerary planning information display component. The itinerary planning component uses the macro model to process prompt information, travel demand-related information, and target itinerary demand information. This macro model is used to plan itineraries based on the target user's travel needs and intentions, generating itinerary planning information. The itinerary planning information display component generates a itinerary planning display page based on the itinerary planning information, allowing the target user to view detailed itinerary planning information related to the itinerary planning strategy.

[0142] In one embodiment, a service system for executing the large model-based itinerary planning method of the embodiment of the present disclosure includes an interaction module, a task execution module, a GUI fusion module, and a data management module.

[0143] The interaction module includes a multimodal driving engine built based on a large multimodal model and an intent parser built based on a large language model.

[0144] The multimodal driving engine utilizes a large multimodal model that integrates natural language processing (NLP), text-to-speech (TTS), motion capture, and other algorithms. This allows virtual objects to be driven by their expressions, movements, and spoken audio data, achieving real-time synchronization of expressions, movements, and speech in the spoken video. For example, if a target user enters a request for "quality travel," the virtual object can nod to confirm and then perform gestures to point to the corresponding target area on the screen.

[0145] The intent parser uses the semantic analysis capabilities of the large language model to identify the target user's needs, the number of days, budget, and travel style contained in the conversation information, and generates prompt information that controls the large model to generate itinerary planning information. For example, the prompt information can be "Generate a 3-day itinerary for City A."

[0146] The task execution engine uses the big model to execute the itinerary planning process. For example, it can combine the target itinerary demand information with intelligent retrieval to obtain real-time data related to the trip, such as the opening hours of tourist attractions and traffic congestion conditions. The big model then processes this real-time data and the target itinerary demand information to perform itinerary planning, generating personalized itinerary information for the target. This itinerary planning information may include, for example, multiple recommended attractions, as well as optimal travel routes and transportation transfers between them.

[0147] The task execution engine can also use the large language model to detect semantic inconsistencies in conversation information, target expansion itinerary requirements, and demand information in real time. It can also detect semantic inconsistencies in the itinerary planning information generated by the large model to obtain planning inconsistency detection results. Planning inconsistency detection results can indicate time conflicts in the itinerary planning information, such as closures of itinerary attractions, or budget overruns caused by the itinerary spending. Reinforcement learning can be used to optimize the large model's itinerary planning process to improve the accuracy of itinerary planning information.

[0148] The GUI fusion module renders the spoken video and the screen elements synchronously to achieve dynamic overlay of screen elements for providing interactive options during the playback of the spoken video, thus enabling the target object to have an immersive interactive experience of "watching and selecting".

[0149] The data management module is used to analyze the interactive behavior of the target object and to determine the preferences of the target object in order to optimize the itinerary planning capabilities of the large model.

[0150] Figure 6 A block diagram of a large model-based trip planning device according to an embodiment of the present disclosure is schematically shown.

[0151] like Figure 6 As shown, the large model-based itinerary planning device 600 includes: an acquisition module 610 , a display module 620 and a itinerary planning information acquisition module 630 .

[0152] The acquisition module 610 is used to acquire the demand information of the target object.

[0153] The display module 620 is used to display the spoken video and screen elements that match the demand information. The spoken video represents the virtual object indicating the screen elements through specified actions during the spoken process, and the screen elements represent the extended travel demand information that has semantic differences from the demand information.

[0154] The itinerary planning information acquisition module 630 is used to use the large model to process at least one of the extended itinerary demand information and the conversation information to obtain itinerary planning information. The conversation information is the input data of the target object for the spoken video, and the extended itinerary demand information is determined based on the interactive behavior with respect to the screen elements.

[0155] According to an embodiment of the present disclosure, the display module 620 includes a first display submodule.

[0156] The first display submodule is used to display screen elements in a target area indicated by a specified action in the spoken video, wherein the spoken video clip related to the specified action is used to explain the extended demand information represented by the screen elements.

[0157] According to an embodiment of the present disclosure, the spoken video is determined based on the following operations: using a large model to perform script editing based on the demand information to obtain a spoken script, which includes a spoken text segment that matches the demand information and action script data that represents a specified action; driving a virtual object based on the spoken text segment and the action script data to obtain a spoken video.

[0158] According to an embodiment of the present disclosure, the itinerary planning device 600 further includes: an extended video clip display module.

[0159] The extended video clip display module is used to display the extended video clip, which represents the virtual object's oral explanation of the content related to the extended itinerary demand information or conversation information.

[0160] According to an embodiment of the present disclosure, the extended video segment is determined based on the following operations: according to at least one of the extended trip demand information and the session information, a script is edited using a large model to obtain an extended segment script; and a virtual object is driven based on the extended segment script to obtain an extended video segment.

[0161] According to an embodiment of the present disclosure, the extended video segment is determined from a preset video segment set according to at least one of the extended trip demand information and the session information.

[0162] According to an embodiment of the present disclosure, there is a semantic contradiction between at least two of the extended video segment representation requirement information, the extended itinerary requirement information, and the first conversation information, and the first conversation information is determined based on the first request of the target object; the itinerary planning device 600 also includes a first obtaining module.

[0163] The first obtaining module is used to respond to the second request of the target object, update the demand information, extend the trip demand information and at least one of the first session information according to the second request, and obtain the target trip demand information. The trip planning information is determined by processing the target trip demand information using the large model.

[0164] According to an embodiment of the present disclosure, the second request is determined based on at least one of the following ways: based on an interactive behavior directed to a screen element; or based on second session information input by a target object.

[0165] According to an embodiment of the present disclosure, the itinerary planning device 600 further includes: a second obtaining module and a first determining module.

[0166] The second obtaining module is used to use the large language model to perform demand detection on the target object according to at least one of the first extended information and the session information to obtain updated second extended information, where the extended trip demand information includes the first extended information and the second extended information.

[0167] The first determining module is configured to determine a screen element representing the second extended information.

[0168] According to an embodiment of the present disclosure, the travel planning information obtaining module 630 includes a first processing submodule.

[0169] The first processing submodule is used to process the demand information, the first extended information and the second extended information using the large model.

[0170] According to an embodiment of the present disclosure, the designated action includes at least one of the following actions of the virtual object: eye action, head action, and gesture action.

[0171] According to an embodiment of the present disclosure, the screen elements are obtained by using a large language model to perform travel demand detection on demand information and historical preference attributes of the target object.

[0172] According to an embodiment of the present disclosure, the interactive behavior for the screen elements includes at least one of the following: voice interactive behavior, gesture interactive behavior, eye movement interactive behavior, and touch operation behavior.

[0173] According to an embodiment of the present disclosure, the itinerary planning information obtaining module 630 further includes: a first display submodule and a second display submodule.

[0174] The first display submodule is used to display the itinerary planning information during the playback of the spoken video.

[0175] The second display submodule is used to display the itinerary planning information when the playback of the spoken video is terminated.

[0176] According to an embodiment of the present disclosure, the itinerary planning device 600 further includes: a third obtaining module and a basic itinerary planning information display module.

[0177] The third acquisition module is used to use the large model to semantically understand the demand information and obtain basic itinerary planning information.

[0178] The basic itinerary planning information display module is used to display basic itinerary planning information.

[0179] According to an embodiment of the present disclosure, the extended trip demand information represents at least one of the following demand attributes: a trip duration attribute, a trip budget attribute, a waypoint demand attribute, a vehicle demand attribute, and a traveler attribute.

[0180] Figure 7 The structural block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is schematically shown.

[0181] In the embodiments of the present disclosure, Figure 7 As shown, the AI agent 700 may include an input module 710 , a processing module 720 and an output module 730 .

[0182] The input module 710 is used to receive input information.

[0183] The processing module 720 is used to determine the target task based on the input information received by the input module, determine the big model based on the target task, and execute the model-based itinerary planning method provided according to the embodiment of the present disclosure to output information by calling the big model.

[0184] The output module 730 is used to output the output information obtained by the processing module.

[0185] According to an embodiment of the present disclosure, the input module 710 is responsible for receiving or perceiving information such as queries, requests, instructions, signals, or data from the outside world (e.g., a user or the external environment) and converting it into a format that can be understood and processed by the AI agent 700. The input module 710 is the primary link for the AI agent 700 to interact with the outside world, enabling the AI agent 700 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0186] In an example, the input module 710 may input the demand information, extended trip demand information, and session information described above.

[0187] In the example, the processing module 720 is the core support for the AI agent 700 to handle complex tasks. The processing module 720 can execute the model-based trip planning method described above.

[0188] In this example, the performance of processing module 720 may be closely related to the large model underlying AI agent 700. To fully leverage the capabilities of the large model, the internal structure of processing module 720 may be designed to be highly configurable and extensible to handle a variety of different types of tasks and requirements in real-world scenarios.

[0189] In the example, after the AI agent 700 obtains the demand information, extended itinerary demand information and conversation information, the processing module 720 can use the large model to process the demand information, extended itinerary demand information and conversation information to obtain itinerary planning information, and pass the itinerary planning information to the output module 730.

[0190] Understandably, while large language models possess excellent language understanding and generation capabilities, like humans, they are limited in the tasks they can perform without tools. However, when AI Agent 700 is empowered with tool-based capabilities, it can perform tasks such as mathematical calculations using a calculator, data analysis using Python, and weather forecasting using search engines.

[0191] In an example, the output module 730 may output the trip planning information described above.

[0192] The AI agent 700 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0193] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0194] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0195] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described above.

[0196] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described above.

[0197] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement a model-based trip planning method according to an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example purposes only and are not intended to limit the implementation of the present disclosure as described and / or claimed herein.

[0198] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0199] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0200] Computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 801 performs the various methods and processes described above, such as the model-based trip planning method. For example, in some embodiments, the model-based trip planning method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the model-based trip planning method described above may be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform the model-based trip planning method via any other suitable means (e.g., via firmware).

[0201] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0202] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0203] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0204] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0205] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0206] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0207] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0208] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A trip planning method based on a large model, comprising: Obtain the target object's demand information; Displaying a spoken video and screen elements that match the demand information, wherein the spoken video represents a virtual object instructing the screen elements through a specified action during the spoken video, and the screen elements represent extended trip demand information that has a semantic difference from the demand information; The large model is used to process at least one of the extended itinerary demand information and the session information to obtain itinerary planning information, wherein the session information is input data of the target object for the voice-over video, and the extended itinerary demand information is determined based on the interactive behavior for the screen elements.

2. The method according to claim 1, wherein The display of spoken video and screen elements matching the demand information includes: The screen element is displayed in the target area indicated by the designated action in the spoken video, wherein the spoken video segment related to the designated action is used to explain the extended demand information represented by the screen element.

3. The method according to claim 1 or 2, wherein: The oral video is determined based on the following operations: Using the large model to perform script editing on the demand information to obtain a spoken script, wherein the spoken script includes a spoken text segment matching the demand information and action script data representing the specified action; The virtual object is driven based on the spoken text segment and the action script data to obtain the spoken video.

4. The method according to claim 1, wherein The method further comprises: An extended video clip is displayed, where the extended video clip represents the virtual object speaking and explaining content related to the extended travel demand information or the conversation information.

5. The method according to claim 4, wherein The extended video segment is determined based on the following operations: Editing a script using the large model according to at least one of the extended trip demand information and the session information to obtain an extended segment script; and The virtual object is driven based on the extended segment script to obtain the extended video segment.

6. The method according to claim 4, wherein: The extended video segment is determined from a preset video segment set according to at least one of the extended trip requirement information and the session information.

7. The method according to any one of claims 4 to 6, wherein The extended video segment indicates that there is a semantic contradiction between at least two of the demand information, the extended trip demand information, and first conversation information, where the first conversation information is determined based on a first request of the target object; the method further comprising: In response to a second request from the target object, at least one of the demand information, the extended trip demand information, and the first session information is updated according to the second request to obtain target trip demand information, and the trip planning information is determined by processing the target trip demand information using the large model.

8. The method according to claim 7, wherein: The second request is determined based on at least one of the following methods: Determining based on the interactive behavior for the screen element; The method is determined based on second session information input by the target object.

9. The method according to claim 1, wherein: The method further comprises: Using a large language model, performing demand detection on the target object based on at least one of the first extended information and the session information to obtain updated second extended information, where the extended trip demand information includes the first extended information and the second extended information; and A picture element representing the second extended information is determined.

10. The method according to claim 9, wherein: Processing at least one of the extended trip demand information and the session information using the large model includes: The demand information, the first extended information and the second extended information are processed using a large model.

11. The method according to claim 1, wherein The specified action includes at least one of the following actions of the virtual object: Eye movements, head movements, hand movements.

12. The method according to claim 1, wherein The screen elements are obtained by performing travel demand detection on the demand information and the historical preference attributes of the target object using a large language model.

13. The method according to claim 1, wherein The interactive behavior for the screen element includes at least one of the following: Voice interaction behavior, gesture interaction behavior, eye movement interaction behavior, and touch operation behavior.

14. The method according to claim 1, wherein The method further comprises: During the playback of the spoken video, display the itinerary planning information; or When the playback of the spoken video is terminated, the itinerary planning information is displayed.

15. The method according to claim 1, wherein The method further comprises: Using the large model to perform semantic understanding on the demand information to obtain basic itinerary planning information; and The basic itinerary planning information is displayed.

16. The method according to claim 1, wherein The extended trip requirement information represents at least one of the following requirement attributes: Journey duration attributes, journey budget attributes, transit point demand attributes, transportation demand attributes, and traveler attributes.

17. A large-scale model-based itinerary planning device, comprising: Acquisition module, used to obtain the target object's demand information; a display module, configured to display a spoken video and screen elements matching the demand information, wherein the spoken video represents a virtual object indicating the screen elements through a specified action during the spoken video, and the screen elements represent extended trip demand information that has a semantic difference from the demand information; A trip planning information acquisition module is used to use a large model to process at least one of the extended trip demand information and the session information to obtain trip planning information, wherein the session information is the input data of the target object for the voice-over video, and the extended trip demand information is determined based on the interactive behavior for the screen elements.

18. An artificial intelligence agent, comprising: An input module, used for receiving input information; a processing module, configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and execute the method according to any one of claims 1 to 16 by calling the large model to obtain output information; An output module is used to output the output information obtained by the processing module.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 16.

21. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images