Trip planning method based on multi-modal large model, electronic device and storage medium

Abstract

This invention provides a trip planning method, electronic device, and storage medium based on a multimodal large model. The method includes: acquiring text and image content specified by a target user from a network platform; inputting the text and image content into a text and image parsing model to perform joint semantic understanding of the text and image information in the content, extracting structured trip data containing location information; inputting the structured trip data into a location verification model to standardize the location information by calling a map service interface, obtaining standard location data; inputting the standard location data into a path planning model to calculate the path between various locations by calling a navigation service interface, generating executable trip data containing a sequence of waypoints and navigation instructions; and returning the executable trip data to the target user. This invention can generate navigable trip planning data using text and image content from a network platform.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method for travel planning based on a multimodal large model, an electronic device, and a storage medium. Background Technology

[0002] With the popularization of internet technology and mobile smart terminals, the online travel industry has developed rapidly, and a large number of travel notes have emerged on online platforms. These notes contain rich text and image information and real travel experiences, serving as an important reference for users to plan personalized trips.

[0003] However, existing trip planning tools mainly rely on structured data input. Users need to filter and organize the mixed text and images and various formats of travel notes on online platforms in advance, and extract the effective information to manually input into the trip planning tool. The process is time-consuming and laborious, and it is easy to miss key information. Summary of the Invention

[0004] This invention provides a multimodal large model-based trip planning method, electronic device, and storage medium, which can generate navigable trip planning data using text and image information from a network platform, enabling convenient and personalized trip planning.

[0005] In a first aspect, the travel planning method based on a multimodal large model provided in the embodiments of the present invention includes: The system retrieves text and image content specified by the target user from the online platform; inputs this content into a text and image parsing model to perform joint semantic understanding of the text and image information, extracting structured trip data containing location information; inputs this structured trip data into a location verification model to standardize the location information by calling a map service interface, obtaining standard location data; inputs this standard location data into a route planning model to calculate the path between various locations by calling a navigation service interface, generating executable trip data containing waypoint sequences and navigation instructions; and finally returns the executable trip data to the target user.

[0006] Secondly, the travel planning device based on a multimodal large model provided in the embodiments of the present invention includes: The acquisition module is used to retrieve the text and image content specified by the target user from the network platform. The extraction module is used to input the text and image content into the text and image parsing model, so as to use the text and image parsing model to perform joint semantic understanding on the text and image information in the text and image content, and extract structured travel data containing location information; The standardization module is used to input structured trip data into the location verification model, so that the location verification model can call the map service interface to standardize the location information and obtain standard location data. The generation module is used to input standard location data into the route planning model, so that the route planning model can call the navigation service interface to calculate the path between various locations, generate executable trip data containing waypoint sequences and navigation instructions, and return the executable trip data to the target user.

[0007] Thirdly, the electronic device provided in the embodiments of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multimodal large model-based travel planning method as described in any embodiment of the present invention.

[0008] Fourthly, the computer-readable storage medium provided in the embodiments of the present invention stores computer instructions thereon, the computer instructions being used to cause a processor to execute and implement the multimodal large model-based travel planning method as in any embodiment of the present invention.

[0009] Fifthly, the computer program product provided in the embodiments of the present invention includes a computer program that, when executed by a processor, implements the multimodal large model-based travel planning method as described in any embodiment of the present invention.

[0010] In this embodiment of the invention, obtaining the text and image content specified by the target user from the network platform can provide personalized reference information for itinerary planning. Inputting the text and image content into a text and image parsing model allows for joint semantic understanding of the text and image information, extracting structured itinerary data containing location information. This improves information acquisition efficiency and enhances the accuracy of location information by performing correlation analysis on unstructured text and image content, preventing information omissions. Inputting the structured itinerary data into a location verification model allows for standardization of location information by calling a map service interface, obtaining standard location data that ensures accuracy and usability. Inputting the standard location data into a path planning model allows the model to call a navigation service interface to calculate paths between various locations, generating executable itinerary data containing waypoint sequences and navigation instructions. Returning the executable itinerary data to the target user avoids the user repeatedly switching between navigation software and itinerary plans, improving user experience. Attached Figure Description

[0011] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a multimodal large model-based route planning method provided in an embodiment of the present invention. Figure 2 This is another flowchart illustrating the route planning method based on a multimodal large model provided in this embodiment of the invention; Figure 3 This is an architecture diagram of the itinerary planning system provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a travel planning device based on a multimodal large model provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1This is a flowchart illustrating a multimodal large-scale model-based trip planning method provided in this embodiment of the invention. This method is applicable to scenarios where navigable trip planning data is generated using text and image information from a network platform. The multimodal large-scale model-based trip planning method can be executed by a multimodal large-scale model-based trip planning device provided in this embodiment, which can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer or server. The following embodiment illustrates the integration of the multimodal large-scale model-based trip planning device into an electronic device. See also... Figure 1 The trip planning method based on a multimodal large model in this embodiment may include the following steps: Step 101: Obtain the text and image content specified by the target user from the online platform.

[0016] Image and text content refers to user-generated content (UGC) containing text and images, specified by the target user on an online platform. This content is typically an unstructured travel guide or travel notes, serving as reference information for the target user's itinerary planning. The online platform is the source of the image and text content. Online platforms include, but are not limited to, various social media platforms and travel service platforms. Specifically, users can flexibly choose image and text content in different formats from various online platforms according to their personal preferences, and specify the content through various means such as sharing links. By parsing the links in the image and text content, the source platform can be extracted. By calling the content retrieval interface provided by the source platform or using web page parsing technology to parse the web page content corresponding to the link in the image and text content, the image and text content specified by the user can be retrieved from the source platform.

[0017] Step 102: Input the text and image content into the text and image parsing model to perform joint semantic understanding of the text and image information in the text and image content, and extract structured travel data containing location information.

[0018] The text-image parsing model is a multimodal large language model capable of simultaneously understanding text and image content, extracting itinerary-related information, and outputting structured data. Specifically, the text-image parsing model can be obtained by fine-tuning a general multimodal large language model using a travel itinerary information corpus.

[0019] Text information refers to natural language information within text and image content. For example, text information may include, but is not limited to, itinerary descriptions, schedules, transportation methods, attraction introductions, precautions, budget information, and accommodation recommendations. Image information refers to visual resources within text and image content. For example, image information may include actual photos of attractions, ticket photos, map screenshots, food photos, and images of accommodations.

[0020] Specifically, by using a multimodal large language model to perform joint semantic understanding of textual and image information, we can capture the implicit information in the image information and perform complementary correlation analysis with the text information, avoiding omissions and recognition errors that may occur when manually screening information or using single-modal textual information.

[0021] Structured itinerary data is a structured data object formed after extracting and organizing text and image content. Specifically, the fields to be included in the structured itinerary data can be defined according to preset prompts. The preset prompts and text and image content are input into the text and image parsing model, so that the model can extract features from the text information and image information respectively, and perform cross-validation and correlation feature fusion of text features and image features to fill the fields defined by the preset prompts, and output structured itinerary data. For example, structured itinerary data can be "{Destination: Province A; Number of travel days: 5; Daily itinerary: [{Number of days: 1; Location list: [{Location name: History Museum of Province A; Location description: ...; Suggested visit duration: 3 hours} ... . ...

[0022] Step 103: Input the structured trip data into the location verification model, and use the location verification model to call the map service interface to standardize the location information and obtain standard location data.

[0023] A location verification model is an intelligent agent used to validate and standardize location information in structured travel data. A location verification model can be a rule engine and / or a large language model. A map service interface refers to the application programming interface (API) exposed by a map service provider, which can provide map data services such as point-of-interest (POI) queries and coordinate transformations.

[0024] Location information refers to descriptive information about travel points mentioned in the text and image content. For example, travel points can be geographical entities mentioned in the text and image content, such as scenic spots, restaurants, hotels, transportation hubs, and shopping areas. Location information can include location names and descriptions. For example, the location name could be "Scenic Area A," and the location description could be "Located in the city center of City B."

[0025] Standard location data consists of point-of-interest (POI) location data retrieved from a map service interface based on location information, and then standardized using a location verification model to obtain valid travel point information. Standard location data can include the standard name, latitude and longitude, and detailed address of the travel point.

[0026] Specifically, standardizing location information can include normalizing location names based on location descriptions to obtain clearly defined standardized names. For example, location names and descriptions can be input into a large language model, and combined with pre-defined prompts, the model can perform semantic understanding and map database retrieval enhancement to generate standardized names. The map service interface can then be called to query the Point of Interest (POI) database to check if the standardized name corresponds to a POI, and retrieve the POI's location data. Invalid location information is removed from the structured itinerary data. Finally, the latitude and longitude coordinates of each location are converted to the same coordinate system. This ensures the accuracy and reliability of location information in the structured itinerary data, guaranteeing the smooth progress of subsequent route planning.

[0027] Step 104: Input the standard location data into the route planning model so that the route planning model can call the navigation service interface to calculate the path between the locations and generate executable trip data containing waypoint sequences and navigation instructions.

[0028] A route planning model is an intelligent agent used to process standard location data to generate executable itinerary data. It can be a rule engine and / or a large language model. Executable itinerary data is a navigable itinerary planning scheme, specifically including a waypoint sequence and navigation instructions. The waypoint sequence refers to the sequence of locations arranged in chronological order of visit time and the paths between these locations; each location may also include arrival and departure times. The paths between locations are returned by the navigation service interface, showing the path between a specific starting point and destination, including path geometry, path distance, and path duration. Navigation instructions are Uniform Resource Identifiers that directly invoke the navigation service. Navigation instructions correspond to the path between two waypoints in the waypoint sequence; clicking a navigation instruction triggers navigation from the current waypoint to the next waypoint.

[0029] Specifically, the route planning model first parses the structured trip data to obtain recommended transportation methods and recommended stay durations between each waypoint. If the structured trip data does not explicitly provide this information, it can also intelligently recommend routes based on user-input trip preferences, obtaining recommended stay durations and recommended transportation methods for each waypoint. Then, combining the recommended transportation methods and standard location data, it calls the navigation service interface to calculate the optimal path between each waypoint and returns the corresponding navigation instructions. Finally, it generates a trip timeline by combining the path information and recommended stay durations between each waypoint. Finally, it assembles the trip timeline and path to obtain executable trip data containing a sequence of waypoints and navigation instructions. This allows users to conveniently access navigation services for multiple consecutive paths within a waypoint sequence without switching between trip planning and navigation software, improving user experience.

[0030] Step 105: Return executable trip data to the target user.

[0031] In this embodiment, obtaining the text and image content specified by the target user from the network platform can provide personalized reference information for itinerary planning. Inputting the text and image content into a text and image parsing model allows for joint semantic understanding of the text and image information, extracting structured itinerary data containing location information. This improves information acquisition efficiency and enhances the accuracy of location information by performing correlation analysis on unstructured text and image content, preventing information omissions. Inputting the structured itinerary data into a location verification model allows for standardization of location information by calling a map service interface, obtaining standard location data that ensures accuracy and usability. Inputting the standard location data into a path planning model allows the model to call a navigation service interface to calculate paths between various locations, generating executable itinerary data containing waypoint sequences and navigation instructions. Returning the executable itinerary data to the target user avoids the user repeatedly switching between navigation software and itinerary plans, improving the user experience.

[0032] The following is combined with Figure 2 The method for travel planning based on a multimodal large model provided in the embodiments of the present invention is further explained. Figure 2 This is another flowchart illustrating the route planning method based on a multimodal large model provided in this embodiment of the invention. (See also...) Figure 2 The trip planning method based on a multimodal large model in this embodiment may include the following steps: Step 201: Obtain the text and image content specified by the target user from the online platform.

[0033] Step 202: Based on the source platform identifier and content identifier of the text and image content, query the preset itinerary database to see if there is executable itinerary data corresponding to the text and image content. If it exists, proceed to step 203; otherwise, proceed to step 204.

[0034] Step 203: Return to the preset itinerary database and retrieve the executable itinerary data corresponding to the text and image content.

[0035] The source platform identifier is the identifier of the online platform where the text and image content resides. The content identifier is an identifier used to uniquely identify a piece of text and image content within a specific source platform, and it typically does not change despite changes to the content. Specifically, a specific piece of text and image content can be identified through the source platform identifier and the content identifier. For example, if a target user shares text and image content with the link "https: / / www.AAAA.com / explore / 123456", then "AAAA" can be used as the source platform identifier, and "123456" can be used as the content identifier.

[0036] Specifically, since a popular image or text content on a network platform may be specified by multiple target users successively, the initially generated executable trip data can be stored in the trip database along with the source platform identifier and content identifier of the image or text content. When the next user shares the same image or text content, a query is performed in the trip database based on the specified source platform identifier and content identifier. This directly returns the previously generated and stored executable trip data, reducing user waiting time and avoiding the waste of computing resources caused by repeatedly generating the same image or text content.

[0037] Step 204: Input the text and image content into the text and image parsing model to perform joint semantic understanding of the text and image information in the text and image content, and extract structured travel data containing location information.

[0038] Step 205: Standardize the location information to obtain the standardized location name.

[0039] A standardized location name is a clear and unambiguous name obtained after standardizing location information. Specifically, since location names in text and image information often have colloquialisms, abbreviations, or homonyms, the location description and location name can be input into a large language model simultaneously. Combined with preset prompts, the large language model is instructed to perform enhanced retrieval by searching a standard place name database, resulting in a clear and standardized location name, thus eliminating ambiguity.

[0040] Step 206: Call the map service interface based on the location specification name to query the point of interest database to see if there is a point of interest corresponding to the location specification name.

[0041] A Points of Interest (POI) database is a database built and maintained by map service providers that stores a large amount of geographic entity information. Points of Interest are geographic entities within this database. Specifically, they can be various places a user might visit or stay at during their travels, including attractions, restaurants, hotels, transportation hubs, shopping malls, etc.

[0042] Specifically, by invoking the map service interface to query the Point of Interest database based on the location's specified name, it can verify whether the location's specified name actually exists in the real geographic space. If the query finds a match, it means that the location information is valid, and the location data corresponding to the specified name can be obtained for subsequent processing. If the query does not find a match, it may indicate that the location information is incorrect or has expired. In this case, exception handling can be performed according to preset strategies, such as deleting invalid location information from the structured trip data or prompting the user to correct the error.

[0043] Step 207: If it exists, obtain the location data of the point of interest, and perform coordinate system transformation on the location data to obtain standard location data.

[0044] Specifically, when acquiring location data, the location verification model may call map service interfaces provided by different map service providers, resulting in the latitude and longitude in the acquired location data using different coordinate systems, such as GCJ-02 and WGS-84. Therefore, standardizing location data can also include converting the latitude and longitude in the location data of each location to the same coordinate system to obtain standard location data.

[0045] In this embodiment, the location information is standardized to obtain standardized location names, which improves the accuracy of the standard location data retrieved subsequently. The map service interface is called based on the standardized location names to query the point of interest database to see if there are any points of interest corresponding to the standardized location names, ensuring the availability and timeliness of the obtained standard location data. The coordinate system transformation of the location data solves the coordinate offset problem between different data sources, resulting in accurate location data and providing a reliable foundation for the subsequent calculation of paths between various locations.

[0046] Step 208: Obtain the itinerary preference information input by the target user, and make intelligent recommendations based on the itinerary preference information and structured itinerary data to obtain the recommended stay duration and recommended mode of transportation for each location.

[0047] Trip preference information refers to personalized needs parameters proactively provided by target users during the trip planning process. Specifically, trip preference information typically includes users' preferred choices regarding travel style, time allocation, and transportation methods. This information can be collected through user interfaces via questionnaires, checkboxes, or voice input. For example, a user could voice input, "Our family of five—one elderly person and two children—is planning a road trip and hopes the itinerary won't be too rushed."

[0048] Recommended transportation options include travel methods between two adjacent locations. These options can include driving, walking, cycling, and public transportation. Specifically, the distance between the two locations, the recommended transportation option field from the structured trip data, and user preference information can be input into a large language model. This model is then combined with pre-defined prompts to generate intelligent recommendations for transportation options.

[0049] The recommended stay duration is the suggested time to visit each location. The route planning model can adjust the recommended stay duration based on the data provided in the structured itinerary, either upwards or downwards, taking into account user preferences. For example, if a user prefers leisure travel, the recommended stay duration can be adjusted upwards; if a user prefers a tight schedule, the recommended stay duration can be adjusted downwards.

[0050] Step 209: Based on the recommended mode of transportation and standard location data, call the navigation service interface to obtain the routes between various locations, and generate a travel timeline based on the routes and the recommended stay duration at each location.

[0051] A travel timeline is a timetable composed of arrival and departure times for each location, arranged chronologically along the routes between them. Specifically, the locations are first arranged chronologically according to the time of visit. Then, based on standard location data for recommended transportation methods, starting and ending points, a navigation service interface is called to obtain the travel duration corresponding to the recommended transportation method. Finally, the travel duration is fused with the recommended stay duration at each location to generate the travel timeline.

[0052] Step 210: Assemble the travel timeline and path to obtain executable travel data containing waypoint sequences and navigation instructions.

[0053] Specifically, the paths between various locations can be arranged in chronological order of the visit, with each location accompanied by its corresponding arrival and departure times from the itinerary timeline, resulting in a waypoint sequence containing both time and spatial information. In the waypoint sequence, each path segment is bound to a corresponding navigation command, and clicking the navigation command will trigger navigation from the current waypoint to the next waypoint.

[0054] Step 211: Return executable trip data to the target user.

[0055] Step 212: Perform a hash operation on the text and image content to obtain the first hash value.

[0056] The first hash value is a fixed-length string obtained by performing a hash algorithm on the text and image content upon which the executable itinerary data depends. For example, text information can be preprocessed, such as removing whitespace and punctuation, and then input into a preset hash algorithm, such as MD5 or SHA-256, to encode it into a fixed-length string. For image information, a perceptual hash algorithm can be used. First, the image is scaled to a fixed size, then image features are extracted and encoded into a fixed-length string. The first hash value is obtained by concatenating the corresponding strings of the text and image information. This avoids triggering the regeneration of itinerary data and reduces computational resource consumption when the original author makes non-substantial modifications to the text and image content, such as changing punctuation, compressing images, or converting formats.

[0057] Step 213: Store the first hash value and executable trip data in the trip database, and establish a mapping relationship between the source platform identifier and content identifier of the text and image content and the executable trip data and the first hash value.

[0058] Specifically, a trip data record in the trip database can include the source platform identifier of the text and image content, the content identifier of the text and image content, the first hash value of the text and image content, and the executable trip data.

[0059] Step 214: Based on the content identifier, obtain the updated text and image content from the source platform of the text and image content.

[0060] In one feasible implementation, content identifiers can be subscribed to on the source platform, enabling the source platform to send a content update message when the corresponding text and image content is updated. In response to the content update message, the updated text and image content can be retrieved from the source platform. This allows for the synchronization of itinerary data with minimal delay when the original text and image content stored in the itinerary database is modified or supplemented by the original author, without requiring frequent access to the source platform, thus saving network communication resources.

[0061] In another feasible implementation, the updated text and image content corresponding to the content identifier can be obtained from the source platform according to the update cycle of the executable trip data; the update cycle is determined based on the access popularity of the executable trip data.

[0062] Access popularity of executable trip data refers to an indicator that quantifies how frequently users access the data. Specifically, this can include the total number of times the trip data is accessed, the number of unique users accessing the data, and the number of times the data is saved. In addition to regenerating trip data based on changes in the text and image content corresponding to the content identifiers in the source platform, it's also necessary to consider updating the trip data synchronously with changes in Points of Interest (POIs) information in the POI database. For example, if a restaurant in a trip data entry closes down, its corresponding POI information becomes invalid, requiring the deletion of the restaurant's location information and the regeneration of the trip data to avoid inaccurate data causing user dissatisfaction. Highly accessed executable trip data has higher timeliness requirements and can use a shorter update cycle, such as one day, while less accessed data can use a longer update cycle, such as one week. By adaptively adjusting the update cycle according to access popularity, computing resources can be allocated rationally while ensuring the timeliness of the executable trip data.

[0063] Optionally, a combination of subscribed content identifiers and proactive acquisition based on the update cycle of executable trip data can be used. This can avoid missing updates of text and image content. A global update cycle can also be set to retrieve the text and image update content corresponding to all content identifiers in the trip database from the source platform according to the global update cycle. This serves as a backup guarantee mechanism to ensure that trip data that has not been subscribed to and has low access popularity can also be updated.

[0064] Step 215: Perform a hash operation on the updated text and image content to obtain the second hash value.

[0065] The second hash value is a fixed-length string obtained by performing a hash algorithm on the text and image update content corresponding to the content identifier obtained from the source platform.

[0066] Step 216: Compare whether the first hash value and the second hash value are consistent.

[0067] Step 217: If there is a discrepancy, regenerate the executable itinerary data based on the updated content of the image and text.

[0068] Specifically, when the original text and image content stored in the trip database is modified or supplemented by the original author, the system can quickly determine whether the text and image content corresponding to the content identifier has been updated by comparing whether the first hash value and the second hash value are consistent. The system can then regenerate the trip data corresponding to the updated text and image data and store the regenerated text and image data in the trip database along with the corresponding content identifier. The trip database can also store multiple historical versions of trip data for target users to choose from.

[0069] Figure 3 This is an architecture diagram of the itinerary planning system provided in an embodiment of the present invention. (See attached diagram.) Figure 3 The system first obtains the text and image content shared by the user on the online platform. Then, it performs data matching and verification in the trip database based on the source platform identifier and the text and image content identifier. If the corresponding executable trip data already exists in the trip database, it directly returns the retrieved executable trip data to the user. If it does not exist, it enters the trip planning module, which uses a text and image parsing model, a location verification model, and a route planning model to collaboratively process and generate executable trip data, which is then returned to the user. At the same time, the executable trip data is stored in the trip database. The trip database also supports periodic data synchronization with the online platform to maintain the real-time performance and validity of the executable trip data.

[0070] In this embodiment, by querying a preset itinerary database, itinerary data corresponding to the stored text and image content is returned. This allows for the reuse of already generated itinerary planning data, improving processing efficiency and avoiding the waste of computing resources caused by the repeated generation of itinerary data when the same popular text and image content is specified by multiple users. It also avoids data inconsistencies that may arise from different users independently generating itinerary data. Standardizing location information to obtain standardized location names improves the accuracy of subsequent standard location data queries. Calling the map service interface based on the standardized location name to query the Point of Interest (POI) database to check if there are any POIs corresponding to the standardized location name ensures the availability and timeliness of the obtained standard location data. Coordinate system transformation of location data solves the coordinate offset problem between different data sources, obtaining accurate location data and providing a reliable foundation for subsequent path calculations between various locations. By comparing the hash values ​​of the text and image content and the updated text and image content from the source platform, the executable itinerary data is automatically updated when the two are inconsistent, ensuring the real-time performance and accuracy of the itinerary data.

[0071] Figure 4 This is a schematic diagram of a travel planning device based on a multimodal large model provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes: The acquisition module 401 is used to acquire the text and image content specified by the target user from the network platform. The extraction module 402 is used to input the text and image content into the text and image parsing model, so as to use the text and image parsing model to perform joint semantic understanding on the text information and image information in the text and image content, and extract structured travel data containing location information; The standardization module 403 is used to input structured trip data into the location verification model, so that the location verification model can call the map service interface to standardize the location information and obtain standard location data. The generation module 404 is used to input standard location data into the route planning model, so that the route planning model can call the navigation service interface to calculate the path between various locations, generate executable trip data containing waypoint sequences and navigation instructions, and return the executable trip data to the target user.

[0072] In one embodiment, the device further includes: a query module, used to query a preset itinerary database to see if there is executable itinerary data corresponding to the text and image content, based on the source platform identifier and content identifier of the text and image content; If it does not exist, then proceed with the step of inputting the text and image content into the text and image parsing model.

[0073] In one embodiment, the device further includes a storage module for storing executable trip data in a trip database and establishing a mapping relationship between the source platform identifier and content identifier of the text and image content and the executable trip data.

[0074] In one embodiment, the device further includes: an update module, comprising: The hash operation unit is used to perform hash operation on the image and text content to obtain the first hash value, store the first hash value in the trip database, and establish a mapping relationship between the source platform identifier and content identifier of the image and text content and the first hash value; The update acquisition unit is used to acquire updated text and image content from the source platform of the text and image content based on the content identifier; The update determination unit is used to determine whether to regenerate the executable process data corresponding to the content identifier based on the first hash value and the updated text and image content.

[0075] In one embodiment, the update acquisition unit is used for: Subscribe to content identifiers on the source platform so that the source platform sends a content update message when the corresponding text and image content is updated; in response to the content update message, retrieve the updated text and image content from the source platform; or: The updated text and image content corresponding to the content identifier is obtained from the source platform according to the update cycle of the executable itinerary data; the update cycle is determined based on the access popularity of the executable itinerary data.

[0076] In one embodiment, the update determination unit is used to: Perform a hash operation on the updated text and image content to obtain a second hash value; Compare whether the first hash value and the second hash value are the same; If there is a discrepancy, the executable itinerary data will be regenerated based on the updated text and image content.

[0077] In one embodiment, the standardization module 403 is specifically used for: The location information is standardized to obtain standardized location names; The map service interface is called based on the location specification name to query the point of interest database to see if there is a point of interest corresponding to the location specification name. If it exists, the location data of the point of interest is obtained, and the location data is processed by coordinate system transformation to obtain standard location data.

[0078] In one embodiment, the device further includes a preference acquisition module, used to acquire travel preference information input by the target user, and to make intelligent recommendations based on the travel preference information and structured travel data to obtain recommended stay duration and recommended mode of transportation for each location; the generation module 404 is specifically used for: The navigation service interface is called to calculate the route between various locations based on the recommended mode of transportation and standard location data, and a timeline of the trip is generated based on the route and the recommended stay time at each location. The timeline and path are assembled to obtain executable trip data containing waypoint sequences and navigation instructions.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0080] The apparatus of this invention acquires text and image content specified by a target user from a network platform, providing personalized reference information for trip planning. It inputs the text and image content into a text and image parsing model to perform joint semantic understanding of the text and image information, extracting structured trip data containing location information, thus improving information acquisition efficiency. Furthermore, it improves the accuracy of location information and avoids information omissions by performing correlation analysis on unstructured text and image content. The structured trip data is then input into a location verification model, which calls a map service interface to standardize the location information, obtaining standard location data that ensures the accuracy and usability of the location information. The standard location data is then input into a path planning model, which calls a navigation service interface to calculate the path between various locations, generating executable trip data containing waypoint sequences and navigation instructions. Finally, the executable trip data is returned to the target user, avoiding repeated switching between navigation software and trip plans, thus improving the user experience.

[0081] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing an electronic device according to embodiments of the present invention. Figure 5The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0082] like Figure 5 As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 502 or programs loaded from storage section 508 into Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the computer system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0083] The following components are connected to I / O interface 505: input section 506 including keyboard, mouse, etc.; output section 507 including cathode ray tube, liquid crystal display, etc., and speakers, etc.; storage section 508 including hard disk, etc.; and communication section 509 including network interface card, such as modem, etc. Communication section 509 performs communication processing via a network such as the Internet. Drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0084] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0085] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, etc., or any suitable combination thereof.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0087] The modules and / or units described in the embodiments of this invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor can be described as including an acquisition module, an extraction module, a standardization module, and a generation module. The names of these modules do not necessarily limit the functionality of the module itself.

[0088] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: The system retrieves text and image content specified by the target user from the online platform; inputs this content into a text and image parsing model to perform joint semantic understanding of the text and image information, extracting structured trip data containing location information; inputs this structured trip data into a location verification model to standardize the location information by calling a map service interface, obtaining standard location data; inputs this standard location data into a route planning model to calculate the path between various locations by calling a navigation service interface, generating executable trip data containing waypoint sequences and navigation instructions; and finally returns the executable trip data to the target user.

[0089] The technical solution of this invention obtains text and image content specified by the target user from a network platform, providing personalized reference information for trip planning; inputting the text and image content into a text and image parsing model allows for joint semantic understanding of the text and image information, extracting structured trip data containing location information, thus improving information acquisition efficiency; and by performing correlation analysis on unstructured text and image content, the accuracy of location information is improved, avoiding information omissions; inputting the structured trip data into a location verification model allows for standardization of location information by calling a map service interface, obtaining standard location data, ensuring the accuracy and usability of location information; inputting the standard location data into a path planning model allows the path planning model to call a navigation service interface to calculate the path between various locations, generating executable trip data containing waypoint sequences and navigation instructions; returning executable trip data to the target user avoids the user repeatedly switching between navigation software and trip plans, improving the user experience.

[0090] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal large model-based route planning method provided in any embodiment of this invention.

[0091] In the implementation of a computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0093] It should be noted that the collection, use, storage, sharing, and transfer of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations, and require notification to the user and obtaining the user's consent or authorization. Where applicable, user personal information has undergone de-identification and / or anonymization and / or encryption technical processing. In addition, a corresponding operation entry is provided for the user to choose to agree to or reject the automated decision result; if the user chooses to reject, the process proceeds to the expert decision-making process.

[0094] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A route planning method based on a multimodal large model, characterized in that, The method includes: Obtain text and image content specified by the target user from the online platform; The text and image content is input into the text and image parsing model to perform joint semantic understanding of the text and image information in the text and image content, and extract structured travel data containing location information. The structured trip data is input into the location verification model, and the location verification model is used to call the map service interface to standardize the location information to obtain standard location data. The standard location data is input into the route planning model so that the route planning model calls the navigation service interface to calculate the path between the locations and generate executable trip data containing a sequence of waypoints and navigation instructions. Return the executable trip data to the target user.

2. The method according to claim 1, characterized in that, Before inputting the text and image content into the text and image parsing model, the following steps are also included: Based on the source platform identifier and content identifier of the text and image content, query the preset itinerary database to see if there is executable itinerary data corresponding to the text and image content; If it does not exist, then perform the step of inputting the text and image content into the text and image parsing model.

3. The method according to claim 2, characterized in that, The method further includes: The executable trip data is stored in the trip database, and a mapping relationship is established between the source platform identifier and content identifier of the text and image content and the executable trip data.

4. The method according to claim 3, characterized in that, The method further includes: The image and text content is hashed to obtain a first hash value, which is then stored in the trip database. A mapping relationship is established between the source platform identifier and content identifier of the image and text content and the first hash value. Based on the content identifier, obtain the updated text and image content from the source platform of the text and image content; Based on the first hash value and the updated text and image content, determine whether to regenerate the executable itinerary data corresponding to the content identifier.

5. The method according to claim 4, characterized in that, Based on the content identifier, obtain updated image and text content from the source platform of the image and text content, including: Subscribe to the content identifier on the source platform so that the source platform sends a content update message when the text and image content corresponding to the content identifier is updated; in response to the content update message, obtain the updated text and image content from the source platform; or: According to the update cycle of the executable itinerary data, the updated text and image content corresponding to the content identifier is obtained from the source platform; the update cycle is determined based on the access popularity of the executable itinerary data.

6. The method according to claim 3, characterized in that, Determining whether to regenerate the executable process data corresponding to the content identifier based on the first hash value and the updated image and text content includes: Perform a hash operation on the updated text and image content to obtain a second hash value; Compare whether the first hash value and the second hash value are consistent; If there is a discrepancy, the executable itinerary data will be regenerated based on the updated text and image content.

7. The method according to claim 1, characterized in that, The structured trip data is input into a location verification model, which then uses the location verification model to call a map service interface to standardize the location information, obtaining standard location data, including: The location information is standardized to obtain standardized location names; The map service interface is invoked based on the specified location name to query the point of interest database to see if there is a point of interest corresponding to the specified location name. If it exists, the location data of the point of interest is obtained, and the location data is subjected to coordinate system transformation to obtain the standard location data.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the travel preference information input by the target user, and make intelligent recommendations based on the travel preference information and the structured travel data to obtain the recommended stay duration and recommended mode of transportation for each location; The step of inputting the standard location data into the route planning model, so that the route planning model calls the navigation service interface to calculate the path between various locations and generates executable trip data containing a sequence of waypoints and navigation instructions, includes: The navigation service interface is invoked based on the recommended mode of transportation and standard location data to calculate the path between various locations, and a travel timeline is generated based on the path and the recommended stay duration at each location. The travel timeline and the path are assembled to obtain executable travel data containing a sequence of waypoints and navigation instructions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal large model-based travel planning method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multimodal large model-based travel planning method as described in any one of claims 1 to 8.

Images