Route generation method and system based on multi-modal large model and storage medium

Abstract

The invention relates to the technical field of intelligent route generation, in particular to a multi-modal large model-based route generation method and system and a storage medium, and the multi-modal large model-based route generation method provided by the invention comprises the following steps: extracting target information corresponding to an obtained webpage link; a target webpage corresponding to the webpage link comprises tourism information of at least two geographic points; determining description information and geographic information corresponding to all target geographic points in the target information based on a multi-modal large model; performing structured processing on the description information and the geographic information of all the target geographic points to obtain structured information; and automatically generating an editable route according to the structured information based on the multi-modal large model, and generating a route text representation and / or a route audio representation of the editable route. The webpage content corresponding to the webpage link is extracted and processed through the multi-modal large model, so that the tourism route of the place where the user wants to go is made by directly utilizing the tourism strategy on the platform.

Description

[Technical field]

[0001] The present invention relates to the technical field of intelligent route generation, and in particular to a route generation method, system and storage medium based on a multimodal large model. [Background technology]

[0002] As people's living standards improve, more and more people are traveling to various city attractions. However, whether it is a short-distance trip or a long-distance trip, it is inevitable to make a travel route.

[0003] Currently, you can view travel guides recommended by the platform or shared by other users on multiple content platforms. However, when users create travel routes, they can only add locations and cannot directly use the travel guides on these platforms to create travel routes to the places they want to go, which makes travel route creation complicated. [Summary of the invention]

[0004] In order to solve the problem that it is impossible to directly use the travel guide on the platform to create a travel route to the place where the user wants to go, which leads to complicated travel route creation, the present invention provides a route generation method, system and storage medium based on a multimodal large model.

[0005] The solution to the technical problem of the present invention is to provide a route generation method based on a multimodal large model, comprising the following steps:

[0006] Obtaining a webpage link and extracting target information corresponding to the webpage link; the target webpage corresponding to the webpage link includes tourism information of at least two geographical points;

[0007] Determine the description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model;

[0008] Structuring the description information and geographic information of all target geographic points to obtain structured information;

[0009] An editable route is automatically generated according to the structured information based on the multimodal big model, and a route text representation and / or a route audio representation of the editable route is generated.

[0010] Preferably, obtaining a webpage link and extracting target information corresponding to the webpage link specifically includes:

[0011] Obtain a webpage link and determine a target webpage corresponding to the webpage link; the target webpage is composed of at least one of text, picture, and video;

[0012] All geographic points and corresponding associated information existing in the target webpage are extracted, and all geographic points and corresponding associated information are used as target information.

[0013] Preferably, extracting all geographical points and corresponding associated information existing in the target webpage and using all geographical points and corresponding associated information as target information specifically includes:

[0014] Determine whether the target webpage includes at least two of text, image, and video;

[0015] If yes, then extract all geographic points and corresponding associated information from all texts, pictures and videos in the target webpage to obtain text information, picture information and video information;

[0016] Text information, picture information and video information are taken as target information.

[0017] Preferably, the description information and geographic information corresponding to all target geographic points in the target information are determined based on the multimodal large model, specifically including:

[0018] Based on the multimodal large model, the text information, image information and video information in the target information are integrated and multimodally represented to obtain multimodal information;

[0019] The descriptive information and geographic information corresponding to all target geographic points corresponding to the multimodal information are determined through the attention mechanism of the multimodal large model.

[0020] Preferably, the description information includes scenic spot information, activity information, time information and traffic information.

[0021] Preferably, the description information and geographic information of all target geographic points are structured to obtain structured information, which specifically includes:

[0022] Construct a knowledge graph with the target geographic point as the core based on multimodal information;

[0023] The knowledge graph is used to connect the geographic information of all target geographic points and the scenic spot information, activity information, time information and traffic information in the description information to obtain structured information.

[0024] Preferably, after automatically generating an editable route according to the structured information based on the multimodal macro model and generating a route text representation of the editable route, the method further comprises:

[0025] In response to the user's route adjustment operation, add, delete or modify the order of geographical points in the editable route;

[0026] A new editable route and a corresponding textual representation of the route and / or an audio representation of the route are regenerated based on all the adjusted geographic points.

[0027] Preferably, before determining the description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model, the method further includes:

[0028] Acquire route information annotation data of multiple modalities, train the multimodal large model, and obtain the trained multimodal large model.

[0029] In order to solve the above technical problems, the present invention further provides a route generation system based on a multimodal large model, which is used to implement the route generation method based on a multimodal large model as described in any one of the above items, comprising:

[0030] An information extraction module is used to obtain web page links and extract target information corresponding to the web page links;

[0031] An information determination module is used to determine the description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model;

[0032] The structuring module is used to perform structural processing on the description information and geographic information of all target geographic points to obtain structured information;

[0033] The route generation module is used to automatically generate an editable route according to structured information based on the multimodal large model, and generate a route text representation and / or a route audio representation of the editable route.

[0034] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a route generation method based on a multimodal large model as described in any one of the above items is implemented.

[0035] Compared with the prior art, the route generation method, system and storage medium based on the multimodal large model provided by the present invention have the following advantages:

[0036] 1. The embodiment of the present invention provides a route generation method based on a multimodal big model, comprising the following steps: obtaining a web page link, extracting target information corresponding to the web page link; the target web page corresponding to the web page link includes tourism information of at least two geographical points; determining the description information and geographical information corresponding to all target geographical points in the target information based on the multimodal big model; performing structured processing on the description information and geographical information of all target geographical points to obtain structured information; automatically generating an editable route based on the structured information based on the multimodal big model, and generating a route text representation and / or route audio representation of the editable route. The multimodal big model can be used to extract key information related to geographical points in the tourism information in the target web page corresponding to the input web page link, and use the key information as the target information, and perform structured processing on the description information and geographical information corresponding to all core target geographical points in the target information, so that the multimodal model automatically generates an editable route composed of multiple target geographical points in series according to the structured information, and introduces all places on the editable route. This enables the extraction, screening, summarization and structured processing of the web page content corresponding to the web link, so that an editable route with a starting point, an end point and the sequence of the route is automatically generated through a multimodal large model, and relevant information is described for the geographical points on the editable route, so that the automatically generated editable route can be more intuitively presented to the user, and then it is possible to directly use the travel guides on the platform to create travel routes to the places the user wants to go, without the user having to manually add places one by one to form a route, making the creation of the entire travel route simpler.

[0037] 2. The method of obtaining a web link and extracting target information corresponding to the web link in the embodiment of the present invention specifically includes: obtaining a web link and determining the target web page corresponding to the web link; the target web page is composed of at least one of text, picture, and video; extracting all geographic points and corresponding associated information existing in the target web page, and using all geographic points and corresponding associated information as target information. The extraction of geographic points and information related to geographic points in the target web page corresponding to the web link is realized, that is, the extraction of the core content of the travel information in the target network line is realized, avoiding the mixing of irrelevant content and affecting the generation of the final editable route.

[0038] 3. The method of extracting all geographic points and corresponding associated information in the target web page and using all geographic points and corresponding associated information as target information in the embodiment of the present invention specifically includes: determining whether the target web page includes at least two of text, pictures, and videos; if so, extracting all geographic points and corresponding associated information in all text, pictures, and videos in the target web page respectively to obtain text information, picture information, and video information; using text information, picture information, and video information as target information. When it is determined that the web page is composed of at least two of text, pictures, and videos, the text, pictures, and videos appearing in the web page need to be extracted separately using different extraction technologies to ensure the accuracy of the information extracted from each part of the content, thereby ensuring the accuracy of the target information obtained.

[0039] 4. In the embodiment of the present invention, the text information, picture information and video information in the target information are integrated based on the multimodal large model and multimodally represented, so as to associate similar semantics related to geographic points between the text information, picture information and video information to reduce the heterogeneity gap, and through the additional semantics provided by the three, the semantic information related to geographic points between the three is complemented, so as to obtain more comprehensive, richer and more accurate multimodal information of multiple geographic points after fusion and complementation, which is conducive to improving the performance and accuracy of the multimodal large model; and the description information and geographic information corresponding to all target geographic points corresponding to the multimodal information are determined through the attention mechanism of the multimodal large model, that is, the information of different modes is weighted through the attention mechanism, so that the multimodal large model can automatically focus on the most relevant part, that is, automatically focus on and understand the geographic points and the corresponding description information and geographic information in the multimodal information, and use the highly relevant geographic points and the corresponding description information and geographic information as the description information and geographic information corresponding to the target geographic points. This enables the integration of multimodal content about route information, thereby obtaining more accurate, comprehensive and core geographical points and related information, and the final automatically generated editable route map is more complete and accurate.

[0040] 5. The descriptive information and geographic information of all target geographic points in the embodiment of the present invention are structured to obtain structured information, specifically including: constructing a knowledge graph with the target geographic point as the core based on multimodal information; connecting the geographic information of all target geographic points and the scenic spot information, activity information, time information and traffic information in the descriptive information through the knowledge graph to obtain structured information. By introducing entities, attributes and relationships in the knowledge graph, the entities and relationships corresponding to the scenic spot information, activity information, time information and traffic information in the geographic information and descriptive information of all target geographic points are connected, so that the multimodal big model can obtain a deeper semantic understanding of the geographic information and descriptive information of the target geographic points, so that the model multimodal big model can better understand the context and improve the accuracy of the route text representation and / or route audio representation of the editable route finally generated.

[0041] 6. In the embodiment of the present invention, in response to the user's route adjustment operation, the geographical points in the editable route are added, deleted or sequentially modified; and a new editable route is regenerated based on all the adjusted geographical points. By responding to the user's adjustment operation, the geographical points on the editable route can be directly added, deleted or sequentially modified, thereby meeting the different personalized needs of users and ensuring the rationality and practicality of the generated editable route.

[0042] 7. In the embodiment of the present invention, the route information annotation data of multiple modes is obtained, and the multimodal large model is trained to obtain the trained multimodal large model. The performance of the multimodal large model on multimodal information is optimized, so that the optimized multimodal large model can better integrate information of different modes, thereby improving the recognition and understanding capabilities of complex scenes and content, and further improving the accuracy of extracting and understanding multimodal information. In addition, it can also help eliminate the information island phenomenon between different modes, so that different modal information can better influence and complement each other, promote the collaborative work of the multimodal large model between multiple modes, and improve the overall performance.

[0043] 8. The present invention also provides a route generation system based on a multimodal large model, which has the same beneficial effects as the above-mentioned route generation method based on a multimodal large model, and will not be elaborated here.

[0044] 9. The present invention also provides a computer-readable storage medium, which has the same beneficial effects as the above-mentioned route generation method based on the multimodal large model, and will not be described in detail here.

Brief Description of the Drawings

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 It is a flowchart of the steps of the route generation method based on the multimodal large model provided in the first embodiment of the present invention.

[0047] Figure 2 It is a flowchart of the specific steps of step S20 of the route generation method based on the multimodal large model provided in the first embodiment of the present invention.

[0048] Figure 3 It is a flowchart of specific steps of step S30 of the route generation method based on the multimodal large model provided in the first embodiment of the present invention.

[0049] Figure 4 It is a block diagram of a route generation system based on a multimodal large model provided by the second embodiment of the present invention. [Specific implementation method]

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] See also Figure 1 The first embodiment of the present invention provides a route generation method based on a multimodal large model, comprising the following steps:

[0052] S10: Obtain a webpage link and extract target information corresponding to the webpage link; the target webpage corresponding to the webpage link includes tourism information of at least two geographical points;

[0053] Specifically, the webpage link can be a link in any format, and this embodiment does not limit this, as long as the target webpage corresponding to the webpage link includes tourism information of at least two geographical points. The webpage link is specifically an input webpage link corresponding to the target webpage selected by the user, and the target webpage can be a webpage on any platform or application that includes tourism information of at least two geographical points. The target webpage includes tourism information of at least two geographical points, and the tourism information includes a route that is sequentially connected by at least two geographical points or information that only has two geographical points but does not form a route. The target information is extracted from the tourism information of the target webpage, specifically all the places involved in the tourism information and information associated with the places are extracted.

[0054] S20: Determine description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model;

[0055] Specifically, through step S20, the target information is integrated and expanded to screen out all core geographical points from all locations of the target information, use the core geographical points as target geographical points, and summarize the associated information corresponding to each target geographical point, and use the associated information as descriptive information.

[0056] S30: Structuring the description information and geographic information of all target geographic points to obtain structured information;

[0057] Specifically, through structural processing, the description information and geographic information of all target geographic points can be summarized, and a structure including all target geographic points and the corresponding description information and geographic information can be established to obtain structured information. The structure represents all target geographic points and the relationship between the description information and geographic information. At the same time, through a clear structure, the multimodal large model can better understand the extracted target information.

[0058] S 40: Automatically generate an editable route according to the structured information based on the multimodal large model, and generate a route text representation and / or a route audio representation of the editable route.

[0059] Specifically, the multimodal big model can be a chatgpt and Wenxinyiyan big model, or it can be other AI big models, and this implementation does not limit this. The editable route is a complete route map that can be directly edited and adjusted. The route map is composed of multiple target geographical points connected in series to form a route, and has a starting point and an end point. The multimodal big model can automatically plan the sorting of all target geographical points in the structured information according to the structured information, so as to determine the location points of the target geographical points in the retrieved map after retrieving the map, and connect the corresponding location points of the target geographical points in the map in series in the retrieved map according to the sorting to form an editable route. At the same time, the route text representation and / or route audio representation in the editable route map can also be automatically generated. The route text representation is a descriptive text of all target geographical points corresponding to the editable route map, and the representation order of the route text representation is consistent with the sorting of all target geographical points in the editable route map. The route audio representation is the audio corresponding to the route text representation, and the representation order of the target geographical points in the audio is also consistent with the sorting of all target geographical points in the editable route map. The route text representation includes but is not limited to recommended attractions and activities, the best mode of transportation between target geographic points in the editable route, recommended travel time and travel sequence for each target geographic point, precautions and tips for each target geographic point, etc.

[0060] It should be noted that the editable route is specifically displayed and directly edited on the retrieved map. In this embodiment, the user's current real-time location can also be obtained, so that navigation from the user's current location to any target geographical point on the editable route can be completed, thereby realizing the continuous operation of route generation and navigation, making it more convenient and practical for users to use. In addition, the setting of direct navigation after route generation is more conducive to the user's temporary travel or tourism needs.

[0061] It can be understood that the multimodal large model can extract the key information related to the geographical points in the tourism information in the target web page corresponding to the input web link, and use the key information as the target information, and perform structured processing on the description information and geographical information corresponding to all the core target geographical points in the target information, so that the multimodal model automatically generates an editable route composed of multiple target geographical points in series according to the structured information, and introduces all the places on the editable route. In this way, the extraction, screening, summarization and structured processing of the web page content corresponding to the web link are realized, so that the multimodal large model automatically generates an editable route with a starting point, an end point and a sequence of route itineraries, and describes the relevant information of the geographical points on the editable route, so that the automatically generated editable route can be more intuitively displayed to the user, and then the travel guide on the platform is directly used to make a travel route to the place where the user wants to go, without the need for the user to manually add places in sequence to form a route, making the production of the entire travel route simpler.

[0062] It should be noted that, since the multimodal large model can automatically plan the order of all target geographic points in the structured information according to the structured information, the order of all target geographic points in the editable route map finally generated may be inconsistent with the order of multiple geographic points in the travel information in the target web page corresponding to the original web page link. Therefore, the editable route generated based on the multimodal large model in this embodiment is not simply to extract the information of the web page for re-display, but to further screen, summarize, integrate and expand the extracted information, so as to generate an editable route map suitable for the user of the current operation, making the editable route map more reasonable and more applicable.

[0063] Furthermore, step S10 specifically includes:

[0064] S101: Obtain a webpage link and determine a target webpage corresponding to the webpage link; the target webpage is composed of at least one of text, picture, and video;

[0065] Specifically, the target web page may be composed entirely of any one of text, pictures, and videos, or a combination of any two of the text, pictures, and videos, or a combination of the three.

[0066] S102: extracting all geographic points and corresponding associated information existing in the target webpage, and taking all geographic points and corresponding associated information as target information.

[0067] It can be understood that, through step S102, no matter what form of web page, all content information on the web page is first extracted, and then key information in the web page is further extracted from all the content in advance, including title, main paragraph, list, itinerary, picture description, user comments, etc., and then data cleaning is performed on the important information, unnecessary tags, formats or advertising information are removed, and the geographic point and corresponding related information are retained, so as to obtain the target information. In this way, the extraction of the geographic point and related information of the target web page corresponding to the web link is realized, that is, the extraction of the core content of the travel information in the target network line is realized, and irrelevant content is avoided from being mixed in and affecting the generation of the final editable route.

[0068] Furthermore, step S102 specifically includes:

[0069] S1021: Determine whether the target webpage includes at least two of text, pictures, and videos;

[0070] S1022: If yes, extract all geographic points and corresponding associated information in all texts, pictures and videos in the target webpage respectively to obtain text information, picture information and video information;

[0071] S1023: Take text information, picture information and video information as target information.

[0072] Specifically, when the target webpage is composed of any two combinations of text, pictures, and videos, or is composed of text, pictures, and videos together, due to the characteristics and different forms of expression of the three, a single extraction technology cannot adapt to the three types of text, pictures, and videos, and it is also impossible to extract the required information from the text, pictures, and videos. Therefore, when the target webpage includes at least two of the three, the three different forms of expression of text, pictures, and videos require their own corresponding extraction technologies for extraction.

[0073] More specifically, for the text portion of the target web page, the text in the target web page is cleaned and segmented, so that the geographic points and corresponding related information are extracted from the text portion as text information, such as destination, transportation method, time arrangement and other information.

[0074] For the image part of the target web page, computer vision technology such as convolutional neural network is used to analyze the image, identify landmarks, scenery, transportation and other information, and extract geographic points and corresponding related information as image information.

[0075] For the video part of the target web page, key frames are extracted through video analysis technology, scene changes, character activities, etc. in the video are identified, and the image and audio information are combined for understanding, so as to obtain the geographic points in the video and the corresponding related information as the video information.

[0076] It can be understood that when it is determined that a web page is composed of at least two of text, pictures, and videos, the text, pictures, and videos appearing in the web page need to be extracted separately using different extraction techniques to ensure the accuracy of the information extracted from each part of the content, thereby ensuring the accuracy of the target information obtained.

[0077] See also Figure 2 Further, step S20 specifically includes:

[0078] S201: Based on the multimodal large model, text information, image information and video information in the target information are integrated and multimodally represented to obtain multimodal information;

[0079] Specifically, when the target web page contains any two of text, picture, and video, the target information extracted from the target web page is at least two of the text information, picture information, and video information, and the contents of the at least two of the information are not exactly the same, and even completely different. Therefore, it is necessary to fuse the extracted text information, picture information, and video information, and integrate the text information, image information, and video information from different modalities into a unified representation space, so that the three can be aligned in the same semantic space, so that the multimodal information corresponding to the multimodal representation contains the information of each modality to capture the relevance between the modalities, so that the subsequent route generation becomes more efficient and accurate. Specifically, the geographic points under the text information, picture information, and video information and the corresponding associated information are fused. For example, the same geographic points and the corresponding associated information in the text information, picture information, and video information can be used as the core information of the fusion, and the associated information related to the core in the text information, picture information, and video information can be determined based on this core. If the associated information in the text information, picture information, and video information is different, the associated information of the three complements each other, and finally obtains multimodal information composed of core information and associated information.

[0080] S202: Determine the description information and geographic information corresponding to all target geographic points corresponding to the multimodal information through the attention mechanism of the multimodal large model.

[0081] Specifically, through the attention mechanism of the multimodal large model, the fusion of text information, image information and video information can be directly focused on the fusion of geographical points under the text information, image information and video information and the corresponding associated information. For example, the same geographical points and corresponding associated information in the text information, image information and video information can be used as the core information of the fusion, and the associated information related to the core in the text information, image information and video information can be determined based on this core. If the associated information in the text information, image information and video information is different, the associated information of the three can be supplemented with each other, and finally the description information and geographical information corresponding to all the target geographical points after the fusion can be determined based on the core information and associated information.

[0082] It can be understood that based on the multimodal large model, the text information, image information and video information in the target information are fused and represented multimodally, so as to associate similar semantics related to geographic points between the text information, image information and video information to reduce the heterogeneity gap, and through the additional semantics provided by the three, the complementarity of the semantic information related to geographic points between the three is achieved, so as to obtain more comprehensive, richer and more accurate multimodal information of multiple geographic points after fusion and complementation, which is conducive to improving the performance and accuracy of the multimodal large model; and the descriptive information and geographic information corresponding to all target geographic points corresponding to the multimodal information are determined through the attention mechanism of the multimodal large model, that is, the information of different modes is weighted through the attention mechanism, so that the multimodal large model can automatically focus on the most relevant parts, that is, automatically focus on and understand the geographic points and the corresponding descriptive information and geographic information in the multimodal information, and use the geographical points and the corresponding descriptive information and geographic information with high relevance as the descriptive information and geographic information corresponding to the target geographical points. This enables the integration of multimodal content about route information, thereby obtaining more accurate, comprehensive and core geographical points and related information, and the final automatically generated editable route map is more complete and accurate.

[0083] Furthermore, the description information includes scenic spot information, activity information, time information and traffic information.

[0084] It can be understood that the description information is specifically an introduction to a geographical point, including but not limited to scenic spot information, activity information, time information and transportation information. A geographical point is a place with actual geographical location information, which can be a range of places with a larger location area such as a city, county, tourist attraction, or an actual place such as a shopping mall, hotel, or restaurant. Scenic spot information includes but is not limited to the name of the geographical point, historical background and cultural connotation, natural landscape and characteristic buildings, major attractions, precautions and warm reminders, tourist evaluation and recommendation reasons, ticket prices and other information. Activity information includes but is not limited to information such as the activity experience and entertainment projects corresponding to the geographical point. Time information includes but is not limited to the opening hours of scenic spots, the opening hours of the activity experience and entertainment projects corresponding to the geographical point, the activity and project experience time, and the transportation time. Transportation information includes but is not limited to information on transportation that can reach the geographical point. Geographic information specifically refers to the actual geographical location of the geographical point on the map. Therefore, the multimodal large model can obtain the route text representation and / or route audio representation corresponding to the editable route based on the structured geographic points and the corresponding geographic information and the descriptive information including the scenic spot information, activity information, time information and traffic information, so as to realize the representation of recommended attractions and activities, the best mode of transportation between each target geographic point in the editable route, the recommended travel time and travel sequence for each target geographic point, precautions and tips for each target geographic point, etc.

[0085] See also Figure 3 Further, step S30 specifically includes:

[0086] S 301: Construct a knowledge graph with the target geographic point as the core based on multimodal information;

[0087] S 302: Connect the geographic information of all target geographic points and the scenic spot information, activity information, time information, and traffic information in the description information through the knowledge graph to obtain structured information.

[0088] Specifically, structured information refers to structured travel route information with geographic points as the core. By constructing a knowledge graph, the geographic information of all target geographic points and the scenic spot information, activity information, time information and traffic information in the description information can be connected through the knowledge graph to obtain structured information. For example, multimodal information can be arranged into a table format, marking the main attractions, activities, time, etc.

[0089] It can be understood that, by introducing entities, attributes and relationships in the knowledge graph, the entities and relationships corresponding to the geographical information and scenic spot information, activity information, time information and traffic information in the description information of all target geographical points are connected, thereby forming a structured travel route information with geographical points as the core. Through this structured information, the multimodal big model can obtain a deeper semantic understanding of the geographical information and description information of the target geographical points, so that the model multimodal big model can better understand the context and improve the accuracy of the route text representation and / or route audio representation of the editable route generated in the end.

[0090] Furthermore, after step S40, the method further includes:

[0091] S 50: responding to the user's route adjustment operation, adding, deleting or sequentially modifying geographic points in the editable route;

[0092] Specifically, since the editable route is a complete route map that can be directly edited and adjusted, if the user wants to modify the route, he or she can directly perform at least one route adjustment operation on the editable route's geographic points, including adding geographic points, deleting geographic points, and sequentially modifying multiple geographic points.

[0093] S 51 : Regenerate a new editable route and a corresponding route text representation and / or route audio representation based on all adjusted geographical points.

[0094] It can be understood that by responding to the user's route adjustment operation, the geographical points on the editable route can be directly added, deleted or modified in sequence, and a new editable route and corresponding route text representation and / or route audio representation can be regenerated based on all the adjusted geographical points. Thus, the initial editable route automatically generated by the multimodal large model is further edited and optimized, meeting the different personalized needs of users and ensuring the rationality and practicality of the generated editable route.

[0095] It should be noted that users can also provide feedback on the generated travel routes for further adjustments and optimizations, so that the multimodal large model can iterate based on the feedback and improve the generated results.

[0096] Furthermore, before step S20, the method further includes:

[0097] S19: Acquire route information annotation data of multiple modalities, train the multimodal large model, and obtain the trained multimodal large model.

[0098] Understandably, supervised training of a multimodal large model using route information annotation data of multiple modalities can optimize the performance of the trained multimodal large model on multimodal information, so that the optimized multimodal large model can better integrate information of different modalities, thereby improving the ability to recognize and understand complex scenes and content, and further improving the accuracy of extracting and understanding multimodal information. In addition, it can also help eliminate the phenomenon of information islands between different modalities, so that different modal information can better influence and complement each other, promote the collaborative work of multimodal large models between multiple modalities, and improve overall performance.

[0099] See also Figure 4 The second embodiment of the present invention provides a route generation system 1 based on a multimodal large model, which is used to implement the route generation method based on a multimodal large model described in any one of the first embodiments, including:

[0100] The information extraction module 10 is used to obtain a web page link and extract target information corresponding to the web page link;

[0101] An information determination module 20, for determining description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model;

[0102] The structuring module is used to perform structural processing on the description information and geographic information of all target geographic points to obtain structured information;

[0103] The route generation module 30 is used to automatically generate an editable route according to the structured information based on the multimodal large model, and generate a route text representation and / or a route audio representation of the editable route.

[0104] It can be understood that the route generation system 1 based on the multimodal big model is used to implement the route generation method based on the multimodal big model described in any one of the first embodiments, and has the same beneficial effects as the above-mentioned route generation method based on the multimodal big model, which will not be repeated here.

[0105] The third embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the route generation method based on a multimodal large model described in any one of the first embodiments is implemented. The method has the same beneficial effects as the route generation method based on a multimodal large model described above, and will not be described in detail here.

[0106] In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0107] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0108] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the above-mentioned processes does not mean the necessary order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] The flow chart and block diagram in the accompanying drawings of the present invention illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is determined based on the functions involved. It should be particularly noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs a specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0110] Compared with the prior art, the route generation method, system and storage medium based on the multimodal large model provided by the present invention have the following advantages:

[0111] 1. The embodiment of the present invention provides a route generation method based on a multimodal big model, comprising the following steps: obtaining a web page link, extracting target information corresponding to the web page link; the target web page corresponding to the web page link includes tourism information of at least two geographical points; determining the description information and geographical information corresponding to all target geographical points in the target information based on the multimodal big model; performing structured processing on the description information and geographical information of all target geographical points to obtain structured information; automatically generating an editable route based on the structured information based on the multimodal big model, and generating a route text representation and / or route audio representation of the editable route. The multimodal big model can be used to extract key information related to geographical points in the tourism information in the target web page corresponding to the input web page link, and use the key information as the target information, and perform structured processing on the description information and geographical information corresponding to all core target geographical points in the target information, so that the multimodal model automatically generates an editable route composed of multiple target geographical points in series according to the structured information, and introduces all places on the editable route. This enables the extraction, screening, summarization and structured processing of the web page content corresponding to the web link, so that an editable route with a starting point, an end point and the sequence of the route is automatically generated through a multimodal large model, and relevant information is described for the geographical points on the editable route, so that the automatically generated editable route can be more intuitively presented to the user, and then it is possible to directly use the travel guides on the platform to create travel routes to the places the user wants to go, without the user having to manually add places one by one to form a route, making the creation of the entire travel route simpler.

[0112] 2. The method of obtaining a web link and extracting target information corresponding to the web link in the embodiment of the present invention specifically includes: obtaining a web link and determining the target web page corresponding to the web link; the target web page is composed of at least one of text, picture, and video; extracting all geographic points and corresponding associated information existing in the target web page, and using all geographic points and corresponding associated information as target information. The extraction of geographic points and information related to geographic points in the target web page corresponding to the web link is realized, that is, the extraction of the core content of the travel information in the target network line is realized, avoiding the mixing of irrelevant content and affecting the generation of the final editable route.

[0113] 3. The method of extracting all geographic points and corresponding associated information in the target web page and using all geographic points and corresponding associated information as target information in the embodiment of the present invention specifically includes: determining whether the target web page includes at least two of text, pictures, and videos; if so, extracting all geographic points and corresponding associated information in all text, pictures, and videos in the target web page respectively to obtain text information, picture information, and video information; using text information, picture information, and video information as target information. When it is determined that the web page is composed of at least two of text, pictures, and videos, the text, pictures, and videos appearing in the web page need to be extracted separately using different extraction technologies to ensure the accuracy of the information extracted from each part of the content, thereby ensuring the accuracy of the target information obtained.

[0114] 4. In the embodiment of the present invention, the text information, picture information and video information in the target information are integrated based on the multimodal large model and multimodally represented, so as to associate similar semantics related to geographic points between the text information, picture information and video information to reduce the heterogeneity gap, and through the additional semantics provided by the three, the semantic information related to geographic points between the three is complemented, so as to obtain more comprehensive, richer and more accurate multimodal information of multiple geographic points after fusion and complementation, which is conducive to improving the performance and accuracy of the multimodal large model; and the description information and geographic information corresponding to all target geographic points corresponding to the multimodal information are determined through the attention mechanism of the multimodal large model, that is, the information of different modes is weighted through the attention mechanism, so that the multimodal large model can automatically focus on the most relevant part, that is, automatically focus on and understand the geographic points and the corresponding description information and geographic information in the multimodal information, and use the highly relevant geographic points and the corresponding description information and geographic information as the description information and geographic information corresponding to the target geographic points. This enables the integration of multimodal content about route information, thereby obtaining more accurate, comprehensive and core geographical points and related information, and the final automatically generated editable route map is more complete and accurate.

[0115] 5. The descriptive information and geographic information of all target geographic points in the embodiment of the present invention are structured to obtain structured information, specifically including: constructing a knowledge graph with the target geographic point as the core based on multimodal information; connecting the geographic information of all target geographic points and the scenic spot information, activity information, time information and traffic information in the descriptive information through the knowledge graph to obtain structured information. By introducing entities, attributes and relationships in the knowledge graph, the entities and relationships corresponding to the scenic spot information, activity information, time information and traffic information in the geographic information and descriptive information of all target geographic points are connected, so that the multimodal big model can obtain a deeper semantic understanding of the geographic information and descriptive information of the target geographic points, so that the model multimodal big model can better understand the context and improve the accuracy of the route text representation and / or route audio representation of the editable route finally generated.

[0116] 6. In the embodiment of the present invention, in response to the user's route adjustment operation, the geographical points in the editable route are added, deleted or sequentially modified; and a new editable route is regenerated based on all the adjusted geographical points. By responding to the user's adjustment operation, the geographical points on the editable route can be directly added, deleted or sequentially modified, thereby meeting the different personalized needs of users and ensuring the rationality and practicality of the generated editable route.

[0117] 7. In the embodiment of the present invention, the route information annotation data of multiple modes is obtained, and the multimodal large model is trained to obtain the trained multimodal large model. The performance of the multimodal large model on multimodal information is optimized, so that the optimized multimodal large model can better integrate information of different modes, thereby improving the recognition and understanding capabilities of complex scenes and content, and further improving the accuracy of extracting and understanding multimodal information. In addition, it can also help eliminate the information island phenomenon between different modes, so that different modal information can better influence and complement each other, promote the collaborative work of the multimodal large model between multiple modes, and improve the overall performance.

[0118] 8. The present invention also provides a route generation system based on a multimodal large model, which has the same beneficial effects as the above-mentioned route generation method based on a multimodal large model, and will not be elaborated here.

[0119] 9. The present invention also provides a computer-readable storage medium, which has the same beneficial effects as the above-mentioned route generation method based on the multimodal large model, and will not be described in detail here.

[0120] The above is a detailed introduction to a route generation method, system and storage medium based on a multimodal large model disclosed in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention. Any modifications, equivalent substitutions and improvements made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A route generation method based on a multimodal large model, characterized in that: The following steps are involved: Obtaining a webpage link and extracting target information corresponding to the webpage link; the target webpage corresponding to the webpage link includes tourism information of at least two geographical points; Determine the description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model; Structuring the description information and geographic information of all target geographic points to obtain structured information; An editable route is automatically generated according to the structured information based on the multimodal big model, and a route text representation and / or a route audio representation of the editable route is generated.

2. The route generation method based on a multimodal large model as claimed in claim 1, characterized in that: Get the web link and extract the target information corresponding to the web link, including: Obtain a webpage link and determine a target webpage corresponding to the webpage link; the target webpage is composed of at least one of text, picture, and video; All geographic points and corresponding associated information existing in the target webpage are extracted, and all geographic points and corresponding associated information are used as target information.

3. The route generation method based on the multimodal large model as claimed in claim 2, characterized in that: Extract all geographic points and corresponding related information in the target webpage, and use all geographic points and corresponding related information as target information, specifically including: Determine whether the target webpage includes at least two of text, image, and video; If yes, then extract all geographic points and corresponding associated information from all texts, pictures and videos in the target webpage to obtain text information, picture information and video information; Text information, picture information and video information are taken as target information.

4. The route generation method based on a multimodal large model as claimed in claim 3, characterized in that: Determine the description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model, including: Based on the multimodal large model, the text information, image information and video information in the target information are integrated and multimodally represented to obtain multimodal information; The descriptive information and geographic information corresponding to all target geographic points corresponding to the multimodal information are determined through the attention mechanism of the multimodal large model.

5. The route generation method based on a multimodal large model as claimed in claim 4, characterized in that: The description information includes scenic spot information, activity information, time information and traffic information.

6. The route generation method based on a multimodal large model as claimed in claim 5, characterized in that: Structural processing is performed on the description information and geographic information of all target geographic points to obtain structured information, including: Construct a knowledge graph with the target geographic point as the core based on multimodal information; The knowledge graph is used to connect the geographic information of all target geographic points and the scenic spot information, activity information, time information and traffic information in the description information to obtain structured information.

7. The route generation method based on a multimodal large model as claimed in claim 1, characterized in that: After automatically generating an editable route based on the structured information based on the multimodal large model and generating a route text representation of the editable route, the method further includes: In response to the user's route adjustment operation, add, delete or modify the order of geographical points in the editable route; A new editable route and a corresponding textual representation of the route and / or an audio representation of the route are regenerated based on all the adjusted geographic points.

8. The route generation method based on a multimodal large model as claimed in claim 4, characterized in that: Before determining the description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model, the method further includes: Acquire route information annotation data of multiple modalities, train the multimodal large model, and obtain the trained multimodal large model.

9. A route generation system based on a multimodal large model, used to implement the route generation method based on a multimodal large model as claimed in any one of claims 1 to 8, characterized in that: include: An information extraction module is used to obtain web page links and extract target information corresponding to the web page links; An information determination module is used to determine the description information and geographic information corresponding to all target geographic points in the target information based on the multimodal large model; The structuring module is used to perform structural processing on the description information and geographic information of all target geographic points to obtain structured information; The route generation module is used to automatically generate an editable route according to structured information based on the multimodal large model, and generate a route text representation and / or a route audio representation of the editable route.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a route generation method based on a multimodal large model as described in any one of claims 1 to 8.

Images