Travel itinerary planning method and system based on multi-modal data, terminal and medium

Abstract

The invention belongs to the technical field of travel itinerary planning, and particularly discloses a travel itinerary planning method and system based on multi-modal data, a terminal and a medium. Comprising the following steps: collecting text data, voice data and image data input by a user, and generating a multi-modal feature vector; inputting the multi-modal feature vector into a semantic analysis model, and outputting a semantic analysis result; executing corresponding processing based on the semantic analysis result; when the semantic analysis result is a question and answer request, generating a question and answer result based on semantic matching and knowledge reasoning; when the semantic analysis result is an accompanying interaction request, generating interaction content and driving a virtual accompanying role to output; when the semantic analysis result is an itinerary planning request, constructing a constraint optimization model and executing path search, and outputting at least one itinerary planning scheme; user feedback information is received in the whole service process, and the scheme is dynamically adjusted and iteratively updated. The problems that in the prior art, route planning and interaction are dispersed, and anthropomorphic accompanying service cannot be simulated are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of travel planning, and particularly relates to a travel itinerary planning method and system based on multi-modal data, a terminal and a medium. BACKGROUND

[0002] With the deep integration of the tourism industry and the Internet, intelligent tourism service platforms have gradually become mainstream. In the tourism travel scenario, users often obtain destination information, consult travel plans, book tickets and hotels through mobile terminals, and share travel experiences. In recent years, artificial intelligence, big data and natural language processing technologies have been introduced into tourism services, enabling travel recommendations based on historical data, consultation services based on dialogue systems, and path planning based on geographic information.

[0003] In the prior art, one type of method relies on single-modal data input, such as a tourism question and answer system based on text keyword matching or a recommendation model based on historical click behavior. Such systems can meet the basic information retrieval and travel planning needs of users to some extent. Another type of method attempts to incorporate sentiment analysis and knowledge graphs into the system to improve the accuracy of the recommendation results, but usually only introduces emotion labels or scoring parameters in the recommendation sorting stage, and fails to form a dynamic interaction model throughout the entire process. Some research proposes to provide tourism interpretation through virtual tours or digital avatars, but they are mostly fixed voice broadcasts or preset action plays, and cannot dynamically adjust the interaction content according to the real-time emotions and feedback of users.

[0004] The existing tourism travel planning and interaction technology still has the following shortcomings: first, the data processing dimension is relatively single, lacking cross-modal fusion and deep semantic analysis, resulting in inaccurate intent recognition and often generalized and lack of personalized recommendation results. Second, the virtual interaction mechanism remains at the static level, lacking emotional adaptation and natural dialogue capabilities, and cannot simulate continuous companion services of humanized companions. SUMMARY

[0005] The present application provides a tourism travel planning method and system based on multi-modal data to solve the problem of lack of cross-modal fusion and deep semantic analysis in the background art, resulting in inaccurate intent recognition and often generalized and lack of personalized recommendation results. It also solves the problem of virtual interaction mechanism remaining at the static level in the background art, lacking emotional adaptation and natural dialogue capabilities, and being unable to simulate continuous companion services of humanized companions.

[0006] The technical solution adopted by the present application is as follows: In a first aspect, the present application provides a tourism travel planning method based on multi-modal data, which comprises the following steps: Step S1, collect user input text data, voice data and image data, perform preprocessing on the input data, extract text features, voice features and image features, fuse the text features, voice features and image features and generate a multi-modal feature vector; Step S2, input the multi-modal feature vector into a semantic analysis model, output a semantic analysis result, the semantic analysis result includes a question and answer request, a companion interaction request or a travel planning request; Step S3, based on the semantic analysis result, perform corresponding processing, including: When the semantic analysis result is a question and answer request, generate a question and answer result based on semantic matching and knowledge reasoning; When the semantic analysis result is a companion interaction request, generate interaction content in combination with the multi-modal feature vector and drive a virtual companion character to output; When the semantic analysis result is a travel planning request, construct a constraint optimization model and perform path search in combination with travel time, budget, interest preference and companion information, and output at least one travel planning scheme; Step S4, receive user feedback information throughout the service, dynamically adjust and iteratively update the question and answer result, companion interaction content and travel planning scheme.

[0007] Further, in step S1: Perform word segmentation, word vector encoding and semantic feature extraction on the collected text feature data, use BERT or RoBERTa model to convert text feature input into high-dimensional semantic vector; Perform endpoint detection and noise removal on the collected voice feature data, extract acoustic features based on mel-frequency cepstral coefficients, and generate emotion features through an emotion recognition network; Perform convolutional neural network or visual Transformer processing on the collected image feature data, output scene category labels and corresponding probabilities, and obtain image embedding vectors; Concatenate the scene category probability and the image embedding vector and input them into an interest mapping model, the interest mapping model is a multi-layer network structure, and outputs a weight distribution on a preset interest dimension set; the weight distribution is used as a user interest bias feature corresponding to the image.

[0008] Further, in step S2: Input the multi-modal feature vector into the multi-channel encoding layer of the semantic analysis model; The multi-channel encoded result is aligned across modalities through a multi-head attention mechanism in the semantic analysis stage, and a unified semantic representation is output; Input the unified semantic representation into an intent classification network to generate a probability distribution belonging to a question and answer request, a companion interaction request or a travel planning request, and determine the analysis result based on the maximum likelihood criterion; A confidence score is recorded when determining the resolution result, and a clarification dialogue process is triggered when the confidence is lower than a threshold.

[0009] Further, in step S3: When the semantic resolution result is a question and answer request, knowledge graph retrieval and semantic similarity calculation are performed, the user question is matched with RDF triples in a preset knowledge base, and the results are filtered based on a cosine similarity threshold; When the semantic resolution result is a companion interaction request, a context memory network is constructed, the user's historical dialogue state is modeled, and the context information and real-time emotion parameters are combined when generating the current interaction content to drive the synchronous output of the virtual companion's speech synthesis and expression animation; When the semantic resolution result is a travel planning request, the user's constraint conditions are mapped to optimization variables, and multiple candidate travel plans are generated based on genetic algorithm or reinforcement learning path search, and are sorted according to interest preference and budget matching degree.

[0010] Further, when the semantic resolution result is a companion interaction request, further comprising: Extracting user voice tone features and text emotion labels to form a user real-time emotion vector, the emotion vector being composed of multiple parameter dimensions, each dimension corresponding to a different emotion factor; In the interaction generation stage, the user emotion vector and the semantic features are spliced and input into the emotion adaptive network to output semantic representations fused with emotion features; The virtual companion role drives the language generation model based on the semantic representation to synthesize voice content with emotion regulation, and simultaneously calls the action library to output expressions and actions corresponding to the parameters of the emotion vector.

[0011] Further, in the interaction generation stage, the user real-time emotion vector and the semantic features are spliced and input into the emotion adaptive network to output semantic representations fused with emotion features, and the emotion adaptive network adopts the following calculation method:

[0012] Wherein, is the semantic representation; ; is the semantic feature matrix, , is the user emotion vector, is the temperature coefficient, is the query mapping matrix of the semantic features, is the key mapping matrix of the emotion vector;

[0013]

[0014] is a scale factor generation matrix, is a scale factor bias vector, is an offset factor generation matrix, is an offset factor bias vector;

[0015] is a global representation of semantic features, is a semantic projection matrix, is an emotion projection matrix, r is an interaction rank parameter, represents a per-dimension multiplication;

[0016] is a signal-to-noise ratio indicator of the input speech, is a cosine similarity between the emotion vector and the semantic mean vector; is a mean vector of the semantic feature matrix H, is a weight vector, is a scalar bias.

[0017] Further, in step S4: Periodically collect user feedback information throughout the service cycle, and the feedback information includes text feedback, voice feedback and operation behavior data; According to the feedback content, dynamically adjust the question and answer retrieval weight, the companion interaction strategy and the travel planning parameter; In the feedback processing process, a mapping relationship between user feedback and output results is established, and a feedback history database is generated; The feedback history database is called each time to update, and the current service output is compared and optimized.

[0018] In a second aspect, the application provides a travel itinerary planning system based on multi-modal data, which is used to realize the travel itinerary planning method based on multi-modal data as described in the first aspect, and the system comprises: An input interface unit is configured to receive text data, voice data and image data input by a user, and transmit the received data to a preprocessing and feature extraction unit; The preprocessing and feature extraction unit is configured to perform word segmentation, word vector encoding and semantic feature extraction on the text data, perform endpoint detection, noise removal and extract acoustic and emotion features on the voice data, perform convolutional neural network or visual Transformer processing on the image data to obtain category labels and embedding vectors, and generate a unified multi-modal feature vector based on a gating mechanism through a multi-modal fusion layer, and output the multi-modal feature vector to a semantic analysis unit; The semantic parsing unit receives the multimodal feature vectors output by the preprocessing and feature extraction unit, performs multi-channel encoding and cross-modal alignment, generates a unified semantic representation, and outputs the unified semantic representation to the intent recognition unit. The intent recognition unit is used to receive the unified semantic representation output by the semantic parsing unit, generate the probability distribution of the request to answer questions, the escort interaction request or the itinerary planning request, determine the parsing result and output it to the corresponding processing unit. When the confidence level is lower than the threshold, the clarification request is sent to the input interface unit. The question-answering processing unit is used to perform knowledge graph retrieval and semantic similarity calculation when it receives a question-answering request from the intent recognition unit. It matches the user's question with the triples in the knowledge base, obtains the question-answering result, and outputs it to the feedback collection and update unit. The escort interaction unit is used to call the user's emotional features and semantic features when it receives the escort interaction request transmitted by the intention recognition unit, concatenate them and input them into the emotion adaptive network to generate a semantic representation that integrates emotional features, and drive the speech synthesis and facial expression control module to output the interaction content based on the semantic representation, and transmit the output interaction content to the feedback acquisition and update unit. The trip planning unit is used to, upon receiving a trip planning request from the intent recognition unit, combine the user's travel time, budget, interests and travel companion information, construct a constrained optimization model and perform path search to generate at least one trip planning scheme, and output the scheme to the feedback collection and update unit. The feedback collection and update unit is used to periodically receive users' text feedback, voice feedback, and operation behavior data throughout the service process, establish a mapping relationship between feedback and output results, dynamically adjust and iteratively update the outputs of the question-and-answer processing unit, the escort interaction unit, and the itinerary planning unit, and return the updated results to the user terminal.

[0019] Thirdly, this application provides a terminal, including: Memory, used to store tourism itinerary planning simulation programs based on multimodal data; A processor is configured to implement the steps of the multimodal data-based travel itinerary planning method as described in the first aspect when executing the multimodal data-based travel itinerary planning device.

[0020] Fourthly, this application provides a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the travel itinerary planning method based on multimodal data as described in the first aspect.

[0021] As can be seen from the above technical solutions, the advantages of the present invention are: By sequentially setting multi-modal data acquisition and feature extraction, semantic analysis and intent recognition, branch processing, virtual companion interaction, travel planning and feedback dynamic updating, a closed-loop system from user input to personalized output can be constructed, solving the problem of scattered travel planning and interaction and lack of integrated processing in the prior art.

[0022] By performing feature extraction in the dimensions of text, voice and image respectively, and performing weighted combination based on the gating mechanism in the fusion layer, the contribution of different modal data in the unified representation can be improved, thereby enhancing the accuracy of user intent recognition, solving the problem of single modal dependence leading to recommendation generalization and insufficient semantic analysis in the prior art.

[0023] By realizing cross-modal feature alignment through multi-channel encoding and multi-head attention mechanism, and introducing intent classification and confidence evaluation mechanism, higher robustness can be achieved in analyzing user requests, and clarification interaction can be triggered when confidence is insufficient, solving the problem of high semantic analysis misjudgment rate and lack of self-correction mechanism in the prior art.

[0024] By constructing knowledge graph retrieval, context memory modeling and intelligent optimization planning three processing mechanisms for three types of requests of question and answer, companion interaction and travel planning, precise response under different demands can be achieved, avoiding the functional limitations and insufficient user experience caused by single processing logic in the prior art.

[0025] By introducing a sentiment adaptive network, the user emotion vector and semantic feature are spliced and processed, which can simultaneously consider both semantic and emotional factors when generating interactive content, realize dynamic driving of speech synthesis and action output, and solve the problem of existing virtual guide staying in static broadcast and lack of emotional interaction.

[0026] By using the combined calculation method of cross-modal attention, feature modulation, bilinear interaction and reliability weight in the sentiment adaptive network, the nonlinear relationship between semantic features and emotional features can be modeled in detail, so that the output fusion semantic representation is more consistent with the user's emotional state, solving the problem of rough emotional modeling and unnatural output in the prior art.

[0027] By continuously collecting text, voice and behavior feedback throughout the service, and establishing a mapping relationship between feedback and results, the parameters of question and answer retrieval, companion strategy and travel planning can be dynamically optimized in the process of multiple interactions, solving the problem of single update of feedback and lack of continuous adaptive ability in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to make the technical solutions of the present application clearer, the drawings needed to be used in the description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor based on these drawings also belong to the protection scope of the present application.

[0029] Figure 1 A step diagram of the travel itinerary planning method based on multi-modal data in the embodiments is shown in the following. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor also belong to the protection scope of the present application.

[0031] The terms in the present application are explained as follows: RDF triple: RDF triple is a form of knowledge representation, which is used to organize and store data in a knowledge base. RDF triple is composed of three parts of subject, predicate and object. The subject is used to represent the object described, the predicate is used to represent the attribute of the object or the relationship between other objects, and the object is used to represent the specific value of the attribute or another object associated with the subject.

[0032] For example, when describing the knowledge that "West Lake is open all day", it can be represented as a triple structure of "West Lake-open time-all day", where "West Lake" is the subject, "open time" is the predicate, and "all day" is the object.

[0033] Please refer to Figure 1 As shown in the drawings, the present application provides a travel itinerary planning method based on multi-modal data, which comprises the following steps: Step S1, collect the text data, voice data and image data input by the user, perform preprocessing on the input data, extract text features, voice features and image features, fuse the text features, voice features and image features and generate a multi-modal feature vector; Perform word segmentation, word vector encoding and semantic feature extraction on the collected text feature data, and convert the text feature input into a high-dimensional semantic vector by using a BERT or RoBERTa model; Perform endpoint detection and noise elimination on the collected voice feature data, extract acoustic features based on mel-frequency cepstral coefficients, and generate emotion features through an emotion recognition network; Perform convolutional neural network or visual Transformer processing on the collected image feature data, output the scenic spot category label and corresponding probability, and obtain the image embedding vector; The scenic spot category probability and the image embedding vector are spliced and input into the interest mapping model. The interest mapping model is a multi-layer network structure, and outputs the weight distribution on the preset interest dimension set. The weight distribution is used as the user interest bias feature corresponding to the image.

[0034] In the system, the user can input a text question, voice instruction or upload a picture through a mobile terminal application. The system automatically identifies the data type through a data preprocessing module. The text data is segmented and converted into a word vector after input, and a high-dimensional representation reflecting the user's semantic intent is obtained through a semantic feature extraction model. When the voice data is input, the environmental noise is first removed and the effective speech segment is divided, and then the acoustic features are extracted and the emotional factors contained in the voice are recognized. After the image data is uploaded, the scenic spot category, composition feature and user interest tendency are obtained through a visual recognition model. All features are finally combined through a multi-modal fusion layer to form a unified vector representation. In the embodiment, the user inputs "recommend a two-day tour of Shanghai" in the application and uploads a photo of the Bund. The system processes the text and image features simultaneously, combines the interest intent of the text with the city label recognized by the image, and outputs a vector representation highly matched with the user's demand.

[0035] Step S2, input the multi-modal feature vector into the semantic analysis model, output the semantic analysis result, and the semantic analysis result includes a question and answer request, a companion interaction request or a travel planning request; The multi-modal feature vector is input into the multi-channel encoding layer of the semantic analysis model; The multi-channel encoded result is aligned across modalities through a multi-head attention mechanism in the semantic analysis stage, and a unified semantic representation is output; The unified semantic representation is input into the intent classification network to generate a probability distribution belonging to the question and answer request, the companion interaction request or the travel planning request, and the analysis result is determined based on the maximum likelihood criterion; A confidence score is recorded when the analysis result is determined, and a clarification dialogue process is triggered when the confidence is lower than a threshold.

[0036] During system operation, the multi-modal feature vectors from step S1 are input into different encoding channels respectively, the text extracts context-related semantics through a language model, the voice extracts voice intonation and emotion features through an emotion recognition network, and the image identifies scene categories and interest preferences through a visual model. All results are fused into a unified semantic representation in the analysis stage and passed to the intent classification network. The intent classification network outputs the request category belonging to question answering, companion interaction, or travel planning according to the training results. If the result confidence is insufficient, the system will immediately initiate a clarification request to the user to avoid misidentification. In the embodiment, the user inputs “Is there a family-friendly playground nearby?” The system identifies it as a travel planning request during analysis, but the confidence is low, so it pops up an option to confirm whether it is “querying specific locations” or “recommending a travel route”. After the user confirms, the system updates the analysis result.

[0037] Step S3, performing corresponding processing based on the semantic analysis result, including: When the semantic analysis result is a question answering request, generating a question answering result based on semantic matching and knowledge reasoning; When the semantic analysis result is a companion interaction request, generating interaction content in combination with the multi-modal feature vector and driving the virtual companion character to output; When the semantic analysis result is a travel planning request, constructing a constraint optimization model and performing path search in combination with travel time, budget, interest preferences, and companion information, and outputting at least one travel planning scheme; When the semantic analysis result is a question answering request, performing knowledge graph retrieval and semantic similarity calculation, matching the user's question with the RDF triples in the preset knowledge base, and filtering the results based on the cosine similarity threshold; When the semantic analysis result is a companion interaction request, constructing a context memory network to model the user's historical dialogue state, combining context information and real-time emotion parameters when generating current interaction content, and driving the virtual companion's speech synthesis and expression animation to be output synchronously; When the semantic analysis result is a travel planning request, mapping user constraints to optimization variables, generating multiple candidate travel schemes based on genetic algorithms or reinforcement learning path search, and sorting them according to interest preferences and budget matching degree.

[0038] Extracting user voice intonation features and text emotion labels to form a user real-time emotion vector, the emotion vector consisting of multiple parameter dimensions, each dimension corresponding to a different emotion factor; In the interaction generation stage, the user emotion vector and semantic features are concatenated and input into the emotion adaptive network to output semantic representations fused with emotion features; The virtual companion character drives the language generation model based on the semantic representation to synthesize voice content with emotion regulation, and simultaneously calls the action library to output expressions and actions corresponding to the parameters of the emotion vector.

[0039] In the interaction generation stage, the user real-time emotion vector is spliced with the semantic feature and input into the sentiment adaptive network, and the semantic representation fused with the emotion feature is output. The sentiment adaptive network adopts the following calculation method:

[0040] wherein, is the semantic representation; ; is the semantic feature matrix, , is the user emotion vector, is the temperature coefficient, is the query mapping matrix of the semantic feature, is the key mapping matrix of the emotion vector;

[0041]

[0042] is the scale factor generation matrix, is the scale factor bias vector, is the offset factor generation matrix, is the offset factor bias vector;

[0043] is the global representation of the semantic feature, is the semantic projection matrix, is the emotion projection matrix, r is the interaction rank parameter, represents the element-wise multiplication;

[0044] is the signal-to-noise ratio index of the input speech, is the cosine similarity between the emotion vector and the semantic mean vector; is the mean vector of the semantic feature matrix H, is the weight vector, is the scalar bias.

[0045] In the processing link of the question and answer request, the system matches the entities and relations in the user's question through knowledge graph retrieval, and screens the most relevant answers in combination with semantic similarity. In the companion interaction request, the system understands the user's historical dialogue through the context memory network, and generates more natural interactive content in combination with real-time emotional parameters. The virtual companion role can be manifested as the synchronous generation of voice tone and expression action, enhancing the realism of interaction. For the itinerary planning request, the system converts the user's input time, budget and interest into constraint conditions, calls the optimization algorithm to search for feasible routes, and sorts multiple candidate schemes according to the user's characteristics. In the embodiment, when the user queries "help me recommend a two-day tour for couples", the system analyzes it as an itinerary planning request, generates multiple candidate routes in combination with the budget and interest, and updates the optimal result according to the user's feedback.

[0046] In an implementation manner, the system adopts a genetic algorithm for searching. First, the representation of the candidate itinerary is coded, and each itinerary is represented as a chromosome structure containing a sequence of scenic spots and corresponding stay time, wherein the gene of the chromosome is a single scenic spot node with stay time information. The system randomly generates several initial solutions of itineraries containing different scenic spot sequences and stay time allocation in the initialization stage, ensuring that the total length of each solution does not exceed the upper limit of the travel time input by the user.

[0047] Then, the fitness value of each initial solution is calculated. The fitness is composed of three main factors: one is the scenic spot category satisfaction, which scores the corresponding coverage degree in the candidate scenic spots according to the user's interest preference; the second is the total cost constraint, which calculates the penalty term according to the deviation of the ticket fee, transportation fee and accommodation fee involved in the itinerary from the user's budget; the third is the itinerary compactness, which balances the score according to the traffic distance between scenic spots and the continuity of daily itinerary. The three are weighted according to the preset weight to obtain the comprehensive fitness value of each solution.

[0048] In the iteration process, the system selects part of the itinerary solutions as parents according to the fitness from high to low, and generates new schemes through the crossover operator. The crossover operation is to select segments of the same length in the scenic spot sequences of the two parent itineraries to exchange, and adjust the time arrangement of the exchanged itinerary to ensure that it does not exceed the upper limit of the travel time. For the generated new scheme, the system adjusts it randomly through the mutation operator, and the mutation operation includes randomly replacing a scenic spot in the itinerary sequence, exchanging the order of two scenic spots or fine-tuning the stay time of a scenic spot to increase the diversity of the solution. If the total cost exceeds the budget, the travel time exceeds the constraint, etc. after crossover or mutation, the solution is directly eliminated.

[0049] The iteration process continues until the termination condition is reached, which can be that the number of iterations reaches the preset upper limit, or the optimal fitness does not improve for a certain number of generations. Finally, the multiple high-fitness itinerary schemes that remain are output as candidate schemes.

[0050] In one specific embodiment, when the user inputs "a two-day trip for couples, budget 3000 yuan, preference for natural scenery and cultural sites", the system first encodes about 20 eligible scenic spots in the candidate scenic spot library, randomly generates 100 initial solutions containing different scenic spot combinations and stay times. In the iteration process, the system gradually eliminates solutions that do not meet the budget or time constraints, and finally outputs three high-adaptability solutions, one of which contains the itinerary "West Lake - Lingyin Temple - Leifeng Pagoda - lakeside hotel", ranked as the optimal recommendation by the system.

[0051] Step S4, receiving user feedback information throughout the service, dynamically adjusting and iteratively updating the question and answer results, companion interaction content and itinerary planning scheme; Periodically collect user feedback information throughout the service cycle, and the feedback information includes text feedback, voice feedback and operation behavior data; According to the feedback content, dynamically adjust the question and answer retrieval weight, companion interaction strategy and itinerary planning parameter; In the feedback processing process, a mapping relationship between user feedback and output results is established, and a feedback history database is generated; Each time the feedback history database is called to compare and optimize the current service output.

[0052] During the service, the system continuously collects user feedback, such as user voice opinions, text modification requests and interaction behavior records, and updates the system database in real time. After collecting feedback, the system iteratively adjusts the retrieval weight of the question and answer module, the action strategy of the companion interaction and the parameters of the itinerary planning according to the feedback. Through the accumulation of the feedback history database, the system can use past experience to optimize the response in subsequent interactions. In the embodiment, the user expresses "the hotel is too expensive" during the use process, and the system adjusts the budget parameter in real time, re-recommends more eligible accommodation and routes, and gives priority to the user's budget preference in subsequent similar scenarios.

[0053] In some embodiments, the present application provides a tourism itinerary planning system based on multi-modal data, which comprises: An input interface unit for receiving user input text data, voice data and image data, and transmitting the received data to the pre-processing and feature extraction unit; During system operation, the user can input questions, voice instructions or upload pictures through the interface of the mobile terminal. The input interface unit is responsible for unified reception of different inputs, identification of data types, and packaging of identification results and original data for transmission to the pre-processing and feature extraction unit. In the embodiment, the user inputs "recommend a three-day trip to Beijing" in the application while uploading a photo of the Forbidden City. The input interface unit receives and transmits the text information and picture data to the downstream at the same time.

[0054] a pre-processing and feature extraction unit configured to perform word segmentation, word vector encoding and semantic feature extraction on text data, perform endpoint detection, noise removal and extract acoustic and emotion features on speech data, perform convolutional neural network or visual Transformer processing on image data to obtain category labels and embedding vectors, and generate unified multi-modal feature vectors based on a gating mechanism through a multi-modal fusion layer, and output the multi-modal feature vectors to a semantic analysis unit; After receiving the data from the input interface unit, the pre-processing and feature extraction unit performs dedicated processing on different modal data respectively. The text data is converted into semantic vectors for subsequent semantic analysis; the speech data is processed to obtain the user's tone and potential emotions; and the image data is recognized by a recognition model to obtain the category label of the scenic spot. The multi-modal fusion layer weights different modal data based on a gating mechanism to ensure that the modal with higher semantic relevance occupies a larger weight in the final fusion. In an embodiment, the user inputs "a park suitable for family play" with a voice request, and the system highlights the user's family scene intent after fusing the text features and speech features.

[0055] a semantic analysis unit configured to receive the multi-modal feature vectors output by the pre-processing and feature extraction unit, perform multi-channel encoding and cross-modal alignment, generate a unified semantic representation, and output the unified semantic representation to an intent recognition unit; After the semantic analysis unit receives the multi-modal fusion vector, it inputs the multi-modal fusion vector into a multi-channel encoding model to align and integrate the embeddings of different modalities, obtaining a unified semantic representation for representing the user's complete input intent. In an embodiment, the user says "help me design a relaxing route for the elderly" through voice and uploads a photo of the Summer Palace. The semantic analysis unit fuses the voice and image features to obtain a unified representation, indicating that the user's demand is a leisure tour route for the elderly.

[0056] an intent recognition unit configured to receive the unified semantic representation output by the semantic analysis unit, generate a probability distribution belonging to a question and answer request, a companion interaction request or a travel planning request, determine an analysis result and output the analysis result to a corresponding processing unit, and send a clarification request to the input interface unit when the confidence is lower than a threshold value; The intent recognition unit classifies the unified semantic representation to obtain the corresponding request type. If the classification result probability is lower than a set threshold value, the clarification mechanism is triggered, a clarification request is sent to the input interface unit, and the user is prompted to make a supplementary explanation. In an embodiment, the user inputs "recommend a good restaurant", and the system determines that the restaurant type is not clear. The intent recognition unit outputs a low confidence result, and then requests the user to select "Chinese restaurant" or "Western restaurant".

[0057] a question and answer processing unit, configured to, when receiving a question and answer request delivered by the intent recognition unit, perform knowledge graph retrieval and semantic similarity calculation, match the user question with the triples in the knowledge base, obtain a question and answer result, and output the question and answer result to the feedback collection and update unit; After receiving the question and answer request, the question and answer processing unit accesses the knowledge graph stored in the system, matches the keywords in the user question with the entities in the knowledge graph, and filters out the most relevant result in combination with semantic similarity calculation. In the embodiment, the user inputs "West Lake opening time", and the question and answer processing unit finds the triple "West Lake - opening time - open all day" in the knowledge graph, and feeds back the result to the user.

[0058] a companion interaction unit, configured to, when receiving a companion interaction request delivered by the intent recognition unit, call the user emotion features and semantic features, input the user emotion features and semantic features into the sentiment adaptive network to generate a semantic representation fused with the emotion features, drive the voice synthesis and expression action control module to output interaction content based on the semantic representation, and deliver the output interaction content to the feedback collection and update unit; The companion interaction unit receives the instruction of the intent recognition unit in runtime, combines the semantic content input by the user with the extracted emotion features, and generates a semantic representation with emotional color through the sentiment adaptive network. Then the virtual companion character is driven to output corresponding voice and expression actions, thereby improving the naturalness of interaction. In the embodiment, the user inputs "I am a little tired" at night, the companion interaction unit recognizes the low-energy emotion in the voice, adjusts the interaction content to "Do you need to rest? I can help you plan a relaxed subsequent itinerary", and accompanies with soothing voice tone and expression.

[0059] a travel planning unit, configured to, when receiving a travel planning request delivered by the intent recognition unit, combine the user travel time, budget, interest preference and companion information, construct a constraint optimization model and perform path search to generate at least one travel planning scheme, and output the scheme to the feedback collection and update unit; After receiving the request, the travel planning unit first parses the constraint conditions input by the user, such as travel date, budget range and interest direction, then calls the planning algorithm to filter the travel schemes that meet the conditions in the candidate resource library, and performs path search to obtain the optimal solution. In the embodiment, the user inputs "budget 3000 yuan, two days, like natural scenery", and the travel planning unit selects multiple scenic spots and accommodation combinations in the candidate resources to output a travel scheme including "Huangshan two-day tour".

[0060] The feedback collection and updating unit is configured to periodically receive the text feedback, voice feedback and operation behavior data of the user in the whole service process, establish a mapping relationship between the feedback and the output result, dynamically adjust and iteratively update the output of the question and answer processing unit, the companion interaction unit and the travel plan unit, and return the updated result to the user terminal.

[0061] The feedback collection and updating unit continuously records the reaction of the user in the interaction, including text modification, voice correction and operation behavior on the recommended result. Based on the difference between the historical feedback and the current output, the unit dynamically adjusts the parameters of the upstream unit, so that the system has self-adaptive ability. In the embodiment, the user selects "cancel hotel A and change to hotel B" in the first travel recommendation, and the system writes the preference into the feedback database. The next time, hotel B or similar options are preferentially recommended under similar budget, and the updated result is directly returned to the user terminal.

[0062] In some embodiments, the present application provides a terminal, comprising: a memory configured to store a travel itinerary planning program based on multi-modal data; a processor configured to execute the travel itinerary planning program based on multi-modal data to implement the steps of the travel itinerary planning method based on multi-modal data.

[0063] In some embodiments, the present application provides a computer readable storage medium, which stores computer instructions. When the computer reads the computer instructions in the storage medium, the computer executes the travel itinerary planning method based on multi-modal data.

[0064] It can be understood that the system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a personal digital assistant, a tablet computer, a wearable device or a combination of any of these devices.

[0065] In a typical configuration, the computer includes one or more processors (CPU), input / output interfaces, network interfaces and memories.

[0066] The memory can include a non-persistent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM). The memory is an example of the computer readable medium.

[0067] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage, quantum memory, graphene-based storage medium or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0068] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0069] It should be understood that although one or more embodiments of the specification can employ the terms first, second, third, etc. to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the specification, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0070] The above description is only the preferred embodiment of one or more embodiments of the specification and does not limit one or more embodiments of the specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of one or more embodiments of the specification shall be included in the scope of protection of one or more embodiments of the specification.

Claims

1. A tourism itinerary planning method based on multimodal data, characterized in that, Includes the following steps: Step S1: Collect user-input text data, voice data, and image data; perform preprocessing on the input data; extract text features, voice features, and image features; fuse the text features, voice features, and image features to generate a multimodal feature vector. Step S2: Input the multimodal feature vector into the semantic parsing model and output the semantic parsing results, which include question-answering requests, escort interaction requests, or itinerary planning requests. Step S3: Perform corresponding processing based on the semantic parsing results, including: When the semantic parsing result is a question-answering request, a question-answering result is generated based on semantic matching and knowledge reasoning. When the semantic parsing result is a companion interaction request, the interactive content is generated by combining the multimodal feature vector and driving the virtual companion character to output; When the semantic parsing result is a trip planning request, a constrained optimization model is constructed by combining travel time, budget, interests and preferences and information of fellow travelers, and a path search is performed to output at least one trip planning solution; Step S4: Receive user feedback throughout the service process and dynamically adjust and iterate the Q&A results, escort interaction content, and itinerary planning scheme.

2. The tourism itinerary planning method based on multimodal data according to claim 1, characterized in that, In step S1: The collected text feature data is segmented, word vector encoded, and semantic feature extracted. The BERT or RoBERTa model is used to transform the text feature input into a high-dimensional semantic vector. Endpoint detection and noise cancellation are performed on the collected speech feature data, acoustic features are extracted based on Mel frequency cepstral coefficients, and emotion features are generated through an emotion recognition network. The collected image feature data is processed by a convolutional neural network or a visual Transformer to output the scenic spot category labels and corresponding probabilities, and to obtain the image embedding vector. The probability of the attraction category is concatenated with the image embedding vector and then input into the interest mapping model. The interest mapping model is a multi-layer network structure that outputs the weight distribution on a preset set of interest dimensions. The weight distribution serves as the user interest bias feature corresponding to the image.

3. The tourism itinerary planning method based on multimodal data according to claim 1, characterized in that, In step S2: The multimodal feature vectors are input into the multichannel encoding layer of the semantic parsing model respectively; The results of multi-channel encoding are aligned across modalities during the semantic parsing stage using a multi-head attention mechanism to output a unified semantic representation. The unified semantic representation is input into the intent classification network to generate a probability distribution belonging to a question-and-answer request, a companion interaction request, or a trip planning request. The parsing result is determined based on the maximum likelihood criterion. When determining the analysis result, the confidence score is recorded. When the confidence score is lower than the threshold, a clarification dialogue process is triggered.

4. The tourism itinerary planning method based on multimodal data according to claim 1, characterized in that, In step S3: When the semantic parsing result is a question-and-answer request, knowledge graph retrieval and semantic similarity calculation are performed. The user's question is matched with RDF triples in the preset knowledge base, and the results are filtered based on the cosine similarity threshold. When the semantic parsing result is a companion interaction request, a context memory network is constructed to model the user's historical dialogue state. When generating the current interaction content, context information and real-time emotion parameters are combined to drive the virtual companion's speech synthesis and facial expression animation to be output synchronously. When the semantic parsing result is a trip planning request, the user constraints are mapped to optimization variables, and multiple candidate trip plans are generated based on genetic algorithm or reinforcement learning path search, and sorted according to interest preference and budget matching degree.

5. The tourism itinerary planning method based on multimodal data according to claim 4, characterized in that, When the semantic parsing result is a companion interaction request, it further includes: Extract user voice tone features and text emotion tags to form a real-time emotion vector. The emotion vector consists of multiple parameter dimensions, each of which corresponds to a different emotion factor. During the interaction generation stage, the user's emotion vector and semantic features are concatenated and input into the emotion adaptive network, which outputs a semantic representation that integrates emotion features. The virtual escort character uses a semantic representation-driven language generation model to synthesize speech content with emotion regulation, while calling an action library to output facial expressions and actions corresponding to the parameters of the emotion vector.

6. The tourism itinerary planning method based on multimodal data according to claim 5, characterized in that, In the interaction generation stage, the user's real-time emotion vector is concatenated with semantic features and then input into the emotion adaptation network. The output is a semantic representation that integrates emotion features. The emotion adaptation network uses the following calculation method: in, For semantic representation; ； The semantic feature matrix, , For user emotion vectors, For temperature coefficient, The query mapping matrix is ​​a semantic feature. The key mapping matrix for the emotion vector; Generate a matrix for the scale factor. This is the scale factor bias vector. Generate a matrix for the offset factor. This is the offset vector of the offset factor; This is a global representation of semantic features. For semantic projection matrix, Let r be the emotion projection matrix, and r be the interaction rank parameter. This represents dimension-wise multiplication; The signal-to-noise ratio (SNR) is the metric for input speech. The cosine similarity between the emotion vector and the semantic mean vector; Let H be the mean vector of the semantic feature matrix. For the weight vector, This is a scalar bias.

7. The tourism itinerary planning method based on multimodal data according to claim 1, characterized in that, In step S4: User feedback is collected periodically throughout the service process, including text feedback, voice feedback, and user behavior data. Dynamically adjust the question-and-answer search weights, escort interaction strategies, and itinerary planning parameters based on feedback. During the feedback processing, a mapping relationship between user feedback and output results is established, and a feedback history database is generated. Each update calls the feedback history database to compare and optimize the current service output.

8. A tourism itinerary planning system based on multimodal data, used to implement the tourism itinerary planning method based on multimodal data as described in claim 1, characterized in that, The system includes: The input interface unit is used to receive text data, voice data and image data input by the user, and to transmit the received data to the preprocessing and feature extraction unit; The preprocessing and feature extraction unit is used to perform word segmentation, word vector encoding and semantic feature extraction on text data, perform endpoint detection, noise removal and extract acoustic and emotion features on speech data, perform convolutional neural network or visual Transformer processing on image data to obtain category labels and embedding vectors, and generate a unified multimodal feature vector based on a gating mechanism through a multimodal fusion layer, and output the multimodal feature vector to the semantic parsing unit. The semantic parsing unit receives the multimodal feature vectors output by the preprocessing and feature extraction unit, performs multi-channel encoding and cross-modal alignment, generates a unified semantic representation, and outputs the unified semantic representation to the intent recognition unit. The intent recognition unit is used to receive the unified semantic representation output by the semantic parsing unit, generate the probability distribution of the request to answer questions, the escort interaction request or the itinerary planning request, determine the parsing result and output it to the corresponding processing unit. When the confidence level is lower than the threshold, the clarification request is sent to the input interface unit. The question-answering processing unit is used to perform knowledge graph retrieval and semantic similarity calculation when it receives a question-answering request from the intent recognition unit. It matches the user's question with the triples in the knowledge base, obtains the question-answering result, and outputs it to the feedback collection and update unit. The escort interaction unit is used to call the user's emotional features and semantic features when it receives the escort interaction request transmitted by the intention recognition unit, concatenate them and input them into the emotion adaptive network to generate a semantic representation that integrates emotional features, and drive the speech synthesis and facial expression control module to output the interaction content based on the semantic representation, and transmit the output interaction content to the feedback acquisition and update unit. The trip planning unit is used to, upon receiving a trip planning request from the intent recognition unit, combine the user's travel time, budget, interests and travel companion information, construct a constrained optimization model and perform path search to generate at least one trip planning scheme, and output the scheme to the feedback collection and update unit. The feedback collection and update unit is used to periodically receive users' text feedback, voice feedback, and operation behavior data throughout the service process, establish a mapping relationship between feedback and output results, dynamically adjust and iteratively update the outputs of the question-and-answer processing unit, the escort interaction unit, and the itinerary planning unit, and return the updated results to the user terminal.

9. A terminal, characterized in that, include: Memory, used to store tourism itinerary planning simulation programs based on multimodal data; A processor is configured to implement the steps of the multimodal data-based travel itinerary planning method as described in claim 1 when executing the multimodal data-based travel itinerary planning device.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions from the storage medium, the computer executes the travel itinerary planning method based on multimodal data as described in claim 1.

Images