An intelligent guide method and system based on intelligent token and semantic fusion

Abstract

An intelligent guide method and system based on intelligent tokens and semantic fusion, which includes acquiring multi-modal scenic spot consultation input and converting it into a semantic representation vector; retrieving a tourism knowledge base through a dual-mode retrieval method that fuses dense vector retrieval and sparse vector retrieval to obtain a candidate document set; generating an initial answer based on the candidate document set; performing a verification and correction cycle on the initial answer, including S410, splitting the initial answer into multiple clauses and retrieving the tourism knowledge base based on each clause to obtain corresponding supporting documents; S420, verifying the logical support relationship between each clause and the corresponding supporting documents; S430, generating a correction prompt for the initial answer based on the verification result, and regenerating the answer according to the correction prompt; S440, taking the regenerated answer as the current answer, and repeating S410-S430 until the logical support relationship of all clauses in the current answer meets the preset standard, and outputting the current answer.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent tour guide method and system based on smart tokens and semantic fusion. Background Technology

[0002] Traditional electronic tour guide systems rely on preset scripts and fixed content templates, playing pre-stored content through trigger mechanisms, and cannot provide dynamic explanations based on tourists' personalized needs. With the development of artificial intelligence technology, intelligent tour guide solutions based on retrieval-enhanced generative (RAG) models have emerged, overcoming the shortcomings of traditional systems by retrieving information from knowledge bases and dynamically generating text. However, conventional RAG models still have the following technical limitations in practical applications:

[0003] First, the semantic understanding is insufficient. Existing search engines mainly rely on keyword matching, which makes it difficult to capture the complex semantic relationships, cultural connotations, and historical background in tourism information, resulting in low search accuracy.

[0004] Secondly, the generated content lacks credibility. The model may produce "illusory" information that cannot be verified by the knowledge base, seriously affecting the accuracy of tour guide services.

[0005] In summary, existing technologies suffer from insufficient semantic understanding depth and low credibility of generated content, resulting in a poor user experience. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an intelligent tour guide method based on smart token and semantic fusion, comprising the following steps:

[0007] S100: Obtain multimodal scenic spot information input and convert it into a semantic representation vector through a smart token mechanism;

[0008] S200. Based on the semantic representation vector, the tourism knowledge base is retrieved by a dual-mode retrieval method that integrates dense vector retrieval and sparse vector retrieval to obtain a set of candidate documents related to the scenic spot consultation input; the tourism knowledge base includes a scenic spot knowledge base and a historical knowledge base.

[0009] S300. Based on the candidate document set, generate an initial answer corresponding to the scenic spot inquiry input;

[0010] S400. Perform a verification and correction loop on the initial answer, including:

[0011] S410. The initial answer is split into multiple clauses, and the tourism knowledge base is searched based on each clause to obtain the corresponding supporting documents;

[0012] S420. Verify the logical support relationship between each clause and the corresponding supporting document;

[0013] S430. Generate a correction suggestion for the initial answer based on the verification result, and regenerate the answer according to the correction suggestion;

[0014] S440. The regenerated answer is taken as the current answer, and S410-S430 are repeated until the logical support relationship of all clauses in the current answer reaches the preset standard, and then the current answer is output.

[0015] Optionally, S100 includes the following steps:

[0016] S110. Obtain multimodal scenic spot information input, extract at least two feature vectors respectively and use them together as the input sequence;

[0017] S120. Perform a linear transformation on the input sequence to generate a first query vector, a first key vector, and a first value vector;

[0018] S130. Adaptive pooling is performed on the first query vector to generate a smart token sequence; the number of smart tokens is dynamically determined based on the length of the input sequence.

[0019] S140. Perform multi-head splitting on the first query vector, the first key vector, the first value vector and the smart token sequence to obtain the multi-head first query vector, the multi-head first key vector, the multi-head first value vector and the multi-head smart token sequence respectively.

[0020] S150, Generate a first intermediate feature through the first attention interaction between the multi-head smart token sequence and the multi-head first key vector after feature transposition;

[0021] S160, A second intermediate feature is generated through the second attention interaction between the multi-head first query vector and the multi-head smart token sequence after feature transposition;

[0022] S170. Perform multi-head merging on the second intermediate feature and the first multi-head value vector respectively to obtain the first merged feature and the second merged feature respectively;

[0023] S180. After performing second feature reshaping and one-dimensional convolution on the second merged feature, perform second feature reshaping again to obtain local context features.

[0024] S190. The merged features are added and fused with the local context features, and then linear transformation and random deactivation are performed to obtain the semantic representation vector.

[0025] Optionally, S150 includes the following steps:

[0026] S151. Perform feature transpose processing on the multi-head first key vector, and calculate the first attention weight matrix between the multi-head smart token sequence and the feature transpose multi-head first key vector;

[0027] S152. The first value vector of the multi-head is weighted based on the first attention weight matrix to obtain the first intermediate feature.

[0028] Optionally, S160 includes the following steps:

[0029] S161. Perform feature transpose processing on the multi-head smart token sequence, and calculate the second attention weight matrix between the first multi-head query vector and the multi-head smart token sequence after feature transpose processing;

[0030] S162. The first intermediate feature is weighted based on the second attention weight matrix to obtain the second intermediate feature.

[0031] Optionally, S200 includes the following steps:

[0032] S210. Based on the semantic representation vector, generate dense retrieval vector and sparse retrieval vector respectively;

[0033] S220. Perform a fully connected transformation on the dense retrieval vector to extract high-level semantic features; perform dimensionality reduction on the sparse retrieval vector, and then perform a fully connected transformation to extract keyword features.

[0034] S230. Perform cross-modal attention interaction between the high-level semantic features and the keyword features to generate interaction features;

[0035] S240. Combine the interaction features with the keyword features to form a composite feature;

[0036] S250. Input the composite features into a dynamic gating network to generate fusion weights;

[0037] S260. Based on the fusion weight, the interaction features and keyword features are weighted and fused to obtain fused features;

[0038] S270. Perform a nonlinear transformation on the fused features and output the first document relevance score;

[0039] S280. Based on the first document relevance score, the documents in the tourism knowledge base are reordered to obtain a candidate document set.

[0040] Optionally, the dynamic gating network includes two fully connected layers and a Sigmoid activation function to output the fusion weights between 0 and 1.

[0041] Optionally, the nonlinear transformation includes sequential fully connected layer processing, activation layer processing, and fully connected layer processing.

[0042] Optionally, S410 includes the following steps:

[0043] S411. Divide the initial response into multiple independent clauses according to semantic boundaries;

[0044] S412. Convert each of the clauses into an independent second query vector;

[0045] S413. Based on the second query vector, retrieve the tourism knowledge base using a dual-mode retrieval method, and calculate the second document relevance score between each second query vector and the documents in the tourism knowledge base;

[0046] S414. Based on the relevance score, select the supporting documents corresponding to each clause from the tourism knowledge base.

[0047] Optionally, S430 includes the following steps:

[0048] S431. Integrate the aforementioned attraction information input, supporting documents, and validated clauses;

[0049] S432. Construct a prompt template, the prompt template including verified attraction information and content to be corrected;

[0050] S433. Based on the prompt template, generate a corrected prompt and input it into the generative model to regenerate the answer.

[0051] Corresponding to the aforementioned intelligent tour guide method based on smart token and semantic fusion, this invention provides an intelligent tour guide system based on smart token and semantic fusion, comprising:

[0052] The semantic representation vector generation module is used to obtain multimodal scenic spot information input and convert it into semantic representation vectors through a smart token mechanism;

[0053] The dual-mode retrieval module is used to retrieve the tourism knowledge base based on the semantic representation vector by using a dual-mode retrieval method that combines dense vector retrieval and sparse vector retrieval, to obtain a set of candidate documents related to the scenic spot consultation input; the tourism knowledge base includes a scenic spot knowledge base and a historical knowledge base;

[0054] The initial answer generation module is used to generate an initial answer corresponding to the tourist attraction inquiry input based on the candidate document set.

[0055] The initial answer optimization module is used to perform a verification and correction loop on the initial answer, including: S410, splitting the initial answer into multiple clauses, and searching the tourism knowledge base based on each clause to obtain corresponding supporting documents; S420, verifying the logical support relationship between each clause and the corresponding supporting document; S430, generating a correction prompt for the initial answer based on the verification result, and regenerating the answer according to the correction prompt; S440, using the regenerated answer as the current answer, and repeating S410-S430 until the logical support relationship of all clauses in the current answer reaches a preset standard, and then outputting the current answer.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) The intelligent token mechanism in step S100 converts the multimodal scenic spot consultation input into semantic representation vectors, effectively models sentence-level or paragraph-level semantics, integrates global sequence information, dynamically captures complex semantic associations, cultural connotations and historical backgrounds in tourism consultations, significantly improves the depth and accuracy of semantic understanding, and effectively reduces the computational complexity of the model when dealing with long contexts, meeting the needs of real-time interaction; Step S200, based on semantic representation vectors, retrieves the tourism knowledge base by integrating dense vector retrieval and sparse vector retrieval in a dual-mode retrieval method, overcoming the superficial limitations of traditional keyword matching and achieving accurate positioning of documents in the tourism knowledge base; Through the verification and correction loop mechanism, the initial answer is split into clauses and their logical support relationship with the supporting documents in the tourism knowledge base is verified one by one, accurately identifying illusory content, iteratively eliminating illusory information in the generated content that cannot be verified by the knowledge base, fundamentally ensuring the traceability and credibility of the answer content. The synergistic effect of the above-mentioned technical means in this invention directly solves the technical problems of insufficient semantic understanding depth and low credibility of generated content in the prior art, so that the output tour guide answers not only have deep semantic understanding capabilities, but also ensure that each clause is reliably supported by the tourism knowledge base, thereby greatly improving the user experience.

[0058] (2) By acquiring multimodal scenic spot consultation input and extracting at least two feature vectors as input sequences, comprehensive encoding of multimodal information is achieved; linear transformation is performed on the input sequence to generate the first query vector, the first key vector and the first value vector, and adaptive pooling is performed on the first query vector to generate a smart token sequence. The number of smart tokens is dynamically determined based on the length of the input sequence, realizing a dynamic information compression and distribution strategy, and enhancing the model's adaptability to consultations of different lengths; through the fusion of dual attention interaction mechanism and one-dimensional convolution, long-distance semantic dependence and local context features are effectively aggregated, improving the semantic representation vector's expressive power and generalization performance for complex tourism consultations.

[0059] (3) By calculating the first attention weight matrix between the multi-head smart token sequence and the multi-head first key vector after feature transposition, and by weighting the multi-head first value vector based on the first attention weight matrix, the first intermediate feature is obtained, enabling the smart token to adaptively focus on the knowledge dimension most relevant to the input intent of scenic spot consultation. This invention enhances the sensitivity of the smart token in capturing key cultural and historical elements in the tourism knowledge base, providing a preliminary and weighted information foundation for subsequent feature interaction.

[0060] (4) By calculating the second attention weight matrix between the first multi-head query vector and the multi-head smart token sequence after feature transposition, and by weighting the first intermediate feature based on the second attention weight matrix to obtain the second intermediate feature, a secondary fine-tuning of the information from input query to smart token aggregation is achieved. This invention enables the model to dynamically adjust the contribution of aggregated features according to the specific needs of the original scenic spot consultation input, thereby enhancing the adaptability and specificity of semantic representation.

[0061] (5) By independently processing and dynamically gating the fusion of dense and sparse retrieval vectors, the model adaptively balances the weights of high-level semantic similarity and keyword matching. This invention effectively overcomes the one-sidedness of a single retrieval mode, fully leverages the efficiency of dense vector retrieval and the accuracy of sparse encoder retrieval, and makes the retrieval results both semantically rich and detail-accurate, improving the document recall accuracy and ranking quality in complex tourism consultation scenarios, and significantly enhancing the accuracy and reliability of retrieval.

[0062] (6) By generating continuous fusion weights between 0 and 1 element by element, adaptive and smooth weighting of interactive features and sparse retrieval vectors after dimensionality reduction is achieved.

[0063] (7) Nonlinear transformation enhances the nonlinear expression capability of fused features to the first document relevance score through the cascade structure of double-layer fully connected and activation layers, enabling the model to capture more complex query-document matching patterns. This structural design improves the stability of the feature fusion process and the accuracy of score prediction, ensuring the reliability of the dual-mode retrieval results.

[0064] (8) By splitting the initial answer into independent clauses according to semantic boundaries and converting them into a second query vector, and independently searching the tourism knowledge base through a dual-mode retrieval method, fine-grained source verification of the generated content is achieved. This mechanism decomposes the traditional whole answer verification into atomic-level clause verification, which can accurately locate specific illusion fragments not supported by the tourism knowledge base and avoid information redundancy caused by regenerating the whole sentence. Based on the relevance score, the supporting documents corresponding to each clause are selected independently, so that the verification results have clear document basis, significantly improving the verification accuracy and the pertinence of subsequent corrections, and shortening the iteration cycle.

[0065] (9) By integrating the scenic spot consultation input, supporting documents and verified clauses, a prompt template is constructed. The prompt template includes verified scenic spot information and content to be corrected. Based on the prompt template, a correction prompt is generated and input into the generative model to regenerate the answer. When the model is regenerated, it will inherit the verified information first and only make targeted corrections to the unverified parts, thus providing structured constraints for the generative model. This improves the logical consistency, content completeness and credibility of the corrected answer and the scenic spot consultation input, ensures that the generated content is consistent with the support of relevant documents, and achieves efficient iterative optimization. Attached Figure Description

[0066] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0067] Figure 1 This is a simplified flowchart of an embodiment of the intelligent tour guide method based on smart token and semantic fusion of the present invention;

[0068] Figure 2 This is a schematic diagram of the data processing flow of the smart token mechanism in an embodiment of the smart guide method based on smart token and semantic fusion of the present invention.

[0069] Figure 3 This is a schematic diagram of the data processing flow of a dual-mode retrieval method according to an embodiment of the intelligent guide method based on smart token and semantic fusion of the present invention;

[0070] Figure 4 This is a schematic diagram of the data processing flow of an embodiment of the intelligent guide method based on smart token and semantic fusion of the present invention;

[0071] Figure 5 This is a schematic diagram of the cross-modal interactive data processing flow of an embodiment of the intelligent guide method based on smart token and semantic fusion of the present invention;

[0072] Figure 6 This is a framework diagram of an embodiment of the intelligent tour guide system based on smart token and semantic fusion of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] like Figure 1 As shown, the present invention provides an intelligent tour guide method based on smart token and semantic fusion, which includes the following steps:

[0075] S100. Obtain multimodal scenic spot information input and convert it into a semantic representation vector through a smart token mechanism; preferably, the multimodal scenic spot information input includes at least two of the following: voice input, image input, and text input;

[0076] S200. Based on semantic representation vectors, a dual-mode retrieval method that integrates dense vector retrieval and sparse vector retrieval is used to retrieve the tourism knowledge base and obtain a set of candidate documents related to the scenic spot information input; the tourism knowledge base includes a scenic spot knowledge base and a historical knowledge base.

[0077] S300: Based on the candidate document set, generate the initial answer corresponding to the scenic spot inquiry input;

[0078] S400. Perform a verification and correction loop on the initial answer, including:

[0079] S410. Break down the initial answer into multiple clauses, and search the tourism knowledge base based on each clause to obtain the corresponding supporting documents;

[0080] S420. Verify the logical support relationship between each clause and its corresponding supporting document;

[0081] S430. Generate correction suggestions for the initial answer based on the verification results, and regenerate the answer according to the correction suggestions;

[0082] S440. Take the regenerated answer as the current answer, and repeat S410-S430 until the logical support relationship of all clauses in the current answer meets the preset standard, and then output the current answer.

[0083] This invention uses a smart token mechanism in step S100 to convert multimodal scenic spot consultation inputs into semantic representation vectors, effectively modeling sentence-level or paragraph-level semantics, integrating global sequence information, and dynamically capturing complex semantic relationships, cultural connotations, and historical backgrounds in tourism consultations. This significantly improves the depth and accuracy of semantic understanding and effectively reduces the computational complexity of the model when dealing with long contexts, meeting the needs of real-time interaction. Step S200, based on semantic representation vectors, uses a dual-mode retrieval method that integrates dense vector retrieval and sparse vector retrieval to search the tourism knowledge base, overcoming the superficial limitations of traditional keyword matching and achieving accurate document location in the tourism knowledge base. Through a verification and correction loop mechanism, the initial answer is split into clauses and their logical support relationship with supporting documents in the tourism knowledge base is verified one by one. This accurately identifies illusory content and iteratively eliminates illusory information in the generated content that cannot be verified by the knowledge base, fundamentally ensuring the traceability and credibility of the answer content. The synergistic effect of the above-mentioned technical means in this invention directly solves the technical problems of insufficient semantic understanding depth and low credibility of generated content in the prior art, so that the output tour guide answers not only have deep semantic understanding capabilities, but also ensure that each clause is reliably supported by the tourism knowledge base, thereby greatly improving the user experience.

[0084] Preferably, the attraction knowledge base is used to store descriptive information about the attraction, such as opening hours and location information; the historical knowledge base is used to store in-depth information related to the attraction, such as background culture, historical events, and biographies of people.

[0085] The smart token mechanism described in this invention enhances sequence modeling capabilities by introducing smart tokens, thereby achieving more efficient feature extraction and information interaction. The core of this mechanism lies in the dynamic interaction of smart tokens, which effectively captures contextual information within the sequence. Through dynamic information compression and distribution strategies, the computational complexity of long sequence modeling is significantly reduced, while hierarchical feature integration and adaptive semantic focusing are achieved. This mechanism not only reduces redundant computation but also enhances the model's ability to capture key information and improves robustness against noise interference.

[0086] Specifically, the smart token mechanism first generates three sets of feature representations—query, key, and value—through a linear transformation. Then, it uses adaptive pooling to compress the input sequence into a fixed number of smart tokens, each representing a global feature of the input sequence. In this way, the model can effectively extract important information while reducing computational complexity. The smart token mechanism is a low-level, general-purpose attention computing architecture innovation integrated into the Transformer's encoder to replace traditional multi-head self-attention layers.

[0087] When calculating attention, the interaction between the smart token and the key generates a positional bias, thereby dynamically adjusting the attention weights. This process is achieved through matrix multiplication, ensuring that the smart token can provide weighted attention to different parts of the input sequence. Furthermore, the interaction between the query and the smart token further enhances the flow of information, enabling the model to flexibly capture both local and global context within the sequence.

[0088] Finally, after multi-head attention computation, the model merges the multi-head outputs and introduces additional information through local contextual convolution operations, enhancing the richness of feature representation. The final output is projected through a linear transformation to obtain a feature representation suitable for subsequent tasks. This smart token-based attention mechanism not only improves model performance but also effectively reduces computational overhead, making it excellent at handling long sequences. The data processing flowchart of the smart token mechanism is shown below. Figure 2 (This is for illustrative purposes only and does not constitute an undue limitation of the invention.)

[0089] In this embodiment, S100 includes the following steps:

[0090] S110. Obtain multimodal scenic spot information input, extract at least two feature vectors respectively, and use them together as the input sequence. The shape is [B,N,D], where B is the batch data size, N is the sequence length, and D is the feature vector length; preferably, image input, voice input, and text input are processed by a visual encoder, a speech encoder, and a text encoder, respectively.

[0091] S120. Perform a linear transformation on the input sequence to generate a first query vector q, a first key vector k, and a first value vector v; specifically including:

[0092] Input sequence The first feature is obtained through a fully connected layer. Its shape is [B,N,3D];

[0093] For the first feature Perform first and second feature reshaping processes to obtain the second feature. The shape is [3,B,N,D];

[0094] For the second feature By segmenting, we can generate the first query vector q, the first key vector k, and the first value vector v, all with shapes [B, N, D]; that is:

[0095] ;in, Indicates a fully connected layer;

[0096] ;in, This indicates the first feature reshaping process. The second feature reshaping process differs in that: It will change the shape of the input tensor, but not the order of its internal data. This will change the dimensional order of the input tensor, but not the shape of its internal data;

[0097] ;in, Indicates feature segmentation;

[0098] S130. Perform adaptive pooling on the first query vector q to generate a smart token sequence a; the number of smart tokens is based on the input sequence. The length is dynamically determined; including:

[0099] For the first query vector The second feature reshaping process, one-dimensional pooling process, and second feature reshaping process are performed sequentially to obtain the smart token sequence a, with shape [B,M,D]; where, for the first query vector After the second feature reshaping process, its shape is [B,D,N]. After further one-dimensional pooling, its shape is [B,D,M]. After another second feature reshaping process, the smart token sequence a is obtained, with a shape of [B,M,D]. The details are as follows:

[0100] ;in, This represents one-dimensional pooling, which aims to adaptively generate smart tokens through a one-dimensional pooling layer, thereby reducing the sequence length. The size of M is set to... , This indicates that the result of the operation is rounded down;

[0101] S140. Perform multi-head splitting on the first query vector q, the first key vector k, the first value vector v, and the smart token sequence a to obtain the multi-head first query vectors respectively. Multi-head first key vector Multi-head first value vector and multi-headed smart token sequences First query vector First key vector First value vector The shapes are all [B,h,N,d], The shape is [B,h,M,d]; specifically as follows:

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] ； Where h represents the number of heads in the multi-head self-attention mechanism, and d is the feature dimension of a single attention head;

[0107] S150, via multi-head smart token sequence The first key vector of the multi-head sequence after feature transpose First attention interaction generates first intermediate features The steps include:

[0108] S151, First key vector of multi-head Perform feature transpose processing and compute the multi-head smart token sequence. The first key vector of the multi-head after feature transpose processing The first attention weight matrix between them;

[0109] S152, Based on the first attention weight matrix, the first value vector of the multi-head system. After weighting, the first intermediate feature is obtained. The details of S151-S152 are as follows:

[0110] ;in, Indicates feature transpose. The shape is [B,H,d,N]. This represents matrix multiplication, resulting in a matrix with the shape [B, h, M, N]. This indicates the first position encoding, with a shape of [B, hM, N]. `softmax` indicates softmax processing. The shape is [B,h,M,N];

[0111] ;in, The shape is [B,h,M,N];

[0112] Where s is a scaling factor used to alleviate the vanishing softmax gradient problem caused by excessively large d;

[0113] S160, via the first query vector of multiple heads Multi-head smart token sequence after feature transpose The second attention interaction generates the second intermediate feature. The steps include:

[0114] S161, Multi-headed smart token sequence Perform feature transpose processing and calculate the first query vector for the multi-head query. Multi-head smart token sequence after feature transposition The second attention weight matrix between;

[0115] S162, Applying the first intermediate feature based on the second attention weight matrix After weighting, the second intermediate feature is obtained. S161-S162 are detailed below:

[0116] ;in, The shape is [B,h,d,M]. This represents the second positional encoding, with a shape of [B,h,M,N]. After matrix multiplication, the shape becomes [B,h,N,M]. The shape is [B,h,N,M];

[0117] ;in, The shape is [B,h,N,d];

[0118] ;

[0119] S170, Regarding the second intermediate feature With the first value vector of the multi-head Perform multi-head merging separately to obtain the first merging feature. With the second merged feature ;in, The shape is [B,N,D]. The shape is [B, N, D];

[0120] S180, Regarding the second merging feature After performing the second feature reshaping process and one-dimensional convolution, the second feature reshaping process is performed again to obtain local context features. The role of one-dimensional convolutional layers is to capture feature relationships within local regions and model the local context to enhance feature representation; specifically as follows:

[0121] ;in, It is a one-dimensional convolutional layer; The shape is [B,D,N]. The shape is [B,D,N]. The shape is [B, N, D];

[0122] S190, Merging Features With local context features Adding and fusing together yields the third feature. ; and then the third feature A linear transformation and random deactivation are performed to obtain the semantic representation vector. The details are as follows:

[0123] ;in, The shape is [B, N, D];

[0124] ;in, This indicates random deactivation. The shape is [B,N,D].

[0125] It should be noted that a semantic representation vector refers to a fixed-length numerical vector, such as a floating-point array with dimensions of 512 or 768. For image input, a visual encoder is used to process the image, which converts the raw pixel data into a high-dimensional feature vector containing the image's visual semantic information. For speech input, a speech encoder is used to process the audio signal, which first converts the speech waveform into acoustic features and then extracts a semantic representation vector. For text input, a text encoder is used to process the text, converting word sequences into context-sensitive semantic vectors.

[0126] This invention achieves comprehensive encoding of multimodal information by acquiring multimodal tourist attraction information input and extracting at least two feature vectors as the input sequence. A linear transformation is applied to the input sequence to generate a first query vector, a first key vector, and a first value vector. Adaptive pooling is then applied to the first query vector to generate a smart token sequence, with the number of smart tokens dynamically determined based on the length of the input sequence. This implements a dynamic information compression and distribution strategy, enhancing the model's adaptability to inquiries of varying lengths. Furthermore, by fusing a dual attention interaction mechanism with one-dimensional convolution, long-distance semantic dependencies and local contextual features are effectively aggregated, improving the semantic representation vector's expressive power and generalization performance for complex tourist information.

[0127] This invention calculates a first attention weight matrix between a multi-head smart token sequence and a multi-head first key vector, and then weights the multi-head first value vector based on this first attention weight matrix to obtain a first intermediate feature. This allows the smart token to adaptively focus on the knowledge dimension most relevant to the input intent for scenic spot information. This invention enhances the sensitivity of the smart token in capturing key cultural and historical elements in the tourism knowledge base, providing a pre-screened and weighted information foundation for subsequent feature interactions.

[0128] This invention achieves secondary fine-tuning of the process from input query to smart token aggregation information by calculating a second attention weight matrix between the multi-head first query vector and the multi-head smart token sequence, and then weighting the first intermediate features based on the second attention weight matrix to obtain the second intermediate features. This invention enables the model to dynamically adjust the contribution of aggregated features according to the specific needs of the original scenic spot information input, enhancing the adaptability and specificity of semantic representation.

[0129] The following is a brief illustration of the semantic representation vector generation steps through a specific example. For instance, the input for a multimodal tourist attraction inquiry is: text input "What place is this?" followed by uploading a photo of the Yonghe Temple; the data processing procedure is as follows:

[0130] Image features and text features are extracted. The image is processed by a visual encoder to generate an image feature vector, and the text is processed by a text encoder to generate a text feature vector. The two together constitute the input sequence.

[0131] A linear transformation is performed on the input sequence to generate three sets of feature representations: the first query vector, the first key vector, and the first value vector.

[0132] A sequence of smart tokens representing global semantics is generated by adaptively pooling and compressing the sequence.

[0133] The first query vector, the first key vector, the first value vector, and the smart token sequence are split into multiple heads to obtain the multi-head first query vector, the multi-head first key vector, the multi-head first value vector, and the multi-head smart token sequence, respectively.

[0134] In attention calculation, the interaction between the multi-head smart token and the multi-head first key vector generates a positional bias, dynamically adjusting the attention weights of each part to achieve focus on key areas (such as the main body of a building); at the same time, the interaction between the multi-head first query vector and the multi-head smart token sequence enhances cross-modal information flow.

[0135] After multi-head attention computation, the outputs of each head are merged, and fine-grained contextual information is supplemented through local convolution operations. Finally, a semantic representation vector is output after linear transformation.

[0136] This invention, based on a semantic fusion dual-mode retrieval system, efficiently addresses the limitations of traditional retrieval systems in semantic understanding by combining dense vector retrieval and sparse encoder retrieval. It cleverly utilizes the semantic condensation capabilities of dense vector retrieval to map text content into low-dimensional, dense semantic vectors, accurately capturing the overall semantics of the text. Simultaneously, leveraging the high-dimensional sparsity of sparse encoder retrieval, it decomposes the text into high-dimensional sparse vectors, thereby meticulously characterizing local features and key semantics. In actual retrieval, this module comprehensively analyzes the results of both retrieval methods, calculates the fusion weight through a dynamic gating mechanism, directly outputs a relevance score, and finally uses this score to re-rank documents. This fusion mechanism fully leverages the efficiency of dense vector retrieval and the accuracy of sparse encoder retrieval, resulting in retrieval results that are both semantically rich and detail-oriented, significantly improving retrieval accuracy and reliability, and providing higher-quality data support for upper-layer applications. The dual-mode retrieval method integrating dense and sparse vector retrieval (data processing flow as follows) Figure 3 As shown, Figure 3 (This is for illustrative purposes only and does not constitute an undue limitation of the invention.) It effectively combines the semantic understanding capabilities of dense vector retrieval with the precise matching advantages of sparse encoder retrieval, ultimately achieving accurate understanding of query semantics and efficient retrieval of relevant documents. This improves the model's retrieval performance and accuracy.

[0137] In this embodiment, S200 includes the following steps:

[0138] S210. Based on the semantic representation vector, generate dense retrieval vectors respectively. With sparse search vectors The shapes are [B,E1] and [B,E2], where B is the batch size, E1 is the length of the dense retrieval vector, and E2 is the length of the sparse retrieval vector. Preferably, a pre-trained syntactic transformer model or transformer model with a smart token mechanism is used to convert the semantic representation vector into a low-dimensional dense vector representation to capture the deep semantic information of the text, making the vectors of semantically similar texts closer in space. A sparse encoder is used to map the semantic representation vector to a high-dimensional sparse vector space to capture the keyword and phrase information of the document.

[0139] S220, Dense Search Vectors Perform a fully connected transformation to reduce the length of the dense retrieval vector to S, and extract high-level semantic features. The shape is [B,S]; for sparse search vectors Dimensionality reduction is performed, followed by fully connected transformation to extract keyword features. The shape is also [B, S]; as follows:

[0140] ;

[0141] ;in, This represents the dimensionality reduction layer, preferably using dimensionality reduction methods such as PCA or TruncatedSVD, and the specific method should be selected according to actual needs.

[0142] S230, High-level semantic features Keyword Features Perform cross-modal attention interactions and generate interaction features. The details are as follows:

[0143] ;in, For the cross-modal interaction module, its structure diagram can be found here. Figure 4 (This is merely an illustrative example and does not constitute an undue limitation on the present invention.)

[0144] S240, Interactive Features Keyword Features splicing to form composite features The shape is [B, 2S]; the details are as follows:

[0145] ;in, Indicates feature splicing;

[0146] S250, Composite Features Input dynamic gating network generates fusion weights The shape is [B,1]; the details are as follows:

[0147] ;in, For the dynamic gating module, its data processing flow can be found here. Figure 5 ;

[0148] S260, Based on Fusion Weights Interaction features Keyword Features Weighted fusion is performed to obtain fusion features. Its shape is [B, 256]; details are as follows:

[0149] ;

[0150] S270, Regarding fusion characteristics Perform a nonlinear transformation and output the first document relevance score. The shape is [B,1]; the details are as follows:

[0151] ;in, For activation layer;

[0152] S280, Based on the first document relevance score The documents in the tourism knowledge base are reordered to obtain a set of candidate documents.

[0153] In this embodiment, the present invention preferentially adopts mature existing technologies to implement sparse encoders to ensure the feasibility and practicality of the solution;

[0154] Here's an example: BM25 (Best Matching 25): This is the most classic and widely used sparse retrieval algorithm in the field of information retrieval. It calculates a weight for each term based on term frequency, inverse document frequency, and document length normalization, generating a sparse vector representation of the document. BM25 requires no training and is simple to implement.

[0155] SPLADE is an advanced sparse encoder based on deep learning. Utilizing pre-trained language models such as BERT, it not only identifies the original query word but also infers related extended words and assigns weights to them, generating a richer and more semantically meaningful sparse vector. SPLADE combines the semantic understanding capabilities of deep learning with the precise matching advantages of sparse representation.

[0156] This invention enables the model to adaptively balance the weights of high-level semantic similarity and keyword matching by independently processing and dynamically gating dense and sparse retrieval vectors. This invention effectively overcomes the one-sidedness of a single retrieval mode, giving full play to the efficiency of dense vector retrieval and the accuracy of sparse encoder retrieval, so that the retrieval results are both semantically rich and detail-accurate, improving the document recall accuracy and ranking quality in complex tourism consultation scenarios, and significantly enhancing the accuracy and reliability of retrieval.

[0157] In this embodiment, the dynamic gating network includes two fully connected layers and a Sigmoid activation function, which is used to output fusion weights between 0 and 1.

[0158] This invention achieves adaptive and smooth weighting of interactive features and sparse retrieval vectors after dimensionality reduction by generating continuous fusion weights between 0 and 1 element by element.

[0159] In this embodiment, the nonlinear transformation includes sequential processing of the fully connected layer, activation layer, and fully connected layer.

[0160] The nonlinear transformation of this invention enhances the nonlinear expressive power of the fused features to the first document relevance score through the cascaded structure of two fully connected layers and activation layers, enabling the model to capture more complex query-document matching patterns. This structural design improves the stability of the feature fusion process and the accuracy of score prediction, ensuring the reliability of the dual-mode retrieval results.

[0161] In this embodiment, S410 includes the following steps:

[0162] S411. Divide the initial response into multiple independent clauses according to semantic boundaries;

[0163] S412. Convert each clause into an independent second query vector;

[0164] S413. Based on the second query vector, retrieve the tourism knowledge base using a dual-mode retrieval method, and calculate the second document relevance score between each second query vector and the documents in the tourism knowledge base;

[0165] S414. Based on the relevance score, select the supporting documents corresponding to each clause from the tourism knowledge base. The supporting documents may include tourist attraction introductions, historical background, user reviews, etc.

[0166] This invention achieves fine-grained source verification of generated content by splitting the initial answer into independent clauses according to semantic boundaries and converting them into a second query vector. A dual-mode retrieval method is then used to independently search the tourism knowledge base. This mechanism decomposes traditional whole-answer verification into atomic-level clause verification, enabling precise location of specific illusionary fragments not supported by the tourism knowledge base and avoiding information redundancy caused by regenerating entire sentences. Supporting documents corresponding to each clause are independently selected based on relevance scores, providing clear document evidence for the verification results, significantly improving verification accuracy and the targeted nature of subsequent corrections, and shortening the iteration cycle.

[0167] In this embodiment, S420 can verify the logical support relationship between each clause and the corresponding supporting document through a natural language reasoning model, thereby identifying potential illusions or inaccurate content.

[0168] Examples of preferred models that can be used for natural language inference:

[0169] RoBERTa-large-mnli: Based on the RoBERTa architecture, it is finely tuned on large-scale, multi-type natural language inference datasets and has powerful textual entailment judgment capabilities.

[0170] DeBERTa-v3-large-mnli: An improved version of the DeBERTa model, which excels in understanding complex semantic relationships.

[0171] BART-large-mnli: This tool utilizes the BART model for sequence-to-sequence natural language reasoning tasks and is suitable for generative verification scenarios.

[0172] Specific indicators for judging support:

[0173] The core task of natural language reasoning models is to determine the logical relationship between "premises" and "assumptions". In this embodiment:

[0174] Prerequisite: Supports document content.

[0175] Assume that this is the output of the RAG model (or its clauses).

[0176] Natural language inference models output three types of probability scores:

[0177] Implies: The document content supports this output, and the logic holds true.

[0178] Contradiction: The document content directly conflicts with the output.

[0179] Neutral: The document content neither supports nor opposes this output.

[0180] In this embodiment, the core indicator for the natural language reasoning model to determine "support" is that the probability score of the "implication" category is greater than or equal to a preset implication probability threshold. If it is less than the preset implication probability threshold, it is considered that the output lacks sufficient evidence and there may be a risk of hallucination.

[0181] In this embodiment, the preset standards include:

[0182] The natural language inference model is used to calculate the implication probability between each clause and the supporting document; when the implication probability of all clauses is greater than or equal to the preset implication probability threshold, it is determined that the preset standard has been met; preferably, the preset implication probability threshold is 0.8.

[0183] Alternatively, when the number of iterations equals a preset threshold, it is determined that the preset standard has been met, the loop is forcibly terminated, and the current answer is output; preferably, the preset threshold is 5 times.

[0184] In this embodiment, S430 includes the following steps:

[0185] S431, Integrate scenic spot information input, supporting documents, and validated clauses;

[0186] S432. Construct a prompt template, which includes verified attraction information and content to be corrected;

[0187] S433. Based on the prompt template, generate corrected prompts and input them into the generative model to regenerate the answer.

[0188] This invention integrates scenic spot information input, supporting documents, and validated clauses to construct a prompt template. The prompt template includes validated scenic spot information and content to be corrected. Based on the prompt template, a correction prompt is generated and input into a generative model to regenerate the answer. During regeneration, the model prioritizes inheriting validated information and only makes targeted corrections to unvalidated parts, providing structured constraints for the generative model. This improves the logical consistency, content completeness, and credibility of the corrected answer with the scenic spot information input, ensures that the generated content is consistent with the support of relevant documents, and achieves efficient iterative optimization.

[0189] like Figure 6 As shown, the present invention also provides an intelligent tour guide system based on smart tokens and semantic fusion, which includes:

[0190] The semantic representation vector generation module 10 is used to obtain multimodal scenic spot information input and convert it into a semantic representation vector through a smart token mechanism;

[0191] The dual-mode retrieval module 20 is used to retrieve the tourism knowledge base based on semantic representation vectors by combining dense vector retrieval and sparse vector retrieval in a dual-mode retrieval method, and obtain a set of candidate documents related to the scenic spot information input; the tourism knowledge base includes a scenic spot knowledge base and a historical knowledge base;

[0192] The initial answer generation module 30 is used to generate the initial answer corresponding to the scenic spot inquiry input based on the candidate document set;

[0193] The initial answer optimization module 40 is used to perform a verification and correction loop on the initial answer, including: S410, splitting the initial answer into multiple clauses and searching the tourism knowledge base based on each clause to obtain the corresponding supporting documents; S420, verifying the logical support relationship between each clause and the corresponding supporting document; S430, generating correction prompts for the initial answer based on the verification results, and regenerating the answer according to the correction prompts; S440, using the regenerated answer as the current answer, and repeating S410-S430 until the logical support relationship of all clauses in the current answer meets the preset standard, and then outputting the current answer.

[0194] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0195] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0196] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A smart guide method based on intelligent token and semantic fusion, characterized in that, The method comprises the following steps: S100, acquiring a multi-modal scenic spot consultation input and converting it into a semantic representation vector through an intelligent token mechanism; S200, based on the semantic representation vector, retrieving a tourism knowledge base through a dual-mode retrieval method of fusing dense vector retrieval and sparse vector retrieval to obtain a candidate document set related to the scenic spot consultation input; the tourism knowledge base comprises a scenic spot knowledge base and a historical knowledge base; S300, based on the candidate document set, generating an initial answer corresponding to the scenic spot consultation input; S400, performing a verification and correction cycle on the initial answer, comprising: S410, splitting the initial answer into multiple clauses and retrieving the tourism knowledge base based on each clause to obtain corresponding support documents; S420, verifying the logical support relationship between each clause and the corresponding support document; S430, generating a correction prompt for the initial answer based on the verification result, and regenerating an answer according to the correction prompt; S440, taking the regenerated answer as a current answer, and repeating S410-S430 until the logical support relationship of all clauses in the current answer meets a preset standard, and outputting the current answer; S100 comprises the following steps: S110, acquiring a multi-modal scenic spot consultation input, and extracting at least two feature vectors as an input sequence; S120, performing linear transformation on the input sequence to generate a first query vector, a first key vector, and a first value vector; S130, performing adaptive pooling processing on the first query vector to generate an intelligent token sequence; the number of intelligent tokens is dynamically determined based on the length of the input sequence; S140, performing multi-head splitting processing on the first query vector, the first key vector, the first value vector, and the intelligent token sequence respectively to obtain multi-head first query vectors, multi-head first key vectors, multi-head first value vectors, and multi-head intelligent token sequences respectively; S150, generating a first intermediate feature through first attention interaction between the multi-head intelligent token sequence and the multi-head first key vector after feature transposition; S160, generating a second intermediate feature through second attention interaction between the multi-head first query vector and the multi-head intelligent token sequence after feature transposition processing; S170, performing multi-head merging on the second intermediate feature and the multi-head first value vector respectively to obtain first merged features and second merged features respectively; S180, after performing second feature reshaping processing and one-dimensional convolution processing on the second merged feature, performing second feature reshaping processing again to obtain a local context feature; S190, adding and fusing the first merged feature and the local context feature, and then performing linear transformation and random inactivation processing to obtain a semantic representation vector. 2.The smart guide method based on smart token and semantic fusion according to claim 1, characterized in that, S150 comprises the following steps: S151, performing feature transposition processing on the multi-head first key vector, and calculating a first attention weight matrix between the multi-head intelligent token sequence and the multi-head first key vector after feature transposition processing; S152, based on the first attention weight matrix, performing weighted processing on the multi-head first value vector to obtain a first intermediate feature. 3.The smart guide method based on smart token and semantic fusion according to claim 2, characterized in that, S160 comprises the following steps: S161, performing feature transposition processing on the multi-head intelligent token sequence, and calculating a second attention weight matrix between the multi-head first query vector and the multi-head intelligent token sequence after the feature transposition processing; S162, performing weighting processing on the first intermediate feature based on the second attention weight matrix to obtain a second intermediate feature. 4.The smart guide method based on smart token and semantic fusion according to claim 1, characterized in that, S200 includes the following steps: S210, generating a dense retrieval vector and a sparse retrieval vector based on the semantic representation vector, respectively; S220, performing full connection transformation on the dense retrieval vector to extract high-level semantic features, and performing dimension reduction processing on the sparse retrieval vector and then performing full connection transformation to extract keyword features; S230, performing cross-modal attention interaction on the high-level semantic features and the keyword features to generate interaction features; S240, concatenating the interaction features and the keyword features to form composite features; S250, inputting the composite features into a dynamic gating network to generate fusion weights; S260, performing weighted fusion on the interaction features and the keyword features based on the fusion weights to obtain fusion features; S270, performing nonlinear transformation on the fusion features to output a first document relevance score; S280, reordering the documents in the tourism knowledge base based on the first document relevance score to obtain a candidate document set. 5.The smart guide method based on smart token and semantic fusion according to claim 4, characterized in that, The dynamic gating network includes two fully connected layers and a Sigmoid activation function, which is used to output the fusion weight between 0 and 1. 6.The smart guide method based on smart token and semantic fusion according to claim 4, characterized in that, The nonlinear transformation includes sequentially performing full connection layer processing, activation layer processing, and full connection layer processing. 7.The smart guide tour method based on smart token and semantic fusion according to claim 1, characterized in that, S410 includes the following steps: S411, splitting the initial answer into multiple independent clauses according to semantic boundaries; S412, converting each of the clauses into an independent second query vector; S413, retrieving the tourism knowledge base through a dual-mode retrieval method based on the second query vector, and calculating a second document relevance score of each of the second query vectors and the documents in the tourism knowledge base; S414, selecting support documents corresponding to each of the clauses from the tourism knowledge base based on the relevance scores. 8.The smart guide tour method based on smart token and semantic fusion according to claim 1, characterized in that, S430 includes the following steps: S431, integrating the scenic spot consultation input, support documents, and verified clauses; S432, constructing a prompt template, the prompt template including verified scenic spot information and content to be corrected; S433, generating a correction prompt based on the prompt template and inputting it into a generative model to regenerate the answer.

9. An intelligent tour guide system based on intelligent token and semantic fusion, characterized in that, It includes: The semantic representation vector generation module is configured to obtain a multi-modal scenic spot consultation input and convert it into a semantic representation vector through an intelligent token mechanism, including the following steps: S110, obtaining a multi-modal scenic spot consultation input, and extracting at least two feature vectors respectively and taking them as an input sequence; S120, performing linear transformation on the input sequence to generate a first query vector, a first key vector, and a first value vector; S130, performing adaptive pooling processing on the first query vector to generate an intelligent token sequence; the number of intelligent tokens is dynamically determined based on the length of the input sequence; S140, performing multi-head splitting processing on the first query vector, the first key vector, the first value vector, and the intelligent token sequence respectively to obtain multi-head first query vectors, multi-head first key vectors, multi-head first value vectors, and multi-head intelligent token sequences respectively; S150, generating a first intermediate feature through first attention interaction between the multi-head intelligent token sequence and the multi-head first key vector; S160, performing feature transposition processing on the multi-head intelligent token sequence, and performing weighting processing on the first intermediate feature through second attention interaction between the multi-head first query vector and the multi-head intelligent token sequence after the feature transposition processing, to generate a second intermediate feature; S170, performing multi-head merging on the second intermediate feature and the multi-head first value vector respectively to obtain first merged features and second merged features respectively; S180, performing second feature reshaping processing and one-dimensional convolution processing on the second merged features, and then performing second feature reshaping processing again to obtain a local context feature; S190, adding and fusing the first merged features and the local context feature, and then performing linear transformation and random inactivation processing to obtain a semantic representation vector; The dual-mode retrieval module is configured to retrieve a tourism knowledge base based on the semantic representation vector through a dual-mode retrieval method of fusing dense vector retrieval and sparse vector retrieval to obtain a candidate document set related to the scenic spot consultation input; the tourism knowledge base includes a scenic spot knowledge base and a historical knowledge base; The initial answer generation module is configured to generate an initial answer corresponding to the scenic spot consultation input based on the candidate document set; The initial answer optimization module is configured to perform a verification and correction cycle on the initial answer, including: S410, splitting the initial answer into multiple clauses, and retrieving the tourism knowledge base based on each clause to obtain corresponding support documents; S420, verifying the logical support relationship between each clause and the corresponding support document; S430, generating a correction prompt for the initial answer based on the verification result, and regenerating an answer according to the correction prompt; S440, taking the regenerated answer as a current answer, and repeating S410-S430 until the logical support relationship of all clauses in the current answer meets a preset standard, and outputting the current answer.

Images