A multi-modal intelligent tour guide method and system

Abstract

The application discloses a kind of multi-modal intelligent guide method and system, belong to intelligent guide field, the method includes: in mobile terminal side, data is collected;In the mobile terminal side, visual fingerprint for retrieval is generated based on real-time image data, and the visual fingerprint and the real-time positioning data are sent to cloud server;Cloud server is based on real-time positioning data, obtains the pre-stored high-quality visual feature vector corresponding to the target scene, as the first feature vector representing image content;According to real-time positioning data and the user preference data, generate the second feature vector representing user scenario;Through the feature fusion network based on attention mechanism, the first feature vector and the second feature vector are fused to obtain the fusion feature vector;Based on the fusion feature vector, generate guide content.The application can bring lively and technological tour experience for user, and enrich the form of user interaction experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent tour guide, and in particular to a multi-modal intelligent tour guide method and system. BACKGROUND

[0002] With the popularity of mobile internet technology and smart terminals, electronic tour guide systems have become an important auxiliary tool for tourists when visiting scenic spots. However, most of the current electronic tour guide systems provide services based on the geographic location information of the user. Through the mobile application (App), the GPS signal of the user's smart terminal is obtained to determine the user's approximate location, and the system staff pre-divides electronic fences for each scenic spot or point of interest in the scenic area. When the system determines that the user has entered the electronic fence area of a scenic spot, it will automatically trigger and play the pre-recorded audio commentary related to the scenic spot, or display the preset text and image introduction information on the device screen.

[0003] However, this GPS positioning-based electronic tour guide solution has low scene perception accuracy and cannot accurately identify user intent. This solution relies on the accuracy of GPS signals, and when encountering areas such as ancient architectural complexes, indoor exhibition halls, valleys, or high-rise buildings, GPS signals are easily affected by shielding or multipath effects, leading to inaccurate positioning, delays, or drifts, resulting in delayed tour information push or even a terrible experience where the information does not match the user's location. Moreover, this solution can only determine that the user is in a certain macro area, but cannot accurately identify the specific scenic targets in that area that the user is currently observing or truly interested in. For example, when the user is in front of a temple hall, the system cannot distinguish whether the user is appreciating the calligraphy on the hall plaque, studying the god carving on the eaves, or observing the statue at the entrance. The coarse-grained perception of the prior art at the regional level often causes the provided tour information to be disconnected from the user's true focus of attention, lacking in relevance. Furthermore, the content provided in existing solutions is generally pre-set and static, and the same information is pushed to all users entering the area. It is unable to dynamically adjust the depth, angle, and content of the explanation based on the user's personal knowledge background, interest preferences, current specific environment (such as weather, crowd density), or real-time behavior (such as stay duration, observation angle). This one-way, passive, and indoctrination-style service model is increasingly difficult to meet the higher requirements of modern tourists for personalized, immersive, and interactive tour experiences.

[0004] Modern intelligent terminals integrate cameras, GPS, gyroscopes, accelerometers and other sensors, but the existing technical solutions fail to effectively integrate these multi-modal data to form a comprehensive and three-dimensional understanding of the user and the environment in which he is located. In particular, visual information that can most directly reflect the user's current focus of attention (i.e. the picture the user "sees" through the camera) is completely ignored. Therefore, how to overcome the problems of low accuracy, lack of intent recognition, and rigid content caused by simply relying on GPS positioning in the prior art, and to realize a method that can accurately perceive the user's focus of attention and provide personalized and contextualized tour guide services, has become a technical problem that needs to be solved in the field. SUMMARY

[0005] One of the purposes of the present application is to provide a multi-modal intelligent tour guide method to solve the problems of low accuracy of real attention target recognition of tourists by the scenic area tour guide terminal and poor personalization and context relevance of the provided tour guide information in the prior art.

[0006] The present application is realized by the following technical solution, a multi-modal intelligent tour guide method, comprising the following steps: on the mobile terminal side, collecting real-time image data, real-time positioning data and pre-set user preference data; on the mobile terminal side, generating a visual fingerprint for retrieval based on the real-time image data, and sending the visual fingerprint and the real-time positioning data to a cloud server; the cloud server filters out a candidate scenery set in a pre-set visual feature database based on the real-time positioning data, and determines a target scenery in the candidate scenery set by matching the visual fingerprint, and obtains a pre-stored high-quality visual feature vector corresponding to the target scenery as a first feature vector representing image content; generates a second feature vector representing user context according to the real-time positioning data and the user preference data; fuses the first feature vector and the second feature vector through a feature fusion network based on an attention mechanism to obtain a fused feature vector; generates tour guide content based on the fused feature vector.

[0007] Further, filtering out a candidate scenery set in a pre-set visual feature database can be represented by the following formula:

[0008] The candidate set filtering here uses to represent the Euclidean distance weight function, which is used to calculate the geographical distance between the user's real-time location and the location of each scenery in the database Only when the distance is less than a pre-set search radius (e.g. 20 meters), the features of the scenery will be included in the candidate set This reduces the global visual search that originally required thousands of targets to only a few or dozens of candidate targets, thus significantly improving computational efficiency.

[0009] Furthermore, the step of the cloud server determining the first feature vector also includes: when the highest similarity of the visual fingerprint matched in the candidate scene set is lower than a preset matching threshold, a degradation processing mechanism is triggered; the mobile terminal side processes the real-time image data through a local model to generate the first feature vector.

[0010] Furthermore, before generating the visual fingerprint on the mobile terminal side, the process includes: evaluating the ambient brightness of the real-time image data; and performing low-light image enhancement processing on the real-time image data when the ambient brightness is lower than a preset brightness threshold.

[0011] Furthermore, low-light image enhancement processing can be expressed by the following formula:

[0012] ,

[0013] in, It is an evaluation input image A function of average brightness; It is a preset brightness threshold used to determine whether low light conditions have been entered; It is a low-light image enhancement algorithm, such as a deep learning-based enhancement network, with the following parameters: ; This is the image that is ultimately used for subsequent analysis after passing through this conditional preprocessing module.

[0014] Furthermore, on the mobile terminal side, the collected data also includes: real-time pedestrian density data; and the second feature vector is generated through the following steps: the real-time location data, the user preference data, and the real-time pedestrian density data are respectively vectorized and encoded, and the encoded multiple vectors are concatenated to generate the second feature vector.

[0015] Furthermore, when the real-time crowd density data is higher than the preset congestion threshold, the step of generating tour guide content also includes: generating passenger flow guidance prompts in the tour guide content.

[0016] Furthermore, the second eigenvector can be constructed using the following formula:

[0017] ,in, latitude and longitude coordinates Geospatial coding functions that map to vectors; It is a set of user preferences A function to convert to a multi-hot encoded vector; to regard the scalar human flow density data as a one-dimensional vector; representing a vector concatenation operation; the finally generated i.e., a comprehensive scenario feature vector containing user location, preference and environmental information.

[0018] Further, the fusion feature vector can be obtained by the following steps: taking the second feature vector or a linear transformation thereof as a query; taking the first feature vector or a linear transformation thereof as a key and a value; obtaining an attention weight by calculating the similarity between the query and the key, and performing weighted summation on the value using the attention weight, thereby obtaining the fusion feature vector.

[0019] Further, the fusion feature vector can be calculated by the following formula:

[0020] wherein, is a learnable weight matrix for projecting the input vector into the query, key and value space; the second feature vector the query vector generated by the second feature vector dominates the entire matching process and is the key to achieving personalization; the first feature vector the key vector generated by the first feature vector and the value vector provides the content to be filtered; the function converts the matching score into an attention weight in the form of a probability distribution; the finally generated fusion feature vector is a highly scenario-based and personalized information representation that has been dynamically filtered and reorganized according to the current scenario.

[0021] Further, the guide content is realized by a pre-trained sequence-to-sequence model based on the Transformer architecture, wherein the encoder of the sequence-to-sequence model based on the Transformer architecture receives the fusion feature vector, and the decoder generates the text sequence of the guide content autoregressively.

[0022] Further, the multi-modal intelligent guide method further comprises: collecting feedback behavior data of the user on the generated guide content; based on the feedback behavior data, iteratively optimizing the network parameters in the feature fusion network.

[0023] Further, the step of iteratively optimizing the network parameters comprises: using a preset reward function to quantify the quality score of the guide content according to the feedback behavior data; based on the quality score, constructing a loss function aimed at maximizing the expected reward; using the loss function to update the weight matrix in the feature fusion network through a backpropagation algorithm.

[0024] Furthermore, the loss function and update rule can be expressed as follows:

[0025] ,

[0026] ,in, Represents the mathematical expectation. It is a quantitative guide content The quality reward function is based on user behavior data such as likes and listen completion rates; It is the loss function that needs to be minimized, i.e., the negative expected reward; These are the learnable parameters of the attention network; It is the learning rate.

[0027] Furthermore, after generating the tour guide content, the process further includes: calling a text-to-speech engine to convert the tour guide content into speech and playing it; and calling an augmented reality rendering engine to overlay virtual information tags synchronized with the tour guide content onto the image area where the target scene is located on the screen displaying the real-time image data.

[0028] Another aspect of the present invention provides a multimodal intelligent tour guide system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multimodal intelligent tour guide method as described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal intelligent guide method as described above.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] 1. This invention uses real-time visual data collected by a camera as the core input to directly analyze what the user sees, fundamentally solving the problems of vague positioning and inability to identify specific targets in traditional GPS solutions. By combining deep learning models to analyze images, it significantly improves the accuracy and granularity of target recognition, and can improve the triggering precision of tour guide information from the macro-level of regions to the fine-grained target level, ensuring a high degree of consistency between tour guide content and the user's real focus.

[0032] 2、The application constructs a scene feature vector, effectively integrates the user's personal preferences, geographical location and even real-time environmental factors into the content generation process, combines the feature fusion network, especially the attention mechanism, so that the system can dynamically determine which visual features are more important for the current user and the scene, thereby generating a personalized tour guide commentary, realizing the depth personalization and scene adaptation of the tour guide content, and greatly improving the user's immersion and satisfaction.

[0033] 3、The application uses the cloud server to perform deep analysis, ensures that low-delay processing from image acquisition, analysis to content generation can be realized on the mobile terminal with limited computing resources, provides a smooth interactive experience for the user, and introduces an adaptive processing module for special scenes such as low light, which significantly enhances the stability and practicality in the complex and variable real scenic environment.

[0034] 4、The application combines augmented reality technology to directly superimpose the tour guide information in the form of visual tags on the real scenery, realizes the diversification and visualization of the information presentation mode, and brings a more lively and technological tour experience for the user, and enriches the form of user interactive experience. BRIEF DESCRIPTION OF DRAWINGS

[0035] The drawings described herein are used to provide further understanding of the embodiments of the application, constitute a part of the application, and do not constitute a limitation on the embodiments of the application. In the drawings:

[0036] Figure 1 The overall method flowchart is provided for the embodiment 1 of the application.

[0037] Figure 2 The overall timing diagram is provided for the embodiment 1 of the application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the embodiments of the application more clear, the technical scheme in the embodiments of the application will be described clearly and completely in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations.

[0039] Embodiment 1

[0040] The embodiment discloses a multi-modal intelligent tour guide method, which is usually run on a mobile terminal of a user, such as a smart phone or smart glasses, or a portable voice tour guide device dedicated to a scenic spot. The mobile terminal or voice tour guide device has a camera, a GPS module, a processor and a display and audio output device.

[0041] Figure 1 The overall method flowchart in the embodiment is shown, Figure 2 The overall timing diagram in the embodiment is shown, from Figure 1 It can be seen from the embodiment that the embodiment includes the following steps:

[0042] Step 1: First, the guide application program configured on the mobile terminal acquires data of multiple sources in real time by calling a hardware interface, collects multi-modal data, integrates the collected multi-modal data into a complete data packet, and sends the data packet to the cloud.

[0043] Specifically, in the embodiment, the collected multi-modal data includes real-time image data, real-time positioning data, and user preference data.

[0044] The real-time image data refers to one or more images in a video stream continuously collected by the rear camera of the mobile terminal, which directly reflects the field of view currently observed by the user.

[0045] The real-time positioning data is obtained by a GPS module, base station positioning, or Wi-Fi positioning, and is used to provide the latitude and longitude coordinates of the user.

[0046] The user preference data is an interest label (such as historical, architectural, natural scenery, and character anecdotes) actively filled in by the user when using the application for the first time or in the settings, or an implicit preference mined by analyzing the user's historical tour behavior, which is pre-stored in the local storage.

[0047] It can also include real-time crowd density data and weather data covering the entire scenic area obtained from the central server deployed in the scenic area through a network communication module (such as 4G / 5G or Bluetooth, Wi-Fi). The data can be obtained by monitoring cameras, heat map sensors, or communication signaling data analysis in each area of the scenic area, and is issued to the mobile terminal in the form of a grid map.

[0048] Step 2: After receiving the data packet sent by the mobile terminal, the cloud converts the multi-source heterogeneous data in the data packet into a unified and standardized mathematical representation. In order to achieve comprehensive perception of the user's situation, when processing the multi-source heterogeneous data in the data packet, it is necessary to formally describe a complete situation snapshot. In this situation snapshot, the user's view, the precise geographic location, the personal interest preference, and the objective state of the surrounding environment (such as crowd density) are included.

[0049] Specifically, in the present embodiment, this scenario snapshot can be described by defining a multi-modal data set. By integrating the real-time data streams from the camera, GPS, user profile database, and scenic spot server in the data packet into a unified structured framework. At the same time, considering the complexity of the real world, especially the huge impact of light changes on visual recognition, an ambient brightness evaluation module can be introduced to determine whether it is currently in a low light environment. If so, before the core analysis, automatically start the image enhancement algorithm to preprocess the collected images, so as to ensure the input quality of the subsequent visual analysis module, thereby improving the stability and reliability of the entire system under adverse lighting conditions such as dusk and indoor.

[0050] Exemplarily, in the present embodiment, the multi-modal input data set at any time can be represented by the following formula:

[0051] ,

[0052] wherein, is the real-time collected image data; is the real-time positioning data; is the pre-stored user preference label set; is the real-time crowd density data provided by the scenic spot server.

[0053] Exemplarily, in the present embodiment, the ambient brightness evaluation module can be represented by the following formula:

[0054] ,

[0055] wherein, is a function of evaluating the average brightness of the input image ; is a preset brightness threshold value for determining whether to enter low light conditions; is a low light image enhancement algorithm, for example, a deep learning-based enhancement network, whose parameters are ; is the image after the conditional preprocessing module, which is finally used for subsequent analysis.

[0056] Step 3: According to the visual feature database of the scenic spot configured in the cloud server, the key scenic object that the user may be interested in is automatically recognized from the acquired real-time image data, and is converted into a visual feature vector that the machine can understand.

[0057] Specifically, in the present embodiment, a collaborative recognition model that takes into account the low latency and low power consumption of mobile terminals and the high precision of cloud terminals can be constructed to accurately recognize the specific scene currently seen by the user according to the original image data and provide high-quality input for subsequent personalized content generation.

[0058] Generally, high-quality visual feature extraction usually relies on a large and complex deep learning model, which is difficult to run in real time on a mobile terminal with limited computing resources; while a lightweight mobile terminal model has difficulty in meeting the needs of fine tour guide due to the low accuracy of the extracted features. To solve this contradiction, in the present embodiment, a terminal lightweight perception-cloud precise recognition collaborative workflow is designed. The recognition task is divided into two logically decoupled stages: rapid positioning based on what the user is looking at, and in-depth analysis of what it is.

[0059] Rapid positioning by mobile terminal, due to the limited computing power of mobile terminal, it is not responsible for understanding the whole content of the image, but only needs to quickly generate a compact visual fingerprint that is sufficient to distinguish different images. This fingerprint can be regarded as a unique barcode, which has very low calculation cost and is designed for efficient retrieval.

[0060] The cloud server is responsible for in-depth analysis, and a central database containing high-quality features of all key scenes in the scenic area (i.e. the visual feature database of the scenic area) is pre-stored in the cloud server. After receiving the visual fingerprint and geographic location sent by the terminal, the cloud server:

[0061] 1) First, use the geographic location to narrow the search range from the entire database to a few candidate scenes around the user, thereby improving the retrieval efficiency.

[0062] 2) Then, in this small range, the visual fingerprint is compared to accurately match the scene being observed by the user.

[0063] 3) Once the match is successful, the server returns to the terminal a high-dimensional, high-information-content visual feature vector pre-stored in the database and calculated offline by an advanced model.

[0064] 4) At the same time, in order to deal with abnormal situations such as poor network or the target not in the database, the collaborative recognition model also has a degradation processing mechanism. When the cloud collaborative path fails, the system will automatically switch back to the local, and a basic CNN model on the terminal will be used to complete feature extraction, thereby ensuring that the core function is not interrupted.

[0065] For example, in the present embodiment, the core of the entire collaborative recognition model, the visual feature database of the scenic area, can be defined structurally as follows:

[0066] ,

[0067] wherein, represents the scenic spot visual feature database, which is a database containing scenic items, each item is a four-tuple: is the high-quality visual feature vector of the th scenic spot, which is the first feature vector that is finally desired and used for subsequent content generation, the visual feature vector is extracted offline on the server side by a powerful model (such as ViT, ResNet-152), which contains rich semantic information; is the low-dimensional visual fingerprint corresponding to the above, which is designed for fast and efficient similarity matching, and has low dimension, which is convenient for storage and fast calculation; is the geographic coordinates of the scenic spot, which is used as a spatial index and is the key to realize fast candidate set screening; is the unique identifier of the scenic spot.

[0068] Exemplarily, in the embodiment, the entire collaborative workflow can be mathematically described by the following conditional model:

[0069] ,

[0070] The terminal side visual fingerprint extraction, i.e. the generation of input in the above formula, can be as shown in the following formula:

[0071] ;

[0072] To narrow the range of matching, the candidate set screening on the cloud server side can be as shown in the following formula:

[0073] The candidate set screening here uses to represent the Euclidean distance weight function, which is used to calculate the geographic distance between the user's real-time location and the location of each scenic spot in the database. Only when the distance is less than the preset search radius (e.g. 20 meters), the features of the scenic spot will be included in the candidate set , thereby reducing the global visual search originally needed in thousands of targets to only in a few or dozens of candidate targets, so that the calculation efficiency is greatly improved.

[0074] The precise positioning in the candidate set, i.e. the visual matching on the cloud server side, can be as shown in the following formula:

[0075] Here, in the small range of candidate set , the server uses an efficient similarity calculation function Compare the query fingerprints uploaded by the terminals one by one. and each pre-stored fingerprint in the candidate set ,and The function will then return the database entry with the highest similarity score, i.e. .

[0076] Finally, the core judgment logic in the main formula: This takes effect here. This represents the minimum acceptance standard for the matching result. If the highest similarity score is still greater than or equal to this threshold, it proves that this is a high-quality, high-reliability successful match. The system will then adopt this result and include the pre-stored high-quality feature vector in that entry. As the final output .

[0077] Otherwise (for example, a maximum score of only 0.6 indicates that the user may have photographed an object outside the database or from a very difficult angle), the system will determine that the cloud-based matching failed, at which point the fallback mechanism is activated. The system will then switch to local operation and call a backup lightweight CNN model. The system performs basic feature extraction on the image to generate a feature vector that, while not perfect, is usable as output, thus ensuring that the tour guide function can continue to operate even when the user is without a network or is shooting uncommon scenery.

[0078] It should be noted that in the above formula, The first feature vector, which is the final output, will be used for subsequent feature fusion and content generation. This feature vector is a mathematical, high-dimensional, and high-information-content representation of the scene currently seen by the user. The best match in the cloud, also known as a high-quality feature, is a pre-stored high-quality feature vector in the cloud database that best matches the scene from the user's current perspective. This feature vector represents the successful result of the cloud-based collaborative path. When a match is successful, the system selects to use this feature, which is generated offline by a powerful model, to ensure the highest accuracy of subsequent tour guide content. Here, is a similarity function, which measures the degree of similarity between two vectors. In this embodiment, it can be specifically defined as cosine similarity. It is a key basis for decision-making, quantifying the degree of matching between the fingerprint of the user's real-time captured image and the fingerprint of the best candidate in the database. Its value range is usually between [-1, 1], and the closer it is to 1, the more similar it is. To retrieve fingerprints, an ultra-lightweight network on the mobile device is used. For real-time images The generated visual fingerprint of the query is a compact representation of the scene currently seen by the mobile terminal for retrieval purposes; For the best matching fingerprint, this is the candidate set after filtering by geographic location. In the middle, with fingerprint query The pre-stored visual fingerprint with the highest similarity is the best target point found by the cloud database in response to terminal queries; The matching confidence threshold is a pre-defined key hyperparameter used to determine whether a match is valid. It is a floating-point number between 0 and 1 (e.g., 0.9). This confidence threshold defines the system's level of confidence. The system considers a cloud match to be reliable and successful only when the highest similarity score calculated is greater than or equal to this threshold. The setting of this parameter requires a trade-off between recall (not missing correct matches) and precision (not accepting incorrect matches). This is a local CNN model, a lightweight backup CNN model pre-installed on the mobile device. It can be seen as a safety net for the system; when the cloud-based collaborative path fails for any reason (e.g., network interruption, insufficient similarity), this model is activated to process the image locally. Performing a complete feature extraction process, although the quality of the extracted features may not be as good as that from the cloud. However, it guarantees that even in the worst case, the system can still produce a usable feature vector, thus ensuring the robustness and offline availability of the entire tour guide service; This represents the set of learnable parameters for the local CNN model.

[0079] The method in this step yields a first feature vector, namely the visual feature vector. This visual feature vector is a mathematical abstraction of the core scene in the image, enabling the system to truly understand what the user is seeing. This allows the entire system to go beyond simple geolocation and achieve target-level recognition of specific scenes in the field of vision.

[0080] Step 4: In order for the system to understand 'who is watching' and 'in what context', the system needs to uniformly encode the user's personal attributes and environmental factors.

[0081] Specifically, in this step, different encoding functions can be designed to uniformly convert heterogeneous data such as user geolocation, interests, and surrounding pedestrian density into vectorized representations. For example, geographic coordinates are mapped to a discrete grid vector, and user preference tags are converted into a multi-hot encoded vector. Finally, these vectors are concatenated with scalar pedestrian density data to form a unified second feature vector, namely the contextual feature vector. This vector is key to achieving personalized recommendations, providing the model with rich personalized and contextualized information that goes beyond the visual content itself.

[0082] Exemplarily, in the present embodiment, the multi-source context feature vector can be constructed by the following formula:

[0083]

[0084] wherein, the latitude-longitude coordinate is mapped into a vector by a geo-space encoding function; is a function that converts the user preference set into a multi-hot encoding vector; is to treat the scalar people flow density data as a one-dimensional vector; represents a vector concatenation operation; and the final generated is the comprehensive context feature vector containing user location, preference and environmental information.

[0085] Step 5: Based on the first feature vector and the second feature vector, the two feature vectors are effectively fused by an attention mechanism-based feature fusion network to obtain a fusion feature vector, which provides the basis for subsequent generation of personalized content.

[0086] Specifically, in the present embodiment, the attention mechanism-based feature fusion network takes the second feature vector containing user preference and other context information as the query (Query), and takes the first feature vector containing visual information of the scene as the key (Key) and the value (Value). The attention mechanism calculates the similarity between the query and each key to generate a set of attention weights. This set of weights is then used to perform weighted summation on the values (i.e., each channel or spatial position of the visual feature vector). In this way, the system can automatically focus on the most important part of the visual features according to the user's preferences. For example, if the user's preference is architecture, the attention mechanism will automatically assign higher weights to the visual feature channels related to plaques, dougong structures, etc. The fusion feature vector generated in this way can highlight the information most relevant to the user's interest.

[0087] Meanwhile, in order to make the attention mechanism more intelligent, its 'attention weight' can be dynamically learned and optimized. The system can collect the user's subsequent feedback behavior, such as whether the user listened to a certain explanation completely (implicit positive feedback), or quickly skipped a certain explanation (implicit negative feedback), or liked / shared a certain explanation (explicit positive feedback). These feedbacks can be used to build a loss function that aims to minimize the deviation between the predicted content and the user's true interest. Through the backpropagation algorithm, the parameters of the attention network are continuously fine-tuned, making it more and more skilled at capturing the user's personalized needs.

[0088] Exemplarily, in the present embodiment, the attention mechanism-based feature fusion network can be represented by the following formula:

[0089] ,

[0090] wherein, is a learnable weight matrix used to project the input vector into Query, Key and Value space; generated by the second feature vector (context feature) Query vector dominates the whole matching process and is the key to achieve personalization; generated by the first feature vector (visual feature vector) Key vector and Value vector provide the content to be filtered; function converts the matching score into attention weight in the form of probability distribution; the final generated fusion feature vector is a highly contextualized and personalized information representation that has been dynamically filtered and reorganized according to the current context.

[0091] Step 6: Based on the fusion feature vector, it is converted into a lively and fluent natural language explanation that users can directly receive. In this step, according to the fusion feature vector that combines visual and context information, it is translated into natural, fluent, and highly relevant tour guide content.

[0092] Specifically, in this step, a sequence-to-sequence tour guide content generation model based on the Transformer architecture is used to complete the translation work. The model consists of an encoder and a decoder. The encoder is responsible for receiving and understanding the fusion feature vector and encoding it into a semantic-rich context representation. The decoder is an autoregressive generation module that generates a word sequence corresponding to the tour guide text based on the context representation. After generating the text, it can be converted into speech data through a standard text-to-speech (TTS) engine. At the same time, when the fusion feature vector contains high human flow density information, the content generation network can retrieve and generate similar prompt information such as 'the current area is crowded, please visit the XX palace next to you first, which has less people' in its knowledge base.

[0093] Exemplarily, in this embodiment, the tour guide content generation model can be described by the following probability formula:

[0094] ,

[0095] wherein, is the fusion feature vector, is the target generated tour guide text sequence containing words; is the encoder of Transformer, responsible for encoding the input image into context information; is the conditional probability computed by the decoder of Transformer, i.e. the probability of the next word given the context and the sequence of already generated words is the parameter set of the whole generation network.

[0096] It is worth mentioning that the core function of the example probability formula is similar to a translator, which receives the fusion feature vector as input and decodes it into a sequence of words. This generation process is autoregressive, i.e. when generating each word, it refers to the input fusion vector and all the previous generated content. Since the starting point of the decoding itself is highly personalized, the final generated tour guide text is fundamentally determined by the user's preferences, location and environment in terms of content, style and focus, which is the final execution link to realize the thousand faces of tour guide service.

[0097] And in order to make the personalized recommendation ability of the model continue to evolve, a feedback loop can be established to enable the model to learn and optimize itself from the user's real behavior.

[0098] Specifically, a dynamic learning and optimization process of attention weight can be introduced to dynamically fine-tune the core parameters of the attention mechanism.

[0099] First, define a reward function to quantify the user's satisfaction with the generated tour guide content . For example, the user's complete listening to the explanation can be considered as high reward, while fast skipping can be considered as low reward or punishment.

[0100] Then, maximize the long-term expected reward as the optimization goal, and adjust the weight matrix in the attention network through optimization algorithms such as gradient descent. This process enables the attention mechanism to continuously learn the user's real preference patterns. If the model finds that when it 'focuses' on architectural details, the user always gives positive feedback, it will gradually increase the attention weight of this type of visual features, so that future recommendations become more and more accurate.

[0101] Exemplarily, the dynamic learning and optimization process of attention weight in the embodiment can be realized through the following loss function and update rule:

[0102] ,

[0103] ,

[0104] wherein,​​ is a reward function quantifying the quality of the guide content based on the user's like, listening completion rate, and other behavior data; is a loss function that needs to be minimized, i.e., the negative expected reward; is a learnable parameter of the attention network; is a learning rate. It should be noted that this optimization process enables the model to continuously and adaptively adjust according to the user's implicit feedback, truly realizing the improvement of the personalization capability.

[0105] Step S6: Finally, the generated content is presented in a user-friendly manner. The conventional output method is to play the generated voice data through the speaker of the mobile terminal, or to display the text on the screen.

[0106] In this embodiment, the final output step can be combined with augmented reality (AR) technology. In addition to displaying the real-time camera image on the screen of the mobile terminal, the system can also call the AR rendering engine to superimpose a virtual highlight label or information box on the image area where the recognized scenery target (such as a plaque) is located. This AR label can appear synchronously with the guide voice and be attached to the target object in real time as the user's lens moves, thereby providing the user with a more intuitive and interactive visual guide.

[0107] The above specific embodiments further illustrate the purpose, technical solutions, and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal intelligent tour guide method, characterized in that, The multi-modal intelligent guide method comprises: On the mobile terminal side, the collected data comprises real-time image data, real-time positioning data and preset user preference data; On the mobile terminal side, a visual fingerprint for retrieval is generated based on the real-time image data, and the visual fingerprint and the real-time positioning data are sent to a cloud server; The cloud server filters a candidate scene set from a preset visual feature database based on the real-time positioning data, and determines a target scene in the candidate scene set by matching the visual fingerprint, and acquires a pre-stored high-quality visual feature vector corresponding to the target scene as a first feature vector representing image content; A second feature vector representing a user scenario is generated according to the real-time positioning data and the user preference data; The first feature vector and the second feature vector are fused by a feature fusion network based on an attention mechanism to obtain a fused feature vector; Guide content is generated based on the fused feature vector; The fused feature vector is obtained by the following steps: The second feature vector or a linear transformation thereof is taken as a query; The first feature vector or a linear transformation thereof is taken as a key and a value; An attention weight is obtained by calculating the similarity of the query and the key, and the value is weighted and summed using the attention weight, thereby obtaining the fused feature vector; The guide content is realized by a pre-trained sequence-to-sequence model based on a Transformer architecture, wherein The encoder of the sequence-to-sequence model based on the Transformer architecture receives the fused feature vector, and the decoder generates a text sequence of the guide content autoregressively. 2.The multi-modal intelligent tour guide method of claim 1, wherein, The step of determining the first feature vector by the cloud server further comprises: When the highest similarity of the candidate scene set matching the visual fingerprint is lower than a preset matching threshold, a degradation processing mechanism is triggered; The real-time image data is processed by a local model on the mobile terminal side to generate the first feature vector. 3.The multi-modal intelligent tour guide method of claim 1 or 2, wherein, Before the visual fingerprint is generated on the mobile terminal side, the following steps are further included: The ambient brightness of the real-time image data is evaluated; When the ambient brightness is lower than a preset brightness threshold, low-light image enhancement processing is performed on the real-time image data. 4.The multi-modal intelligent tour guide method of claim 1, wherein, The data collected on the mobile terminal side further comprises real-time crowd density data; and The second feature vector is generated by the following steps: The real-time positioning data, the user preference data and the real-time crowd density data are respectively vectorized and encoded, and the encoded vectors are spliced to generate the second feature vector. 5.The multi-modal intelligent tour guide method of claim 1, wherein, The multi-modal intelligent guide method further comprises: Collecting user feedback behavior data on the generated guide content; Iteratively optimizing network parameters in the feature fusion network based on the feedback behavior data. 6.The multi-modal intelligent tour guide method of claim 5, wherein, The step of iteratively optimizing network parameters in the feature fusion network based on the feedback behavior data comprises: A preset reward function is used to quantify the quality score of the guide content according to the feedback behavior data. constructing a loss function aiming to maximize the expected reward based on the quality score; updating the weight matrix in the feature fusion network by using the loss function through a back propagation algorithm. 7.The multi-modal intelligent tour guide method of claim 1, wherein, The step of generating the guide content further comprises: calling a text-to-speech engine to convert the guide content into speech and play the speech; and calling an augmented reality rendering engine to superimpose a virtual information tag synchronized with the guide content on an image region where the target scene is located on a picture displaying the real-time image data.

8. A multi-modal intelligent guide system, characterized in that, The multi-modal intelligent guide system comprises: a processor; a memory storing a computer program, when the computer program is executed by the processor, the multi-modal intelligent guide method as claimed in any one of claims 1-7 is implemented.

Images