A mixed reality tour method, system and device

Abstract

The application discloses a mixed reality guide method, system and device, which comprises the following steps: calculating a user's rough position according to the positioning of a mixed reality device and loading the meta-features of each candidate exhibit; collecting camera images and identifying the core target in the user's current field of view; inputting the user's rough position, the core target and the user's voice instruction into a pre-trained visual language model to extract a real-time feature vector; the visual language model takes the user's rough position as a forced constraint generation space of the real-time feature vector; calculating the similarity between the real-time feature vector and the meta-features to determine the target exhibit; extracting the target task from the user's voice instruction and generating and executing the task execution strategy of the target task according to the information of the target exhibit. By using the semantic recognition capability of the visual language large model to replace the physical space coordinate comparison, the high-precision picture explanation function in a complex environment can be realized.

Description

Technical Field

[0001] This invention relates to the field of interactive experience technology, and in particular to a mixed reality tour guide method, system, and device. Background Technology

[0002] Mixed Reality (MR) combines the features of Virtual Reality (VR) and Augmented Reality (AR), enabling the overlay of virtual elements onto the real environment. Incorporating MR into guided tours can guide visitors to spontaneously explore scenarios such as museums, art galleries, basic agricultural field identification, and lightweight industrial inspections.

[0003] Current mainstream mixed reality guided tour solutions typically rely on SLAM (Simultaneous Localization and Mapping) technology to anchor virtual objects to real space, and combine this with high-precision 3D rendered content for interaction. However, in order to support SLAM calculations and high-precision 3D rendering, the devices must use high-end chips, resulting in expensive hardware, short battery life (usually less than 2 hours), and severe heat generation. Summary of the Invention

[0004] The embodiments of the present invention aim to provide a mixed reality tour guide method, system and device, which utilizes the semantic recognition capability of a large visual language model to replace physical space coordinate comparison, and can achieve a highly accurate map viewing and explanation function in complex environments.

[0005] In a first aspect, embodiments of the present invention provide a mixed reality guided tour method, comprising: Based on the location of the mixed reality device, calculate the user's approximate location and load the meta-features of each candidate exhibit within the area where the user's approximate location is located; Capture images from the camera of the mixed reality device and identify the core targets in the user's current field of view; In response to a user's voice command, the user's approximate location, the core target, and the user's voice command are input into a pre-trained visual language model to extract real-time feature vectors; the visual language model uses the user's approximate location as a forced constraint generation space for the real-time feature vectors. Calculate the similarity between the real-time feature vector and the meta-feature, and determine the target exhibit based on the similarity. The target task is extracted from the user's voice command, and a task execution strategy for the target task is generated and executed based on the information of the target exhibit in the guide knowledge base.

[0006] As an improvement to the above scheme, the step of calculating the user's approximate location based on the positioning of the mixed reality device and loading the meta-features of each candidate exhibit within the area where the user's approximate location is located includes: The approximate location of the user is calculated based on the Bluetooth signal strength received by the mixed reality device; Using the user's approximate location as the center, determine the area where the user's approximate location is located according to a preset area radius; All exhibits within the region are selected from the local knowledge base as candidate exhibits, and the meta-features of each candidate exhibit are obtained.

[0007] As an improvement to the above solution, the acquisition of camera images from the mixed reality device, and the identification of core targets in the user's current field of vision, includes: Capture camera images from mixed reality devices; A lightweight object detection algorithm is used to extract bounding boxes that represent exhibits from the camera images. The detection box is deduplicated to obtain the core target in the user's current field of view.

[0008] As an improvement to the above scheme, the visual language model includes a visual encoder, a position encoder, a speech encoder, and a cross-modal attention module. The method for constructing the visual language model includes: A position encoder is added to a pre-trained general visual language model to inject position priors; the general visual language model includes a visual encoder, a speech encoder, and a cross-modal attention module.

[0009] As an improvement to the above scheme, the training method of the visual language model includes: Obtain the triplet dataset for the current guided tour scenario; the triplet dataset includes exhibit images, exhibit coordinates, and expert commentary. Freeze the visual encoder and the speech encoder, and add a low-rank adaptive bypass to the cross-modal attention module; Using the triplet dataset, the visual language model with the visual encoder and the speech encoder frozen is trained, and the parameters of the position encoder and the low-rank adaptive bypass are updated to obtain the trained visual language model.

[0010] As an improvement to the above solution, the step of responding to user voice commands by inputting the user's approximate location, the core target, and the user's voice command into a pre-trained visual language model to extract real-time feature vectors includes: In response to a user's voice command, the user's approximate location, the core target, and the user's voice command are input into a pre-trained visual language model; The location encoder extracts the position vector of the user's approximate location, the visual encoder extracts the image features of the core target, and the voice encoder extracts the voice command features of the user's voice command. The location vector, image features, and voice command features are fused using a cross-modal attention module to obtain a real-time feature vector.

[0011] As an improvement to the above scheme, the step of calculating the similarity between the real-time feature vector and the meta-feature, and determining the target exhibit based on the similarity, includes: Calculate the cosine similarity between the real-time feature vector and the meta-features of each candidate exhibit; The cosine similarity scores are sorted, and if the highest cosine similarity score is greater than a preset threshold, the corresponding candidate exhibit is determined as the target exhibit.

[0012] As an improvement to the above solution, the step of extracting the target task from the user's voice command and generating and executing a task execution strategy for the target task based on the information of the target exhibit in the guide knowledge base includes: Extract the target task from the user's voice command; the target task includes at least one of generating visual information and generating audible information. When the target task includes generating visual information, visual materials are obtained based on the information of the target exhibit in the guide knowledge base; A lightweight monocular depth estimation network is used to generate a relative depth map of the visualization material. Based on the virtual imaging surface distance of the mixed reality device and the human interpupillary distance, the parallax of the relative depth map is calculated, a relative depth map with parallax is generated, and the relative depth map with parallax is rendered on the off-axis freeform surface module of the mixed reality device. When the target task includes generating audible information, audible materials are obtained based on the information of the target exhibit in the guide knowledge base; Based on the audible material, narration audio is generated and loaded onto the audio module of the mixed reality device.

[0013] Secondly, embodiments of the present invention also provide a mixed reality tour guide system, comprising: The meta-feature acquisition module is used to calculate the user's approximate location based on the positioning of the mixed reality device, and load the meta-features of each candidate exhibit within the area where the user's approximate location is located. The core target recognition module is used to acquire camera images from the mixed reality device and identify the core targets in the user's current field of view; The real-time feature vector extraction module is used to respond to user voice commands by inputting the user's approximate location, the core target, and the user's voice command into a pre-trained visual language model to extract real-time feature vectors; the visual language model uses the user's approximate location as a forced constraint generation space for the real-time feature vectors. The target exhibit determination module is used to calculate the similarity between the real-time feature vector and the meta-feature, and determine the target exhibit based on the similarity. The strategy generation module is used to extract the target task from the user's voice command and generate a task execution strategy for the target task based on the information of the target exhibit in the guide knowledge base.

[0014] Thirdly, embodiments of the present invention also provide a mixed reality guided tour device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the mixed reality guided tour method as described above.

[0015] Compared with existing technologies, the present invention discloses a mixed reality tour guide method, system, and device. This method calculates the user's approximate location based on the positioning of the mixed reality device and loads the meta-features of each candidate exhibit within the area where the user's approximate location is located. It then acquires camera images from the mixed reality device to identify core targets in the user's current field of view. In response to user voice commands, it inputs the user's approximate location, the core targets, and the user's voice commands into a pre-trained visual language model to extract real-time feature vectors. The visual language model uses the user's approximate location as a forced constraint to generate a space for the real-time feature vectors. It calculates the similarity between the real-time feature vectors and the meta-features, and determines the target exhibit based on the similarity. Finally, it extracts the target task from the user's voice commands and generates and executes a task execution strategy for the target task based on the information of the target exhibit in the tour guide knowledge base. By utilizing the semantic recognition capabilities of a large visual language model to replace physical space coordinate comparison, this invention enables highly accurate map interpretation and explanation functions in complex environments. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a mixed reality guided tour method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a mixed reality tour guide system provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In the description and claims, it should be understood that the terms "first," "second," etc., used in the description and claims are only for the purpose of distinguishing the description of the same technical features, and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated, nor necessarily the order of description or chronological order. The terms are interchangeable where appropriate. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0019] This invention provides a mixed reality guided tour method. Please refer to [link / reference]. Figure 1 In this embodiment, the mixed reality guided tour method is specifically executed through steps S1 to S5: S1. Based on the positioning of the mixed reality device, calculate the user's approximate location and load the meta-features of each candidate exhibit within the area where the user's approximate location is located; S2. Acquire images from the camera of the mixed reality device and identify the core targets in the user's current field of vision; S3. In response to the user's voice command, the user's approximate location, the core target, and the user's voice command are input into a pre-trained visual language model to extract real-time feature vectors; the visual language model uses the user's approximate location as a forced constraint generation space for the real-time feature vectors. S4. Calculate the similarity between the real-time feature vector and the meta-feature, and determine the target exhibit based on the similarity. S5. Extract the target task from the user's voice command, and generate and execute the task execution strategy based on the information of the target exhibit in the guide knowledge base.

[0020] Mixed reality devices are head-mounted or handheld smart devices that can integrate real-world views with virtual information displays, and have positioning, image acquisition, and voice interaction capabilities.

[0021] In a preferred embodiment, the mixed reality device employs a split, lightweight architecture, comprising a glasses component and a host component: The optical system at the end of the glasses adopts an off-axis freeform surface optical solution, characterized by a real-scene transmittance of up to 80% and a thin module thickness, which is compatible with most myopia glasses on the market.

[0022] The glasses' sensing element consists of only a single wide-angle camera (720P / 1080P) for capturing a first-person view image stream. Expensive IMU arrays and depth sensors are eliminated. The main unit uses a neckband or hanging design, connecting to the glasses via a single Type-C cable. The main unit can utilize a low-end commercial SoC, focusing on enhancing the performance of the neural network processing unit to adapt to edge-side inference of the visual language model. A dual-microphone array is also integrated into the main unit, leveraging physical location advantages and algorithms for targeted noise reduction, focusing on capturing the user's voice.

[0023] It should be noted that the process of calculating the user's coarse location does not involve complex positioning algorithms such as visual feature point extraction, environmental map construction, or user trajectory tracking. Therefore, in this embodiment of the invention, the hybrid display device does not need to be configured with an expensive SLAM (Simultaneous Localization and Mapping) module, but performs coarse positioning via Bluetooth or GPS, and shifts the logic to large model-driven operation, reducing power consumption and weight, solving the pain point of positioning failure in congested environments, and significantly reducing hardware costs.

[0024] Meta-features refer to the static attribute vectors pre-stored for each candidate exhibit, such as exhibit name, category, theme, and keywords. These meta-features are stored in the form of numerical values ​​or embedded vectors and are used for subsequent similarity comparison with real-time feature vectors.

[0025] A single frame or consecutive frames of images captured in real time on a mixed reality device can cover a real-world scene with a field of view of approximately 60° to 100° from the user's current field of vision. The camera images serve as input for visual content recognition. Through object detection algorithms such as YOLO and Faster R-CNN, the camera images can be processed to output entities that occupy a large area, have high confidence, or are close to the center of the image.

[0026] The natural language statements spoken by users through the microphone of a mixed reality device contain their interactive intentions, but these may be ambiguous and require visual cues for clarity. Therefore, in this embodiment of the invention, a pre-trained visual language model maps image input and text input to the same semantic space.

[0027] It is worth noting that, in generating the implementation feature vector, this embodiment of the invention uses the user's approximate location as a mandatory spatial constraint. That is, during encoding, the visual language model does not treat location information as a regular input feature, but instead forces the model to activate and fuse features only within the exhibit meta-feature subspace corresponding to the user's location through an attention mask. In this way, the output implementation feature vector naturally only contains the information dimension of exhibits within that region, while exhibit meta-features outside the region are masked before similarity calculation.

[0028] Real-time feature vectors represent the sum of the user's current intent and their perception of the scene, while meta-features are the inherent attributes of the exhibits. The similarity between real-time feature vectors and meta-features reflects the degree of matching between the content the user wants to see and the exhibit itself. Therefore, there is no need for complex rule engines or state machines; target exhibits can be determined through vector retrieval.

[0029] The guide knowledge base stores various multimedia information about the exhibits, including but not limited to text descriptions, audio files, 3D model paths, links to related exhibits, historical timelines, and frequently asked questions, in order to accurately respond to users' voice commands.

[0030] Compared to the conventional approach of inputting the entire image into a large model for blind guessing, this embodiment of the invention first narrows the search range by physical location, then extracts a local image of the core target, and then performs vector comparison. This funnel mechanism reduces the computational power consumption and recognition error rate on the edge.

[0031] In the above scheme, the semantic recognition capability of the visual language large model is used to replace the physical space coordinate comparison, which can realize the function of high-precision image interpretation in complex environments.

[0032] As a preferred implementation, step S1, based on the positioning of the mixed reality device, calculates the user's approximate location and loads the meta-features of each candidate exhibit within the area where the user's approximate location is located, including: The approximate location of the user is calculated based on the Bluetooth signal strength received by the mixed reality device; Using the user's approximate location as the center, determine the area where the user's approximate location is located according to a preset area radius; All exhibits within the region are selected from the local knowledge base as candidate exhibits, and the meta-features of each candidate exhibit are obtained.

[0033] Bluetooth signal strength changes monotonically with the distance between the mixed reality device and each Bluetooth beacon, serving as the raw data source for calculating the user's approximate location. By using the signal strength values ​​from multiple received Bluetooth beacons, and employing algorithms such as triangulation, signal attenuation models, or fingerprint matching, an estimate representing the user's current spatial location can be calculated.

[0034] Bluetooth positioning has inherent biases, but the user's actual location will certainly fall within a certain error range of this estimated point. Therefore, the user's approximate location is expanded into an area with a preset radius to correct for uncertain positioning results. Understandably, the area radius can be flexibly configured by the exhibition hall based on the density of exhibits, demonstrating scene adaptability.

[0035] For example, the host receives signal strength indication via Bluetooth Beacon, calculates the user's approximate location, delineates a location granularity of 3 meters in radius, and locates the user in Hall 2, Area A. Based on this, the host retrieves candidate exhibits and their meta-features from the local knowledge base in Hall 2, Area A.

[0036] The above scheme uses Bluetooth signal strength as the sole source of spatial information, fundamentally avoiding the increased hardware costs and computational burden brought about by high-precision positioning schemes such as visual SLAM or depth sensing.

[0037] As a preferred implementation, step S2, acquiring camera images from the mixed reality device and identifying key targets in the user's current field of view, includes: Capture camera images from mixed reality devices; A lightweight object detection algorithm is used to extract bounding boxes that represent exhibits from the camera images. The detection box is deduplicated to obtain the core target in the user's current field of view.

[0038] In some preferred embodiments, the YOLO-lite algorithm is used to detect the core target. This lightweight target detection algorithm can run quickly on resource-constrained mixed reality devices, locating and identifying objects of specific categories from images.

[0039] It should be noted that in the process of identifying the core target, we do not focus on what type of exhibit the core target is or its static attributes. The process only involves segmenting detection boxes from the image that may have the semantics of the exhibit.

[0040] Multiple overlapping or highly overlapping bounding boxes may be detected in the image. Through algorithms such as merging, filtering, or suppression, redundant boxes can be eliminated, and only the most important and representative single exhibit target that is ultimately determined in the user's field of vision can be retained as the visual focus for subsequent processes.

[0041] In the above solution, by introducing a lightweight object detection algorithm, visual processing can be run locally on the device in real time without uploading images to the host, thus avoiding network transmission delays. At the same time, the detection only focuses on whether the semantics are those of an exhibit, which greatly reduces the number of model parameters and inference computation.

[0042] In a preferred embodiment, the visual language model includes a visual encoder, a position encoder, a speech encoder, and a cross-modal attention module, and the method for constructing the visual language model includes: A position encoder is added to a pre-trained general visual language model to inject position priors; the general visual language model includes a visual encoder, a speech encoder, and a cross-modal attention module.

[0043] It should be noted that the general visual language model has been pre-deposited on the large public dataset Wancun, and it possesses general cross-modal understanding capabilities. However, it should also be noted that since the general visual language model only contains a visual encoder, a speech encoder, and a cross-modal attention module, its cross-modal understanding capability is limited to both visual and speech modalities.

[0044] In this embodiment of the invention, an encoder module is added to the existing architecture of the general visual language model. Its function is to receive the user's rough location information and encode it into location features, which are then injected into the model as additional prior inputs, so that the model is guided by spatial location when generating features.

[0045] The location prior is injected during the feature generation stage, ensuring that the cross-modal attention module always uses spatial location as a background reference when calculating the relationship between vision and speech, resulting in features that naturally have a regional bias. Furthermore, as a hard spatial cue, the location prior, unlike ordinary input features, plays a role in constraining the range of attention distribution during attention computation, effectively guiding the model to focus on the semantics of the exhibit corresponding to the current location.

[0046] The above scheme enables the visual language model to have dedicated spatial orientation perception capabilities, so as to perform joint visual and speech reasoning within a given spatial region.

[0047] Further, preferably, the training method of the visual language model includes: Obtain the triplet dataset for the current guided tour scenario; the triplet dataset includes exhibit images, exhibit coordinates, and expert commentary. Freeze the visual encoder and the speech encoder, and add a low-rank adaptive bypass to the cross-modal attention module; Using the triplet dataset, the visual language model with the visual encoder and the speech encoder frozen is trained, and the parameters of the position encoder and the low-rank adaptive bypass are updated to obtain the trained visual language model.

[0048] Exhibit images are photographs or video frames taken in real guided tour scenarios, reflecting the visual characteristics of the exhibits, such as their appearance and shape; exhibit coordinates are the physical coordinates of the exhibits in the exhibition hall. It should be noted that exhibit coordinates can be relative coordinates, such as their location in the physical space of the exhibition hall, rather than absolute coordinates such as latitude and longitude; expert commentary is an explanatory text written by experts in the field, containing relevant knowledge about the exhibit and narrative language, serving as a semantic reference for voice commands.

[0049] Since the basic architecture used in this embodiment of the invention is a trained general visual language model, which has robust visual and speech expression capabilities, freezing the parameters of the visual encoder and speech encoder during training can prevent catastrophic forgetting when the visual language model is fine-tuned on a small amount of scene data.

[0050] Low-rank matrices have fewer trainable parameters and are mathematically equivalent to making low-dimensional perturbations in the output space of the original module. This allows them to adapt to new tasks while maintaining constraints on the original capabilities.

[0051] The cross-modal attention module is responsible for fusing multimodal features in the visual language model. In this embodiment of the invention, by adding a low-rank adaptive bypass to the side of the original network structure of the cross-modal attention module, the module output can be adapted to the needs of the current task through an incremental path learned by a low-rank matrix without changing the original weights.

[0052] During model training, parameters are updated only for two small modules: the position encoder and the low-rank adaptive bypass. This results in faster convergence and good performance on a small amount of data. The final trained visual language model retains general visual language capabilities and gains spatial perception fusion capabilities specific to the current scene.

[0053] In the above scheme, the final model not only inherits the powerful cross-modal understanding foundation of the general pre-trained model, but also obtains the position constraint fusion capability for the current tour guide scenario through lightweight adaptation, and has good transferability. When the tour guide scenario is changed, it is only necessary to re-collect triple data and retrain the position encoder and low-rank bypass, and the general visual speech model part does not need any adjustment.

[0054] Preferably, step S3, in response to a user's voice command, inputs the user's approximate location, the core target, and the user's voice command into a pre-trained visual language model to extract real-time feature vectors, including: In response to a user's voice command, the user's approximate location, the core target, and the user's voice command are input into a pre-trained visual language model; The location encoder extracts the position vector of the user's approximate location, the visual encoder extracts the image features of the core target, and the voice encoder extracts the voice command features of the user's voice command. The location vector, image features, and voice command features are fused using a cross-modal attention module to obtain a real-time feature vector.

[0055] During the guided tour, users speak natural language statements expressing their tour needs or directional intentions to the mixed reality device—that is, user voice commands. These user voice commands serve as trigger signals, causing the system to initiate the feature extraction process as needed, avoiding continuous unnecessary calculations and reducing device power consumption.

[0056] In the visual-speech model, three encoders work in parallel, each converting the original input into a vector representation of the same dimension, unifying heterogeneous data into the same numerical feature space. The attention mechanism can automatically calculate the correlation weights between the three modal features and dynamically determine the contribution of position, vision, and speech to the final feature based on the current input.

[0057] In some preferred embodiments, the real-time feature vector is represented as: ; in, Image features; Features of voice commands; It is a position vector; , and This is the pre-trained weight matrix.

[0058] In the above scheme, the final output real-time feature vector integrates location, vision, and speech information. At the same time, it has been constrained to the correct spatial range at the fusion level by the location prior, providing a rich and spatially accurate input for subsequent similarity matching. This allows exhibit matching to be completed through simple vector retrieval without the need for complex spatial post-processing or cross-regional reordering.

[0059] As a preferred implementation, step S4, calculating the similarity between the real-time feature vector and the meta-feature, and determining the target exhibit based on the similarity, includes: Calculate the cosine similarity between the real-time feature vector and the meta-features of each candidate exhibit; The cosine similarity scores are sorted, and if the highest cosine similarity score is greater than a preset threshold, the corresponding candidate exhibit is determined as the target exhibit.

[0060] Cosine similarity measures similarity along vector directions and is insensitive to the absolute value of features. It can effectively handle differences in numerical scales among features of different modalities without requiring additional normalization. Furthermore, since the real-time feature vector in this embodiment of the invention integrates position vectors, image features, and voice command features, if the feature vector is similar to the candidate exhibit in the position, image, or voice dimension, it will be assigned a higher cosine similarity.

[0061] For example, if the user's voice command is "Why is its antenna broken?", and the candidate exhibits include the Great Purple Butterfly missing its left antenna, then the semantic vectors of the candidate exhibit and the user's voice command will highly overlap in the feature space, thus enabling the calculation of a higher cosine similarity.

[0062] If the cosine similarity with all candidate exhibits is very low, it means that the user's current 3D input may not correspond to any exhibit. Therefore, in this embodiment of the invention, a preset threshold is set as the decision boundary. Only when a candidate exhibit has the maximum cosine similarity and the maximum cosine similarity is greater than the preset threshold is the candidate exhibit regarded as the target exhibit.

[0063] The above solution avoids forcibly outputting incorrect results when the user's input is unclear or the input signal quality is poor, thus improving the user experience and system fault tolerance.

[0064] As a preferred implementation, step S5 involves extracting the target task from the user's voice command, generating and executing a task execution strategy for the target task based on the information of the target exhibit in the guide knowledge base, including: Extract the target task from the user's voice command; the target task includes at least one of generating visual information and generating audible information. When the target task includes generating visual information, visual materials are obtained based on the information of the target exhibit in the guide knowledge base; A lightweight monocular depth estimation network is used to generate a relative depth map of the visualization material. Based on the virtual imaging surface distance of the mixed reality device and the human interpupillary distance, the parallax of the relative depth map is calculated, a relative depth map with parallax is generated, and the relative depth map with parallax is rendered on the off-axis freeform surface module of the mixed reality device. When the target task includes generating audible information, audible materials are obtained based on the information of the target exhibit in the guide knowledge base; Based on the audible material, narration audio is generated and loaded onto the audio module of the mixed reality device.

[0065] Visualized information refers to the navigation content presented to users in a visual form, including but not limited to images, graphics, animations, text overlays, and other information forms that can be perceived by the eyes.

[0066] When the target task includes generating visualization information, corresponding high-resolution 2D image materials are retrieved from the material library. A lightweight monocular depth estimation network (such as an edge-compressed version of MiDaS) is used to generate a corresponding relative depth map based on the 2D image. Then, the required disparity for the left and right eyes is calculated based on the interpupillary distance and the distance to the virtual imaging surface.

[0067] In some preferred embodiments, relative depth map Left and right eye parallax Represented as: ; in, The interpupillary distance; Virtual focal length; This represents the distance to the virtual imaging surface.

[0068] After calculating the parallax, by rendering the left and right eye images with parallax on the host side and streaming them to the left and right micro-displays of the off-axis freeform surface module on the glasses, a "pseudo-3D floating window" with a sense of spatial depth can be formed in the user's field of vision by utilizing the principle of binocular parallax.

[0069] Audible information refers to the guide content presented to users in an auditory form, including voice narration, sound effects, audio guidance, and other information forms that can be perceived by the ears. This audible material is then converted into a standard audio format and loaded onto the device's audio module, allowing users to listen directly through the device's speakers or headphones.

[0070] The above solution ensures the consistency and accuracy of the audiovisual content; at the same time, the two links are independent of each other and can be executed in parallel, allowing visual rendering and audio playback to proceed simultaneously without blocking each other.

[0071] In one specific application embodiment, the user's approximate location is obtained via BLE / UWB. and load The system analyzes the meta-features of each candidate exhibit within the designated area; simultaneously, cameras capture real-time images to extract the core target; the user inputs a voice command, "How were the patterns on this stone formed?", and the system synchronously inputs the user's approximate location, core target, and voice command into a domain-fine-tuned visual language model, outputting intent parsing to confirm the target exhibit as a "meteorite" and the task objective as "explaining the formation of flow lines." Next, it retrieves a "magnified illustration of meteorite flow lines (2D)" from the resource library, generates left and right eye disparity maps through monocular depth estimation, and uses an off-axis freeform surface module to anchor and overlay the pseudo-3D image onto the user's actual line of sight, while simultaneously playing the generated audio explanation.

[0072] The mixed reality tour guide method provided by this invention utilizes the semantic recognition capability of a large visual language model to replace physical space coordinate comparison, thereby enabling highly accurate map viewing and explanation functions in complex environments.

[0073] This invention provides a mixed reality navigation system. Please refer to [link / reference]. Figure 2The mixed reality tour guide system includes a meta-feature acquisition module 11, a core target recognition module 12, a real-time feature vector extraction module 13, a target exhibit determination module 14, and a strategy generation module 15, wherein: The meta-feature acquisition module 11 is used to calculate the approximate location of the user based on the positioning of the mixed reality device, and load the meta-features of each candidate exhibit within the area where the approximate location of the user is located. The core target recognition module 12 is used to acquire camera images from the mixed reality device and identify core targets in the user's current field of vision; The real-time feature vector extraction module 13 is used to respond to user voice commands by inputting the user's approximate location, the core target, and the user's voice command into a pre-trained visual language model to extract real-time feature vectors; the visual language model uses the user's approximate location as a forced constraint generation space for the real-time feature vectors. The target exhibit determination module 14 is used to calculate the similarity between the real-time feature vector and the meta-feature, and determine the target exhibit based on the similarity. The strategy generation module 15 is used to extract the target task from the user's voice command and generate a task execution strategy for the target task based on the information of the target exhibit in the guide knowledge base.

[0074] In a preferred embodiment, the meta-feature acquisition module 11 is specifically used for: The approximate location of the user is calculated based on the Bluetooth signal strength received by the mixed reality device; Using the user's approximate location as the center, determine the area where the user's approximate location is located according to a preset area radius; All exhibits within the region are selected from the local knowledge base as candidate exhibits, and the meta-features of each candidate exhibit are obtained.

[0075] In a preferred embodiment, the core target recognition module 12 is specifically used for: Capture camera images from mixed reality devices; A lightweight object detection algorithm is used to extract bounding boxes that represent exhibits from the camera images. The detection box is deduplicated to obtain the core target in the user's current field of view.

[0076] In a preferred embodiment, the visual language model includes a visual encoder, a position encoder, a speech encoder, and a cross-modal attention module, and the method for constructing the visual language model includes: A position encoder is added to a pre-trained general visual language model to inject position priors; the general visual language model includes a visual encoder, a speech encoder, and a cross-modal attention module.

[0077] Further, preferably, the training method of the visual language model includes: Obtain the triplet dataset for the current guided tour scenario; the triplet dataset includes exhibit images, exhibit coordinates, and expert commentary. Freeze the visual encoder and the speech encoder, and add a low-rank adaptive bypass to the cross-modal attention module; Using the triplet dataset, the visual language model with the visual encoder and the speech encoder frozen is trained, and the parameters of the position encoder and the low-rank adaptive bypass are updated to obtain the trained visual language model.

[0078] Preferably, the real-time feature vector extraction module 13 is specifically used for: In response to a user's voice command, the user's approximate location, the core target, and the user's voice command are input into a pre-trained visual language model; The location encoder extracts the position vector of the user's approximate location, the visual encoder extracts the image features of the core target, and the voice encoder extracts the voice command features of the user's voice command. The location vector, image features, and voice command features are fused using a cross-modal attention module to obtain a real-time feature vector.

[0079] In a preferred embodiment, the target exhibit determination module 14 is specifically used for: Calculate the cosine similarity between the real-time feature vector and the meta-features of each candidate exhibit; The cosine similarity scores are sorted, and if the highest cosine similarity score is greater than a preset threshold, the corresponding candidate exhibit is determined as the target exhibit.

[0080] In a preferred embodiment, the strategy generation module 15 is specifically used for: Extract the target task from the user's voice command; the target task includes at least one of generating visual information and generating audible information. When the target task includes generating visual information, visual materials are obtained based on the information of the target exhibit in the guide knowledge base; A lightweight monocular depth estimation network is used to generate a relative depth map of the visualization material. Based on the virtual imaging surface distance of the mixed reality device and the human interpupillary distance, the parallax of the relative depth map is calculated, a relative depth map with parallax is generated, and the relative depth map with parallax is rendered on the off-axis freeform surface module of the mixed reality device. When the target task includes generating audible information, audible materials are obtained based on the information of the target exhibit in the guide knowledge base; Based on the audible material, narration audio is generated and loaded onto the audio module of the mixed reality device.

[0081] The mixed reality tour guide system provided by this invention utilizes the semantic recognition capability of a large visual language model to replace physical space coordinate comparison, enabling highly accurate map viewing and explanation functions in complex environments.

[0082] This invention also provides a mixed reality tour guide device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a mixed reality tour guide method as described above. The working principles and beneficial effects of the two are one-to-one, so they will not be described in detail here.

[0083] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0084] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A mixed reality guided tour method, characterized in that, include: Based on the location of the mixed reality device, calculate the user's approximate location and load the meta-features of each candidate exhibit within the area where the user's approximate location is located; Capture images from the camera of the mixed reality device and identify the core targets in the user's current field of view; In response to a user's voice command, the approximate location of the user, the core target, and the user's voice command are input into a pre-trained visual language model to extract real-time feature vectors. The visual language model uses the user's approximate location as a forced constraint to generate a space for real-time feature vectors; Calculate the similarity between the real-time feature vector and the meta-feature, and determine the target exhibit based on the similarity. The target task is extracted from the user's voice command, and a task execution strategy for the target task is generated and executed based on the information of the target exhibit in the guide knowledge base.

2. The mixed reality guided tour method as described in claim 1, characterized in that, The step of calculating the user's approximate location based on the mixed reality device's positioning and loading the meta-features of each candidate exhibit within the area where the user's approximate location is located includes: The approximate location of the user is calculated based on the Bluetooth signal strength received by the mixed reality device; Using the user's approximate location as the center, determine the area where the user's approximate location is located according to a preset area radius; All exhibits within the region are selected from the local knowledge base as candidate exhibits, and the meta-features of each candidate exhibit are obtained.

3. The mixed reality guided tour method as described in claim 1, characterized in that, The acquisition of camera images from the mixed reality device, and the identification of key targets in the user's current field of vision, includes: Capture camera images from mixed reality devices; A lightweight object detection algorithm is used to extract bounding boxes that represent exhibits from the camera images. The detection box is deduplicated to obtain the core target in the user's current field of view.

4. The mixed reality guided tour method as described in claim 1, characterized in that, The visual language model includes a visual encoder, a position encoder, a speech encoder, and a cross-modal attention module. The method for constructing the visual language model includes: A position encoder is added to a pre-trained general visual language model to inject position priors; the general visual language model includes a visual encoder, a speech encoder, and a cross-modal attention module.

5. A mixed reality guided tour method as described in claim 4, characterized in that, The training methods for the visual language model include: Obtain the triplet dataset for the current guided tour scenario; the triplet dataset includes exhibit images, exhibit coordinates, and expert commentary. Freeze the visual encoder and the speech encoder, and add a low-rank adaptive bypass to the cross-modal attention module; Using the triplet dataset, the visual language model with the visual encoder and the speech encoder frozen is trained, and the parameters of the position encoder and the low-rank adaptive bypass are updated to obtain the trained visual language model.

6. A mixed reality guided tour method as described in claim 4, characterized in that, The step of responding to a user's voice command by inputting the user's approximate location, the core target, and the user's voice command into a pre-trained visual language model to extract real-time feature vectors includes: In response to a user's voice command, the user's approximate location, the core target, and the user's voice command are input into a pre-trained visual language model; The location encoder extracts the position vector of the user's approximate location, the visual encoder extracts the image features of the core target, and the voice encoder extracts the voice command features of the user's voice command. The location vector, image features, and voice command features are fused using a cross-modal attention module to obtain a real-time feature vector.

7. A mixed reality guided tour method as described in claim 1, characterized in that, The step of calculating the similarity between the real-time feature vector and the meta-feature, and determining the target exhibit based on the similarity, includes: Calculate the cosine similarity between the real-time feature vector and the meta-features of each candidate exhibit; The cosine similarity scores are sorted, and if the highest cosine similarity score is greater than a preset threshold, the corresponding candidate exhibit is determined as the target exhibit.

8. A mixed reality guided tour method as described in claim 1, characterized in that, The step of extracting the target task from the user's voice command and generating and executing a task execution strategy for the target task based on the information of the target exhibit in the guide knowledge base includes: Extract the target task from the user's voice command; the target task includes at least one of generating visual information and generating audible information. When the target task includes generating visual information, visual materials are obtained based on the information of the target exhibit in the guide knowledge base; A lightweight monocular depth estimation network is used to generate a relative depth map of the visualization material. Based on the virtual imaging surface distance of the mixed reality device and the human interpupillary distance, the parallax of the relative depth map is calculated, a relative depth map with parallax is generated, and the relative depth map with parallax is rendered on the off-axis freeform surface module of the mixed reality device. When the target task includes generating audible information, audible materials are obtained based on the information of the target exhibit in the guide knowledge base; Based on the audible material, narration audio is generated and loaded onto the audio module of the mixed reality device.

9. A mixed reality tour guide system, characterized in that, include: The meta-feature acquisition module is used to calculate the user's approximate location based on the positioning of the mixed reality device, and load the meta-features of each candidate exhibit within the area where the user's approximate location is located. The core target recognition module is used to acquire camera images from the mixed reality device and identify the core targets in the user's current field of view; The real-time feature vector extraction module is used to respond to user voice commands by inputting the user's approximate location, the core target, and the user's voice command into a pre-trained visual language model to extract real-time feature vectors. The visual language model uses the user's approximate location as a forced constraint to generate a space for real-time feature vectors; The target exhibit determination module is used to calculate the similarity between the real-time feature vector and the meta-feature, and determine the target exhibit based on the similarity. The strategy generation module is used to extract the target task from the user's voice command and generate a task execution strategy for the target task based on the information of the target exhibit in the guide knowledge base.

10. A mixed reality guided tour device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the mixed reality navigation method as described in any one of claims 1 to 8.

Images