Intelligent method and system for realizing cultural and travel space based on MR technology

Abstract

The present application relates to the field of virtual reality technology, in particular to a method and system for realizing intelligent travel space by fusing MR technology. The method performs spatial three-dimensional reconstruction on a travel space to establish a spatial semantic graph, and assigns an independent narrative perspective identifier and a virtual content rendering layer to each tourist. The interaction actions of the tourists are detected and decomposed into physical actions and content triggering events. The physical actions are broadcast to other tourists without virtual content information, and the content triggering events only push corresponding virtual content to the rendering layer of the triggering tourist. For other tourists receiving the physical action broadcast, it is determined whether there is virtual content in the interaction position under the narrative perspective of the tourist. If not, substitute virtual content is generated and presented, and the presentation duration is bound to the interaction duration. The present application realizes differentiated narrative presentation and cross-perspective semantic coherent presentation of multiple users in the same physical space.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, specifically to a method and system for intelligent realization of cultural and tourism spaces that integrates MR technology. Background Technology

[0002] Virtual reality and augmented reality technologies provide users with immersive interactive experiences by integrating virtual content with real-world scenes. Mixed reality technology, as a further fusion of the two, requires virtual content to have precise spatial anchoring capabilities within the real physical space. In cultural and tourism scenarios, mixed reality technology is used to overlay the explanation, restoration, and narrative content of cultural landmarks onto real-world scenes in a virtual form, enabling multiple tourists to have an immersive cultural experience within the same physical space.

[0003] The foundation for achieving spatial fusion in mixed reality lies in the 3D reconstruction of real-world scenes and the establishment of pose estimation capabilities. Existing solutions typically employ a visual-inertial fusion positioning method combining visual sensors and inertial measurement units (IMUs), with some solutions further introducing depth sensors to improve reconstruction accuracy. Regarding multi-user presentation, existing solutions mostly adopt a unified virtual content presentation mechanism, meaning that all users in the same physical scene receive the same virtual content.

[0004] The existing technology has the following shortcomings: First, visual-inertial fusion positioning accumulates drift errors after long-term continuous operation. Existing correction methods rely on loop closure detection, which requires the identification of previously visited scenes before it can be triggered. The correction is lagging and computationally expensive, resulting in positional shifts between virtual content and real cultural landmarks. Second, existing multi-user MR solutions lack independent content presentation mechanisms for different narrative perspectives. Virtual content is visible to each other or completely identical among tourists, making it difficult to meet the needs of differentiated narratives. Third, when tourists interact across perspectives, existing technology does not provide a semantic alignment compensation mechanism for missing virtual content, resulting in some users not receiving a coherent presentation of virtual content at the location of the interaction. Summary of the Invention

[0005] This invention provides a method and system for intelligent realization of cultural and tourism spaces that integrates MR technology, solving the problems of poor independence of virtual content presentation, lack of cross-view content, and virtual-real misalignment caused by pose drift in multi-user MR cultural and tourism scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a method for intelligent realization of cultural and tourism spaces integrating MR technology, comprising: S1: Perform spatial 3D reconstruction of cultural and tourism venues and establish spatial semantic maps; assign narrative perspective identifiers to multiple tourists in the cultural and tourism venues, with each narrative perspective identifier corresponding to an independent virtual content rendering layer; S2: Detect any visitor's interactive action, decompose the interactive action into physical actions and content triggering events; broadcast the physical actions to other visitors in the cultural and tourism venue, the physical actions do not contain virtual content information; based on the content triggering events and the narrative perspective identifier held by the visitor who triggered the interactive action, push the corresponding virtual content only to the visitor's rendering layer; S3: For other tourists who receive the broadcast of the physical action, determine whether there is virtual content at the location where the interactive action occurs from the narrative perspective of the other tourists based on the spatial semantic graph: if it exists, then present the virtual content in the rendering layer of the other tourists; if it does not exist, then generate alternative virtual content at the location where the interactive action occurs according to the preset cross-perspective semantic alignment relationship and present it in the rendering layer of the other tourists, and the presentation duration of the alternative virtual content is bound to the duration of the interactive action.

[0007] As a preferred embodiment of the present invention, in S1, the specific steps of the spatial three-dimensional reconstruction include: Visual sensors, inertial measurement units, and depth sensors are used to collect data from the cultural and tourism sites. Under the factor graph optimization framework, a joint factor graph is constructed with the reprojection error of the visual sensor, the pre-integration constraint of the inertial measurement unit, and the depth consistency residual of the depth sensor as factor nodes. The six-degree-of-freedom pose of the rigid body containing the visual sensor, the inertial measurement unit, and the depth sensor is solved by nonlinear least squares optimization. Based on the solution results, a three-dimensional point cloud map of the cultural and tourism venue is constructed, and the three-dimensional coordinates and semantic categories of cultural landmarks are marked on the three-dimensional point cloud map to obtain the spatial semantic map.

[0008] As a preferred embodiment of the present invention, in S1, the establishment of the spatial semantic map further includes a drift correction step, which includes: Based on historical tourist route data, predict the next cultural landmark that tourists will arrive at; Before tourists arrive at the cultural landmark, a visual feature matching model of the cultural landmark is preloaded. The visual feature matching model stores the local feature descriptors of the cultural landmark and their corresponding three-dimensional coordinates. When the distance between the tourist and the cultural landmark is less than the preset correction trigger distance, and the visual feature matching confidence is higher than the preset confidence threshold, the local feature descriptor of the cultural landmark detected in the current frame image is solved by PnP with the corresponding three-dimensional coordinates stored in the visual feature matching model to obtain the reference pose value. The current pose estimate and the reference pose value are then weighted and fused according to the visual feature matching confidence to obtain the pose correction value, and the pose correction value is used as the initial state for subsequent pose estimation.

[0009] As a preferred technical solution of the present invention, in S1, the step of allocating the narrative perspective identifier includes: Obtain tourist profiles, which include age groups, cultural preference tags, and historical visit records; Based on the tourist profile, a recommended narrative perspective is matched and used to the tourist. The number of online tourists for each narrative perspective within the cultural and tourism venue is counted. When the number of online tourists for a recommended narrative perspective exceeds a preset limit, the narrative perspective with the fewest online tourists is assigned to the tourist.

[0010] As a preferred technical solution of the present invention, in S2, the detection of the interactive action includes input detection of three modalities: eye tracking, gesture recognition, and voice command. When the three modalities are triggered simultaneously, the valid input is determined according to the priority order of gesture recognition, voice command, and eye tracking, and the input of the modality with the highest priority is taken as the interaction action.

[0011] As a preferred embodiment of the present invention, in S2, the content triggering rules of the virtual content rendering layer are stored in a multi-view content package, the multi-view content package including: The public foundation layer stores scene background content shared by all narrative perspectives; The perspective content layer, each narrative perspective identifier corresponds to an independent perspective content layer, which stores the virtual items and narrative information unique to the narrative perspective identifier; A cross-perspective semantic alignment mapping table uses the combination of narrative perspective identifiers and interaction positions as the index key to record the mapping relationship between the interaction positions under each narrative perspective identifier and the corresponding alternative virtual content under other narrative perspective identifiers. The rendering layer binding relationship table records the binding relationship between each narrative perspective identifier and the corresponding independent virtual content rendering layer. The content of the common basic layer and perspective content layer is routed to the rendering layer of the corresponding visitor for presentation through the rendering layer binding relationship table.

[0012] As a preferred technical solution of the present invention, in S3, the specific steps for querying and presenting the mapping relationship of the alternative virtual content include: In the cross-perspective semantic alignment mapping table, the alternative virtual content corresponding to the narrative perspective identifier held by the tourist who triggered the interactive action and the location where the interactive action occurred is queried using the index. The retrieved alternative virtual content is anchored to the location where the interactive action occurs in the spatial semantic map; The alternative virtual content is pushed to the rendering layer of the other tourists. The presentation duration of the alternative virtual content is bound to the duration of the interaction action. After the interaction action ends, the alternative virtual content is removed from the rendering layer of the other tourists. If there is no mapping relationship corresponding to the index in the cross-view semantic alignment mapping table, then based on the narrative perspective identifier held by the tourist who triggered the interaction and the location where the interaction occurred, combined with the narrative perspective identifier held by other tourists who received the physical action broadcast, alternative virtual content corresponding to the semantic location of the location of occurrence is dynamically generated, and the alternative virtual content is anchored to the location where the interaction occurred and then pushed to the rendering layer of the other tourists.

[0013] As a preferred embodiment of the present invention, step S3 further includes a multi-user collaborative narrative triggering step, which includes: Synchronize the narrative progress nodes of each tourist to the collaborative status server in real time; When the narrative progress nodes of multiple tourists holding different narrative perspective identifiers simultaneously reach the preset collaborative trigger node, collaborative narrative content corresponding to their respective narrative perspective identifiers is pushed to the rendering layers of the multiple tourists. The collaborative narrative content unfolds synchronously in time, and each tourist only receives the collaborative narrative content corresponding to the narrative perspective identifier held by that tourist.

[0014] As a preferred embodiment of the present invention, it further includes a cloud-edge collaborative resource scheduling step, wherein the cloud-edge collaborative resource scheduling step includes: Based on ticket reservation data and historical visitor flow data, the cloud platform predicts the number of online visitors in the cultural and tourism venues in future time periods, and generates resource allocation instructions accordingly, which are then sent to edge computing nodes in advance. The edge computing node is configured with computing resources according to the resource allocation instructions, and is used to perform real-time broadcasting of the physical actions and calculation of the generation of the alternative virtual content, and to count the number of online visitors in real time. When the number of online visitors counted by the edge computing node exceeds a preset degradation threshold, the rendering layer content is graded according to the order of priority for core narrative content and degradation for decorative content.

[0015] This invention also proposes an intelligent realization system for cultural and tourism spaces integrating MR technology, comprising: Spatial mapping and allocation module: performs spatial 3D reconstruction of cultural and tourism venues and establishes spatial semantic maps; assigns narrative perspective identifiers to multiple tourists in the cultural and tourism venues, with each narrative perspective identifier corresponding to an independent virtual content rendering layer; Action decomposition and push module: detects any tourist's interactive action, decomposes the interactive action into physical actions and content triggering events; broadcasts the physical actions to other tourists in the cultural and tourism venue, the physical actions do not contain virtual content information; based on the content triggering events and the narrative perspective identifier held by the tourist who triggered the interactive action, pushes the corresponding virtual content only to the tourist's rendering layer; Cross-perspective presentation module: For other tourists who receive the broadcast of the physical action, the module determines whether there is virtual content at the location where the interactive action occurs from the narrative perspective of the other tourists based on the spatial semantic graph. If there is, the virtual content is presented in the rendering layer of the other tourists. If there is no virtual content, alternative virtual content is generated at the location where the interactive action occurs according to the preset cross-perspective semantic alignment relationship and presented in the rendering layer of the other tourists. The presentation duration of the alternative virtual content is bound to the duration of the interactive action.

[0016] The beneficial effects of this invention are: 1. This invention decomposes interactive actions into physical actions and content-triggered events, and combines independent narrative perspective markers with a virtual content rendering layer to achieve a differentiated narrative experience where multiple tourists can perceive physical actions within the same physical space while the virtual content does not interfere with each other. This solution breaks through the limitations of existing multi-user MR solutions that uniformly present virtual content, allowing tourists with different cultural preferences to simultaneously obtain their own matched immersive narrative content.

[0017] 2. This invention utilizes a cross-perspective semantic alignment mapping table and a dynamic generation mechanism to enable visitors who do not possess the corresponding virtual content to obtain semantically corresponding alternative virtual content at the interaction location, and avoids content stagnation by binding the presentation duration to the interaction duration. This mechanism ensures the coherence and consistency of virtual content presentation in cross-narrative perspective interaction scenarios, and enhances the integrity of the multi-user collaborative experience.

[0018] 3. This invention preloads a visual feature matching model of cultural landmarks by predicting tourist routes, and adaptively weights and fuses the baseline pose value with the current pose estimate by combining the confidence level of the visual feature matching. This achieves real-time correction of pose drift without relying on loop closure detection. This solution effectively solves the problem of misalignment between virtual and real content caused by long-term operation, while ensuring the consistency of spatial coordinate systems among multiple users. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the intelligent realization method for cultural and tourism spaces that integrates MR technology according to the present invention. Figure 2 This is a schematic diagram of the structure of the intelligent realization system for cultural and tourism spaces that integrates MR technology according to the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Example 1: As Figure 1 As shown, the present invention provides a method for intelligent realization of cultural and tourism spaces integrating MR technology, comprising: S1: Perform spatial 3D reconstruction of cultural and tourism venues and establish spatial semantic maps; assign narrative perspective identifiers to multiple tourists in the cultural and tourism venues, with each narrative perspective identifier corresponding to an independent virtual content rendering layer; Specifically, spatial 3D reconstruction is used to obtain geometric structural information of cultural and tourism sites and semantic information of cultural landmarks, while the allocation of narrative perspective markers is used to establish independent virtual content presentation channels for different tourists. Together, they form the basis for the precise push and cross-perspective presentation of virtual content in subsequent S2 and S3.

[0022] Furthermore, the specific steps of the spatial three-dimensional reconstruction include: Visual sensors, inertial measurement units (IMUs), and depth sensors are used to collect data from cultural and tourism sites. Visual sensors acquire image sequences of the sites, IMUs collect angular velocity and acceleration data during rigid body motion, and depth sensors collect depth information of the scene. The three types of sensors are either fixedly mounted on the same rigid body support to collect data synchronously, or deployed separately and aligned using timestamps.

[0023] Within the factor graph optimization framework, the reprojection error factor of the visual sensor is used respectively. Pre-integration constraint factor of inertial measurement unit and depth consistency residual factor of depth sensor A joint factor graph is constructed, and the six-degree-of-freedom pose T=[R,t] of the rigid body containing the vision sensor, inertial measurement unit and depth sensor is solved by nonlinear least squares optimization. The six-degree-of-freedom pose includes three-dimensional position and three-dimensional attitude angle.

[0024] It represents the deviation between the observed position of the same 3D spatial point on the pixel plane between adjacent image frames and the predicted position obtained by reprojection based on the current pose estimate; This indicates the deviation between the relative motion between adjacent poses, calculated by pre-integration of the angular velocity and acceleration data output by the inertial measurement unit between adjacent image frames, and the pose estimate. This represents the deviation between the scene depth value measured by the depth sensor and the predicted depth value obtained by back-projection from the current pose estimate and the 3D point cloud map. To obtain the six-degree-of-freedom pose, Let be the rotational component in the pose. The translation component in the pose.

[0025] It should be noted that the specific solution method for nonlinear least squares optimization can be implemented using conventional graph optimization methods in this field. This embodiment does not limit the specific solution algorithm. This step integrates the error factors of the three heterogeneous sensors—vision, inertial, and depth—into the same factor graph for joint solution, so that the observation information of the three types of sensors mutually constrains and compensates for each other. Compared with using only a single sensor for pose estimation, it can obtain more accurate and stable pose estimation results.

[0026] Based on the obtained six-degree-of-freedom pose By combining image data and depth data collected by sensors, a three-dimensional point cloud map of cultural and tourism venues is constructed. The three-dimensional coordinates and semantic categories of cultural landmarks within the cultural and tourism venues are marked on the three-dimensional point cloud map. The cultural landmarks are physical objects within the cultural and tourism venues that have cultural display value, and the semantic categories are the cultural themes or exhibit type labels to which the cultural landmarks belong, thereby obtaining a spatial semantic map.

[0027] Furthermore, the establishment of the spatial semantic map also includes a drift correction step, which includes: Because visual-inertial fusion localization accumulates errors when continuously solving poses over long periods and large areas, without correction, positional shifts will occur between virtual content and cultural landmarks in the real scene, affecting the presentation of virtual-real fusion. Relying on the system's own loop closure detection for error correction requires identifying scenes previously visited by tourists, resulting in a lag in correction. Furthermore, loop closure detection has significant computational overhead, making real-time performance difficult to guarantee on edge computing nodes with a large number of tourists. This embodiment leverages the predictable patterns of tourist routes within cultural and tourism venues, preparing the necessary correction models before tourists reach cultural landmarks, thereby reducing online computation latency during correction.

[0028] The next cultural landmark to be visited by the current tourist is predicted based on historical tourist route data, which is statistical data on the actual movement trajectory of tourists in cultural and tourism venues in the past.

[0029] Before tourists arrive at the cultural landmark, a visual feature matching model for the landmark is preloaded. This model stores local feature descriptors and their corresponding 3D coordinates. The local feature descriptors are key point feature descriptors extracted from the image of the cultural landmark. This preloading method completes the model reading and initialization process before tourists arrive at the correction area, avoiding the delay caused by real-time retrieval of the matching model after tourists arrive at the cultural landmark, and ensuring the real-time performance of drift correction.

[0030] When the distance between the tourist and the cultural landmark is less than the preset correction trigger distance And the confidence level of visual feature matching Higher than the preset reliability threshold At that time, drift correction is triggered. A pre-set distance value is used to define the range of the correction area; A pre-set confidence level value is used to determine whether the matching result is reliable.

[0031] It should be noted that, and The triggering timing for controlling drift correction can be configured according to the spatial scale of the cultural and tourism venue and the accuracy of the sensor. This embodiment does not limit its specific value.

[0032] The calculation of the visual feature matching confidence score c includes the following steps: extracting local feature descriptors from the current frame image; performing nearest neighbor matching between the local feature descriptors and the local feature descriptors stored in the visual feature matching model; and filtering matching pairs through the nearest neighbor distance ratio test to obtain a set of filtered matching pairs; and applying random sampling consistency constraints to the set of filtered matching pairs. Solve to obtain the number of interior points. and the average reprojection error of interior points Based on the number of interior points Total number of matching pairs in the filtered matching pair set The ratio, and the average reprojection error of the interior points. Calculate the confidence score of visual feature matching The confidence level of the visual feature matching is positively correlated with the ratio and negatively correlated with the average reprojection error of the inliers.

[0033] In one implementation, visual feature matching confidence level Calculate as follows: ; In the formula, Indicates the confidence level of visual feature matching. Indicates the number of interior points. This represents the total number of matching pairs in the filtered set. This represents the average reprojection error of the interior points. and Represents the weight coefficients and satisfies , This represents the scaling factor, used to adjust the degree of influence of the average reprojection error of inliers on the confidence level of visual feature matching.

[0034] It should be noted that, and The principle for selecting the value is: if the feature points of cultural landmarks within a cultural and tourism venue are densely distributed and sufficient matching pairs are easily obtained, then the value can be appropriately increased. The proportion of values ​​makes the influence of the number of matches more prominent; if the geometric structure of the cultural landmark is complex and the accuracy requirements for reprojection error are high, then the value can be appropriately increased. The proportion of values ​​for makes the impact of reprojection error more prominent; The value is set based on the actual distribution range of the average reprojection error of the interior points, so that... It can present a distinguishable gradient of change within the error distribution range, avoiding If the value is too large or too small, the value of this item will tend to be constant. If the total number of matching pairs is lower than the preset minimum matching number threshold, the confidence level of visual feature matching is determined to be 0.

[0035] After drift correction is triggered, the local feature descriptors of cultural landmarks detected in the current frame image are matched with the corresponding three-dimensional coordinates stored in the visual feature matching model using a visual feature matching model. Solve to obtain the reference pose value. .

[0036] The current pose estimate Compared with the reference pose value Matching confidence based on visual features Weighted fusion is performed to obtain pose correction values. The fusion method is as follows: ; In the formula, This represents the pose correction value. Represents the reference pose value, by The solution is obtained as follows: This represents the current pose estimate, output by factor graph optimization. This represents the weighting coefficient, which is determined by the confidence level of visual feature matching.

[0037] It should be noted that the weighted fusion performs pose interpolation in the Lie algebra space to ensure that the fusion result satisfies the manifold constraint of the rigid body pose, and avoids destroying the orthogonality or normalization constraint by directly applying linear weights to the rotation matrix or quaternion.

[0038] Weighting coefficients Matching confidence based on visual features To avoid weighting factors A sudden change near a preset confidence threshold causes a jump in pose correction, affecting the weighting coefficients. Determine as follows: ; In the formula, Indicates the weighting coefficient. Indicates the confidence level of visual feature matching. This indicates a pre-set confidence threshold. This represents the smoothing coefficient, used to adjust the weighting coefficients. The confidence level is matched with visual features around a preset confidence threshold. The rate of change.

[0039] It should be noted that the smoothing coefficient The weighting coefficient can be set according to the distribution of visual feature matching confidence in the actual scene. This embodiment does not limit its specific value. The weighting coefficient is determined in the above way so that the weighting coefficient changes continuously and smoothly with the visual feature matching confidence, avoiding sudden changes in pose correction results due to small fluctuations in visual feature matching confidence near the threshold.

[0040] The obtained pose correction value As the initial state for subsequent pose estimation, this system enables periodic correction of accumulated drift errors during the construction of the spatial semantic graph. Since multiple tourists rely on consistent broadcasting of physical actions to achieve spatial consistency during interaction, using the pose correction value as the initial state for subsequent pose estimation for each tourist ensures that pose estimation among different tourists remains consistent within the same spatial coordinate system, avoiding misalignment of the presentation position of alternative virtual content due to inconsistent positioning drift among tourists.

[0041] Furthermore, the assignment step of the narrative perspective identifier includes: A tourist profile is obtained, which includes age group, cultural preference tags, and historical visit records. The cultural preference tags are identifying information that represents the degree of tourist preference for historical, artistic, or popular science cultural content.

[0042] Based on the tourist profile, a recommended narrative perspective is matched for the tourist. That is, according to the matching rules between the tourist profile and each narrative perspective, the narrative perspective that matches the tourist profile is determined as the recommended narrative perspective.

[0043] Statistics on the number of online visitors under various narrative perspectives within cultural and tourism venues. When the number of online visitors corresponding to the recommended narrative perspective exceeds the preset limit. At that time, the narrative perspective identifier with the fewest online visitors is assigned to the tourists. For the first Each narrative perspective identifies the number of online visitors. The pre-set upper limit for the number of online visitors is used to control the concurrent load on the virtual content rendering layer corresponding to a single narrative perspective.

[0044] It should be noted that, The settings can be adjusted according to the carrying capacity of the virtual content rendering layer; however, this embodiment does not limit the specific values.

[0045] Once each narrative perspective identifier is assigned, a virtual content rendering layer is created for the corresponding tourist that uniquely corresponds to that narrative perspective identifier. This layer is used for the independent rendering and presentation of subsequent virtual content, ensuring that the virtual content received by tourists under different narrative perspective identifiers in the same physical location is independent of each other.

[0046] S2: Detect any visitor's interactive action, decompose the interactive action into physical actions and content triggering events; broadcast the physical actions to other visitors in the cultural and tourism venue, the physical actions do not contain virtual content information; based on the content triggering events and the narrative perspective identifier held by the visitor who triggered the interactive action, push the corresponding virtual content only to the visitor's rendering layer; Specifically, physical actions refer to changes in limbs or position that can be directly observed by other tourists within the same physical space during tourist interactions. Examples include tourist movement trajectories and changes in body posture, which do not carry narrative content. Content-triggered events are event information generated during tourist interactions that trigger the presentation of virtual content matching their narrative perspective. Physical actions are broadcast to other tourists within the cultural and tourism venue, enabling them to perceive the tourist's actual actions within the physical space. Content-triggered events are only used to present corresponding virtual content within the rendering layer of the tourist who triggered the interaction, thus achieving the effect that physical actions are visible to multiple tourists within the same physical space, while virtual content remains independent.

[0047] Furthermore, the detection of the interactive action includes input detection of three modalities: eye tracking, gesture recognition, and voice command. When the three modalities are triggered simultaneously, the valid input is determined according to the priority order of gesture recognition, voice command, and eye tracking, and the input of the modality with the highest priority is taken as the interactive action.

[0048] It should be noted that gesture recognition corresponds to actions that tourists actively and deliberately perform, with high intent and a low probability of false triggering, hence it has the highest priority. Voice commands are also active interactions, but the presence of background narration and ambient noise in cultural and tourist venues significantly affects recognition accuracy, thus its priority is second highest. Eye tracking primarily reflects tourists' visual attention, but it has the uncertainty of distinguishing between a tourist's gaze out of interest and a desire to trigger interaction, hence its lowest priority. Eye tracking input is only used as an interaction action when gesture recognition and voice commands fail to trigger valid input. When multiple modalities are triggered simultaneously, only the modal input with the highest priority and a recognition confidence level above a preset threshold is considered a valid interaction action; other modal inputs are not considered valid inputs.

[0049] Furthermore, the content triggering rules of the virtual content rendering layer are stored in a multi-view content package, which includes a common base layer, a view content layer, a cross-view semantic alignment mapping table, and a rendering layer binding relationship table.

[0050] The public base layer stores scene background content shared by all narrative perspectives. This scene background content is virtual content presented by all visitors without distinguishing narrative perspective identifiers.

[0051] Each narrative perspective identifier in the perspective content layer corresponds to an independent perspective content layer, which stores virtual items and narrative information exclusive to that narrative perspective identifier. The virtual items and narrative information are virtual content that can only be seen by tourists who hold the corresponding narrative perspective identifier.

[0052] Cross-perspective semantic alignment map with narrative perspective identifiers and interaction locations The composite index key forms an index that records the mapping relationship between the interactive positions under each narrative perspective identifier and the corresponding alternative virtual content under other narrative perspective identifiers. The interactive positions... This represents the location where the interactive action occurs in the spatial semantic map. Specifically, the cross-perspective semantic alignment mapping table stores the mapping relationship as follows: based on the narrative perspective identifier held by the visitor who triggered the interactive action. and interactive location The composite index key is used as the key to identify the narrative perspective held by other tourists who receive physical action broadcasts. The corresponding alternative virtual content resource identifier and presentation parameters are values.

[0053] The rendering layer binding table records the binding relationship between each narrative perspective identifier and the corresponding independent virtual content rendering layer. The content of the common base layer and perspective content layer is routed to the corresponding visitor's rendering layer for presentation through the rendering layer binding table.

[0054] It should be noted that the layered storage method of the public base layer and the perspective content layer allows the scene background content to be reused across all narrative perspectives without having to repeatedly store the same background content for each narrative perspective, thereby reducing content storage redundancy in multi-perspective scenarios.

[0055] After an interactive action is triggered, based on the content triggering event and the narrative perspective identifier P held by the tourist who triggered the interactive action, the virtual content that matches the content triggering event is queried in the perspective content layer corresponding to the tourist, and the queried virtual content is pushed to the tourist's rendering layer for presentation via the rendering layer binding relationship table.

[0056] S3: For other tourists who receive the broadcast of the physical action, determine whether there is virtual content at the location where the interactive action occurs from the narrative perspective of the other tourists based on the spatial semantic graph: if it exists, then present the virtual content in the rendering layer of the other tourists; if it does not exist, then generate alternative virtual content at the location where the interactive action occurs according to the preset cross-perspective semantic alignment relationship and present it in the rendering layer of the other tourists, and the presentation duration of the alternative virtual content is bound to the duration of the interactive action.

[0057] Specifically, after receiving the physical action broadcast, other tourists will interact based on their location. Search the spatial semantic map for the location's narrative perspective markers held by other tourists. Does the corresponding view content layer contain configured virtual content?

[0058] If the location is found to contain a narrative perspective identifier Matching virtual content is then directly presented in the rendering layer of other visitors, allowing them to perceive the same virtual content scene as the visitor who triggered the interaction.

[0059] If the location is found in the narrative perspective marker If the configured virtual content does not exist in the corresponding view content layer, then the query and rendering of alternative virtual content will be performed. The specific steps include: In the cross-perspective semantic alignment map, the narrative perspective identifier P of the tourist who triggered the interaction and the interaction location are used. A composite index key is used to query the narrative perspective identifiers of other tourists who received the physical action broadcast. Corresponding alternative virtual content .

[0060] If the query is successful, meaning the cross-view semantic alignment mapping table already stores the mapping relationship corresponding to the composite index key, then the alternative virtual content corresponding to that mapping relationship is directly retrieved. .

[0061] If the query fails to find a match, meaning there is no mapping relationship corresponding to the composite index key in the cross-perspective semantic alignment mapping table, then alternative virtual content is dynamically generated according to the preset cross-perspective semantic alignment rules. The cross-perspective semantic alignment rule is based on the semantic tags of the virtual content triggered by the interactive action and the narrative perspective identifiers held by other visitors. The corresponding topic tags are determined to be semantically corresponding to the semantic tags and conform to the narrative perspective identifier. Alternative virtual content with corresponding theme style .

[0062] It should be noted that dynamic generation can be achieved using conventional template matching methods or generative network models in this field, and this embodiment does not limit the specific generation method.

[0063] Replacement virtual content obtained from query hits or dynamic generation Anchored to interactive locations in the spatial semantic graph The anchoring refers to binding the alternative virtual content to the interactive position in spatial coordinates, so that the alternative virtual content is presented at the corresponding position.

[0064] Will replace virtual content The alternative virtual content is pushed to the rendering layer of other tourists. The presentation duration of the alternative virtual content is bound to the duration of the interactive action. After the interactive action ends, the alternative virtual content is removed from the rendering layer of other tourists.

[0065] It should be noted that the presentation duration is bound to the duration, meaning that the start and end times of the presentation of the alternative virtual content in other visitors' rendering layers are consistent with the occurrence and end times of the interactive action, so that the alternative virtual content is only presented during the period when the visitor who triggered the interactive action actually performs the interactive action, and avoids the alternative virtual content remaining in other visitors' rendering layers after the interactive action ends.

[0066] Furthermore, S3 also includes a multi-user collaborative narrative triggering step, which includes: The narrative progress node of each tourist is synchronized to the collaborative state server in real time. The narrative progress node is the identification information that represents the current stage of the narrative process corresponding to the tourist's narrative perspective.

[0067] When the narrative progress nodes of multiple tourists holding different narrative perspective identifiers simultaneously reach the preset collaborative trigger node, collaborative narrative content corresponding to their respective narrative perspective identifiers is pushed to the rendering layers of the multiple tourists. The preset collaborative trigger node is a pre-set narrative progress node used to trigger collaborative narrative among multiple tourists.

[0068] The collaborative narrative content unfolds synchronously in time. Each tourist only receives the collaborative narrative content corresponding to the narrative perspective identifier held by that tourist, enabling multiple tourists holding different narrative perspective identifiers to participate in the same collaborative narrative event based on their respective narrative perspectives within the same time period, while only perceiving the collaborative narrative content that matches their own narrative perspective identifier.

[0069] It should be noted that the preset collaborative trigger node can be configured according to the design requirements of the narrative content, and this embodiment does not limit its specific value.

[0070] Furthermore, it also includes a cloud-edge collaborative resource scheduling step, which includes: Based on ticket reservation data and historical visitor flow data, the cloud platform predicts the number of online visitors in cultural and tourism venues during future time periods and generates resource allocation instructions accordingly, which are then sent to edge computing nodes in advance. The ticket reservation data refers to the ticket reservation information completed by visitors before entering the cultural and tourism venues, and the historical visitor flow data refers to the statistical data on the number of online visitors in the cultural and tourism venues during previous time periods.

[0071] Edge computing nodes are configured with computing resources according to resource allocation instructions. These resources are used to perform real-time broadcasting of physical actions and computational generation of alternative virtual content, and to count the number of online visitors in real time. . This is the number of online tourists currently in cultural and tourism venues, which is obtained in real time from edge computing nodes.

[0072] It should be noted that, This refers to the real-time online visitor count statistics for cultural and tourism venues as a whole, compared with the aforementioned... The number of online visitors corresponding to each narrative perspective. Different statistical dimensions The overall statistical results of the number of online visitors from various narrative perspectives are used to determine the overall computing power configuration and degradation processing of edge computing nodes.

[0073] When the edge computing node counts the current number of online visitors Exceeding the preset downgrade threshold At that time, the content of the rendering layer is classified and processed in order of priority for core narrative content and demotion for decorative content. A pre-defined threshold for the number of online visitors is used to determine whether the content of the rendered layer needs to be downgraded.

[0074] Specifically, each virtual asset is pre-labeled with a content level tag in the resource description file of the multi-view content package. These content level tags include core narrative content tags and decorative content tags. Virtual assets labeled with core narrative content tags are virtual elements that convey the main narrative thread, key interaction results, or cultural knowledge points. Virtual assets labeled with decorative content tags are virtual elements that only serve to enhance the scene's atmosphere and do not affect the narrative process. When the edge computing node loads the multi-view content package, it synchronizes the content level tags of each virtual asset to the resource manager of the rendering pipeline.

[0075] When the hierarchical processing is triggered, the content level tags of each virtual asset in the rendering queue of the rendering layer are traversed: for virtual assets marked with the core narrative content tag, the original rendering accuracy and refresh rate are maintained; for virtual assets marked with the decorative content tag, a degradation process is performed, which includes one or more of the following methods: reducing the number of particle effects emitted, replacing distant virtual assets with simplified models, etc.

[0076] It should be noted that, The specific degradation range of the tiered processing can be set according to the computing power configuration of the edge computing nodes. This embodiment does not limit its specific value.

[0077] The intelligent implementation method for cultural and tourism spaces that integrates MR technology provided in this embodiment achieves differentiated narrative presentation and cross-perspective semantic coherence presentation for multiple tourists in the same physical space through the above-mentioned S1 to S3 and their preferred steps.

[0078] Example 2: like Figure 2 As shown, this embodiment provides an intelligent realization system for cultural and tourism spaces that integrates MR technology, including: Spatial mapping and allocation module: performs spatial 3D reconstruction of cultural and tourism venues and establishes spatial semantic maps; assigns narrative perspective identifiers to multiple tourists in the cultural and tourism venues, with each narrative perspective identifier corresponding to an independent virtual content rendering layer; Action decomposition and push module: detects any tourist's interactive action, decomposes the interactive action into physical actions and content triggering events; broadcasts the physical actions to other tourists in the cultural and tourism venue, the physical actions do not contain virtual content information; based on the content triggering events and the narrative perspective identifier held by the tourist who triggered the interactive action, pushes the corresponding virtual content only to the tourist's rendering layer; Cross-perspective presentation module: For other tourists who receive the broadcast of the physical action, the module determines whether there is virtual content at the location where the interactive action occurs from the narrative perspective of the other tourists based on the spatial semantic graph. If there is, the virtual content is presented in the rendering layer of the other tourists. If there is no virtual content, alternative virtual content is generated at the location where the interactive action occurs according to the preset cross-perspective semantic alignment relationship and presented in the rendering layer of the other tourists. The presentation duration of the alternative virtual content is bound to the duration of the interactive action.

[0079] It should be noted that the intelligent realization system for cultural and tourism spaces that integrates MR technology provided in this embodiment of the invention is used to execute all the process steps of the intelligent realization method for cultural and tourism spaces that integrates MR technology in the above embodiment one. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0080] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligently realizing cultural and tourism spaces by integrating MR technology, characterized in that, include: S1: Perform spatial 3D reconstruction of cultural and tourism venues and establish spatial semantic maps; assign narrative perspective identifiers to multiple tourists in the cultural and tourism venues, with each narrative perspective identifier corresponding to an independent virtual content rendering layer; S2: Detect any visitor's interactive action, decompose the interactive action into physical actions and content triggering events; broadcast the physical actions to other visitors in the cultural and tourism venue, the physical actions do not contain virtual content information; based on the content triggering events and the narrative perspective identifier held by the visitor who triggered the interactive action, push the corresponding virtual content only to the visitor's rendering layer; S3: For other tourists who receive the broadcast of the physical action, determine whether there is virtual content at the location where the interactive action occurs from the narrative perspective of the other tourists based on the spatial semantic graph; if so, present the virtual content in the rendering layer of the other tourists. If it does not exist, then according to the preset cross-view semantic alignment relationship, alternative virtual content is generated at the location where the interactive action occurs and presented in the rendering layer of the other tourists. The presentation duration of the alternative virtual content is bound to the duration of the interactive action.

2. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 1, characterized in that, In S1, the specific steps of the spatial three-dimensional reconstruction include: Visual sensors, inertial measurement units, and depth sensors are used to collect data from the cultural and tourism sites. Under the factor graph optimization framework, a joint factor graph is constructed with the reprojection error of the visual sensor, the pre-integration constraint of the inertial measurement unit, and the depth consistency residual of the depth sensor as factor nodes. The six-degree-of-freedom pose of the rigid body containing the visual sensor, the inertial measurement unit, and the depth sensor is solved by nonlinear least squares optimization. Based on the solution results, a three-dimensional point cloud map of the cultural and tourism venue is constructed, and the three-dimensional coordinates and semantic categories of cultural landmarks are marked on the three-dimensional point cloud map to obtain the spatial semantic map.

3. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 2, characterized in that, In S1, the establishment of the spatial semantic map further includes a drift correction step, which includes: Based on historical tourist route data, predict the next cultural landmark that tourists will arrive at; Before tourists arrive at the cultural landmark, a visual feature matching model of the cultural landmark is preloaded. The visual feature matching model stores the local feature descriptors of the cultural landmark and their corresponding three-dimensional coordinates. When the distance between the tourist and the cultural landmark is less than the preset correction trigger distance, and the visual feature matching confidence is higher than the preset confidence threshold, the local feature descriptor of the cultural landmark detected in the current frame image is solved by PnP with the corresponding three-dimensional coordinates stored in the visual feature matching model to obtain the reference pose value. The current pose estimate and the reference pose value are then weighted and fused according to the visual feature matching confidence to obtain the pose correction value, and the pose correction value is used as the initial state for subsequent pose estimation.

4. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 1, characterized in that, In S1, the assignment step of the narrative perspective identifier includes: Obtain tourist profiles, which include age groups, cultural preference tags, and historical visit records; Based on the tourist profile, a recommended narrative perspective is matched and used to the tourist. The number of online tourists for each narrative perspective within the cultural and tourism venue is counted. When the number of online tourists for a recommended narrative perspective exceeds a preset limit, the narrative perspective with the fewest online tourists is assigned to the tourist.

5. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 1, characterized in that, In S2, the detection of the interactive action includes input detection of three modalities: eye tracking, gesture recognition, and voice commands. When the three modalities are triggered simultaneously, the valid input is determined according to the priority order of gesture recognition, voice command, and eye tracking, and the input of the modality with the highest priority is taken as the interaction action.

6. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 1, characterized in that, In S2, the content triggering rules of the virtual content rendering layer are stored in a multi-view content package, which includes: The public foundation layer stores scene background content shared by all narrative perspectives; The perspective content layer consists of an independent perspective content layer for each narrative perspective identifier, which stores the virtual items and narrative information specific to that narrative perspective identifier. A cross-perspective semantic alignment mapping table uses the combination of narrative perspective identifiers and interaction positions as the index key to record the mapping relationship between the interaction positions under each narrative perspective identifier and the corresponding alternative virtual content under other narrative perspective identifiers. The rendering layer binding relationship table records the binding relationship between each narrative perspective identifier and the corresponding independent virtual content rendering layer. The content of the common basic layer and perspective content layer is routed to the rendering layer of the corresponding visitor for presentation through the rendering layer binding relationship table.

7. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 6, characterized in that, In S3, the specific steps for querying and presenting the mapping relationship of the alternative virtual content include: In the cross-perspective semantic alignment mapping table, the alternative virtual content corresponding to the narrative perspective identifier held by the tourist who triggered the interactive action and the location where the interactive action occurred is queried using the index. The retrieved alternative virtual content is anchored to the location where the interactive action occurs in the spatial semantic map; The alternative virtual content is pushed to the rendering layer of the other tourists. The presentation duration of the alternative virtual content is bound to the duration of the interaction action. After the interaction action ends, the alternative virtual content is removed from the rendering layer of the other tourists. If there is no mapping relationship corresponding to the index in the cross-view semantic alignment mapping table, then based on the narrative perspective identifier held by the tourist who triggered the interaction and the location where the interaction occurred, combined with the narrative perspective identifier held by other tourists who received the physical action broadcast, alternative virtual content corresponding to the semantic location of the location of occurrence is dynamically generated, and the alternative virtual content is anchored to the location where the interaction occurred and then pushed to the rendering layer of the other tourists.

8. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 1, characterized in that, S3 also includes a multi-user collaborative narrative triggering step, which includes: Synchronize the narrative progress nodes of each tourist to the collaborative status server in real time; When the narrative progress nodes of multiple tourists holding different narrative perspective identifiers simultaneously reach the preset collaborative trigger node, collaborative narrative content corresponding to their respective narrative perspective identifiers is pushed to the rendering layers of the multiple tourists. The collaborative narrative content unfolds synchronously in time, and each tourist only receives the collaborative narrative content corresponding to the narrative perspective identifier held by that tourist.

9. The method for intelligent realization of cultural and tourism spaces integrating MR technology according to claim 1, characterized in that, It also includes a cloud-edge collaborative resource scheduling step, which includes: Based on ticket reservation data and historical visitor flow data, the cloud platform predicts the number of online visitors in the cultural and tourism venues in future time periods, and generates resource allocation instructions accordingly, which are then sent to edge computing nodes in advance. The edge computing node is configured with computing resources according to the resource allocation instructions, and is used to perform real-time broadcasting of the physical actions and calculation of the generation of the alternative virtual content, and to count the number of online visitors in real time. When the number of online visitors counted by the edge computing node exceeds a preset degradation threshold, the rendering layer content is graded according to the order of priority for core narrative content and degradation for decorative content.

10. A smart realization system for cultural tourism spaces integrating MR technology, characterized in that: The system is used to execute the intelligent realization method for cultural and tourism spaces integrating MR technology as described in any one of claims 1-9, specifically including: Spatial mapping and allocation module: performs spatial 3D reconstruction of cultural and tourism venues and establishes spatial semantic maps; assigns narrative perspective identifiers to multiple tourists in the cultural and tourism venues, with each narrative perspective identifier corresponding to an independent virtual content rendering layer; Action decomposition and push module: detects any tourist's interactive action, decomposes the interactive action into physical actions and content triggering events; broadcasts the physical actions to other tourists in the cultural and tourism venue, the physical actions do not contain virtual content information; based on the content triggering events and the narrative perspective identifier held by the tourist who triggered the interactive action, pushes the corresponding virtual content only to the tourist's rendering layer; Cross-perspective presentation module: For other tourists who receive the broadcast of the physical action, the module determines whether there is virtual content at the location where the interactive action occurs from the narrative perspective of the other tourists based on the spatial semantic graph. If there is, the virtual content is presented in the rendering layer of the other tourists. If there is no virtual content, alternative virtual content is generated at the location where the interactive action occurs according to the preset cross-perspective semantic alignment relationship and presented in the rendering layer of the other tourists. The presentation duration of the alternative virtual content is bound to the duration of the interactive action.

Images