Immersive scene experience system and method based on mixed reality superposition projection
By using multiple projection screens and MR glasses in the immersive experience system, combined with NFC tag identification and control module, precise alignment and synchronization between the projected image and MR virtual content are achieved. This solves the problem of multi-device collaborative control and gesture recognition that cannot be achieved in existing technologies, and improves the stability and interactivity of the immersive scene experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-03-06
AI Technical Summary
Existing immersive experience systems cannot achieve precise alignment between projected images and MR virtual content in the same space, support gesture interaction and multi-device collaborative control, and traditional MR devices rely on 3DoF posture tracking and cannot obtain position information, resulting in severe interference with gesture recognition in complex lighting environments, and lack of unified control and synchronization mechanisms.
An immersive scene experience system based on mixed reality overlay projection is adopted. Multiple projection screens form an experience space, and MR glasses are used for three-degree-of-freedom tracking and gesture recognition. NFC tags are used to generate a spatial model, and the virtual image and holographic projection scene are aligned and synchronized through a control module, supporting multi-device linkage control.
It achieves seamless integration of virtual and real content, provides reliable interaction and multi-device linkage control, improves the synchronization and stability of immersive scene experience, and supports multiple people to experience it at the same time.
Smart Images
Figure CN121614031A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of mixed reality and immersive display, and particularly to an immersive scene experience system and method based on mixed reality superposition projection. Background Art
[0002] Currently, existing immersive experiences are usually realized through a single technical path. One category is an immersive space formed by multi-channel projection, which provides a visual surrounding effect through dome or dome projection, but lacks real-time interaction and individualized virtual superposition capabilities; the other category is the virtual content superposition display based on MR glasses, which can present virtual objects in a local environment, but its experience space is limited and it cannot be synchronously integrated with a large-scale projection environment. In the prior art, there is no system that can simultaneously achieve precise alignment of projection images and MR virtual content in the same space, support gesture interaction, and enable multi-device collaborative control. In addition, traditional MR devices rely on 3DoF attitude tracking, cannot obtain position information, the projection environment has complex lighting, which interferes with gesture recognition, and there is a lack of unified control and synchronization mechanisms between systems, making it difficult to meet the requirements of immersive theaters or large-scale interactive experiences. Therefore, there is an urgent need for a new technical solution in this field to achieve seamless integration of virtual and real content through the spatial superposition of MR glasses and the projection environment, and to provide reliable interaction and multi-device linkage control.
[0003] Therefore, it is necessary to provide a new immersive scene experience system based on mixed reality superposition projection to solve the above technical problems. Summary of the Invention
[0004] The main object of the present invention is to provide an immersive scene experience system and a steel-lined split ring device based on mixed reality superposition projection, aiming to solve the above problems.
[0005] To achieve the above object, an immersive scene experience system based on mixed reality superposition projection proposed by the first aspect of the present invention includes: An experience module formed by surrounding with multiple projection screens, and an experience space is formed within the multiple projection screens; A projection module for constructing a 360-degree immersive holographic projection scene within the experience space; An MR glasses device for displaying virtual images and spatial interaction, and the MR glasses device has a three-degree-of-freedom tracking function and a gesture recognition function; A spatial positioning module for generating a spatial model and forming multiple NFC tag identifiers for the MR glasses device to perform position recognition within the spatial model; A control module is provided, which receives the position information of the MR glasses in the projection space and controls the playback screen of the projection module and / or the MR glasses device according to the position information, so that the screen of the virtual image is aligned and synchronized with the screen of the holographic projection scene.
[0006] Optionally, the spatial positioning module includes: The scanning unit is used to perform a three-dimensional scan of the experience space and generate a spatial model; The calibration unit is used to import the spatial model into the editing environment for manual position calibration, form multiple spatial anchor points, and store the information of the multiple spatial anchor points in a one-to-one correspondence with different NFC tag identifiers that can be identified by the MR glasses device.
[0007] Optionally, when the MR glasses device triggers the NFC tag, it sends the corresponding NFC tag identifier to the control module; the control module determines the position of the MR glasses device in the projection space based on the NFC tag identifier. After the control module determines the position of the MR glasses device in the projection space, the control module corrects the position of the virtual image according to the position information of the MR glasses device, so that the image of the virtual image is aligned with the image of the holographic projection scene; the position information includes the position of the MR glasses device in the projection space and the orientation information of the MR glasses device.
[0008] Optionally, the MR glasses device calculates the rotation posture of the MR glasses device based on the user's position information, and rotates the MR glasses device so that the image of the virtual image is consistent with the image of the holographic projection scene.
[0009] Optionally, it also includes an infrared fill light module, which is used to provide fill light to the gesture recognition area of the MR glasses device.
[0010] Optionally, it also includes a virtual content layered rendering module; The virtual content layered rendering module is used to perform layered rendering of the holographic projection scene and dynamically adjust the rendering order according to the perspective of the MR glasses device.
[0011] Optionally, it also includes an environmental perception feedback module; The environmental perception feedback module is used to collect environmental information within the experience space and dynamically adjust the parameters of the MR glasses device based on the environmental information; the environmental information includes lighting, temperature, and motion status information; the parameters include brightness, color, and animation parameters.
[0012] A second aspect of the present invention also provides a method for immersive scene experience based on mixed reality overlay projection, implemented using an immersive scene experience system based on mixed reality overlay projection, comprising the following steps: S1: Establish an experience space, which is formed by multiple projection screens surrounding it; S2: Perform a 3D scan of the experience space to generate a spatial model; the spatial model includes multiple identifiable spatial anchor points, and store the information of the multiple spatial anchor points in a one-to-one correspondence with different NFC tag identifiers; S3: When the user puts on the MR glasses and starts the APP, the MR glasses are connected to the control module and acquire the virtual scene and control commands; S4: After the user arrives at the preset experience point according to the control command, the MR glasses device sends the current NFC tag identifier to the control module through the NFC sensing tag. The control module obtains the corresponding spatial location information of the MR glasses device according to the NFC tag identifier. S5: The control module adjusts the position and angle of the virtual image according to the spatial position information of the MR glasses device, so that the image of the virtual image is aligned with the image of the holographic projection scene. S6: The control module uses a timeline scheme to play the images of the holographic projection scene and the virtual image.
[0013] Optionally, in step S6, when the user triggers a virtual event and / or operates a virtual object in the scene through gestures, the control system dynamically adjusts the image of the virtual image in real time according to the position information of the MR glasses device, so that the image of the virtual image is synchronized with the user's actions, while keeping the image of the holographic projection scene consistent with the image of the virtual image.
[0014] Optionally, in step S6, an infrared fill light module is used to illuminate the gesture recognition area of the MR glasses device to ensure stability when the user triggers virtual events and / or operates virtual objects in the scene through gestures.
[0015] In this invention, the MR glasses device triggers an NFC tag, and the control device determines the location information of the MR glasses device based on the information identified by the NFC tag. Before use, the MR glasses device guides the user to look at a pre-set location by playing a guide screen. The control module automatically corrects the angle of the MR glasses device according to the user's orientation, aligning the virtual image played by the MR glasses device with the holographic projection scene. Compared to traditional methods that do not locate the MR glasses device, the fit between the virtual image and the holographic projection scene is much higher. At the same time, the MR glasses device has three degrees of freedom tracking and gesture recognition functions. The user triggers virtual events or operates virtual objects in the scene through gestures, and the APP dynamically adjusts the content of the virtual image based on real-time sensor data to keep it synchronized with the user's actions. Meanwhile, the content displayed on the projection screen is consistent with the content rendered by the MR glasses device, realizing the interaction between the virtual and reality. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0017] Figure 1 This is a schematic diagram of an immersive scene experience system based on mixed reality overlay projection in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the immersive scene experience method based on mixed reality overlay projection in an embodiment of the present invention.
[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0020] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.
[0021] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0022] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0023] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0024] Example 1 Please refer to Figure 1 This invention proposes an immersive scene experience system based on mixed reality overlay projection; comprising: The experience module is formed by multiple projection screens surrounding each other, creating an experience space within the multiple projection screens; A projection module is used to construct a 360-degree immersive holographic projection scene within the experience space; MR glasses device for displaying virtual images and spatial interaction, the MR glasses device has three degrees of freedom tracking function and gesture recognition function; A spatial positioning module is used to generate a spatial model and form multiple NFC tags within the spatial model for location identification by the MR glasses device. A control module is provided, which receives the position information of the MR glasses in the projection space and controls the playback screen of the projection module and / or the MR glasses device according to the position information, so that the screen of the virtual image is aligned and synchronized with the screen of the holographic projection scene.
[0025] In this embodiment, the MR glasses device triggers an NFC tag. The control device determines the location information of the MR glasses device based on the information identified by the NFC tag. Before use, the MR glasses device guides the user to look at a pre-set location by playing a guide screen. The control module automatically corrects the angle of the MR glasses device according to the user's orientation, aligning the virtual image played by the MR glasses device with the holographic projection scene. Compared to traditional methods that do not locate the MR glasses device, the fit between the virtual image and the holographic projection scene is much higher. Simultaneously, the MR glasses device has three degrees of freedom tracking and gesture recognition capabilities. The user triggers virtual events or operates virtual objects in the scene through gestures. The APP dynamically adjusts the content of the virtual image based on real-time sensor data, keeping it synchronized with the user's actions. At the same time, the content displayed on the projection screen is consistent with the content rendered by the MR glasses device, achieving interaction between the virtual and reality. The control module is a multi-device operation platform. It is worth noting that the control module can synchronously control multiple MR glasses devices, meaning that this scene experience system can support multiple users experiencing it simultaneously.
[0026] Optionally, the spatial positioning module includes: The scanning unit is used to perform a three-dimensional scan of the experience space and generate a spatial model; The calibration unit is used to import the spatial model into the editing environment for manual position calibration, form multiple spatial anchor points, and store the information of the multiple spatial anchor points in a one-to-one correspondence with different NFC tag identifiers that can be identified by the MR glasses device.
[0027] Optionally, when the MR glasses device triggers the NFC tag, it sends the corresponding NFC tag identifier to the control module; the control module determines the position of the MR glasses device in the projection space based on the NFC tag identifier. After the control module determines the position of the MR glasses device in the projection space, the control module corrects the position of the virtual image according to the position information of the MR glasses device, so that the image of the virtual image is aligned with the image of the holographic projection scene; the position information includes the position of the MR glasses device in the projection space and the orientation information of the MR glasses device.
[0028] In this embodiment, after detecting an NFC tag, the MR glasses device reads the spatial positioning information corresponding to the NFC tag identifier via a network request to determine the current position of the MR glasses in the projection space.
[0029] Optionally, the MR glasses device calculates the rotation posture of the MR glasses device based on the user's position information, and rotates the MR glasses device so that the image of the virtual image is consistent with the image of the holographic projection scene.
[0030] Optionally, it also includes an infrared fill light module, which is used to provide fill light to the gesture recognition area of the MR glasses device.
[0031] In this embodiment, the infrared fill light module is installed on the top of the experience space. The infrared fill light module emits infrared light with wavelengths in the invisible light range to fill light on the gesture recognition area of the MR glasses device to enhance the gesture recognition accuracy without affecting the projection display.
[0032] Optionally, it also includes a virtual content layered rendering module; The virtual content layered rendering module is used to perform layered rendering of the holographic projection scene and dynamically adjust the rendering order according to the perspective of the MR glasses device.
[0033] In this embodiment, the virtual objects in the projection scene are rendered in layers according to the foreground, background and midground.
[0034] Optionally, it also includes an environmental perception feedback module; The environmental perception feedback module is used to collect environmental information within the experience space and dynamically adjust the parameters of the MR glasses device based on this information. The environmental information includes lighting, temperature, and motion status information; the parameters include brightness, color, and animation parameters. The environmental perception feedback module enables adaptive matching between virtual content and the real environment, improving the stability and realism of the scene experience system under complex lighting and dynamic environments.
[0035] Example 2 Please refer to Figure 2 An immersive scene experience method based on mixed reality overlay projection, implemented using the immersive scene experience system based on mixed reality overlay projection described in Example 1, includes the following steps: S1: Establish an experience space, which is formed by multiple projection screens surrounding it; S2: Perform a 3D scan of the experience space to generate a spatial model; the spatial model includes multiple identifiable spatial anchor points, and store the information of the multiple spatial anchor points in a one-to-one correspondence with different NFC tag identifiers; S3: When the user puts on the MR glasses and starts the APP, the MR glasses are connected to the control module and acquire the virtual scene and control commands; S4: After the user arrives at the preset experience point according to the control command, the MR glasses device sends the current NFC tag identifier to the control module through the NFC sensing tag. The control module obtains the corresponding spatial location information of the MR glasses device according to the NFC tag identifier. S5: The control module adjusts the position and angle of the virtual image according to the spatial position information of the MR glasses device, so that the image of the virtual image is aligned with the image of the holographic projection scene. S6: The control module uses a timeline scheme to play the images of the holographic projection scene and the virtual image.
[0036] Optionally, in step S6, when the user triggers a virtual event and / or operates a virtual object in the scene through gestures, the control system dynamically adjusts the image of the virtual image in real time according to the position information of the MR glasses device, so that the image of the virtual image is synchronized with the user's actions, while keeping the image of the holographic projection scene consistent with the image of the virtual image.
[0037] Optionally, in step S6, an infrared fill light module is used to illuminate the gesture recognition area of the MR glasses device to ensure stability when the user triggers virtual events and / or operates virtual objects in the scene through gestures.
[0038] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. An immersive scene experience system based on mixed reality overlay projection, characterized by, The system comprises: an experience module, surrounded by a plurality of projection curtains, forming an experience space in the plurality of projection curtains; a projection module, configured to construct a 360-degree immersive holographic projection scene in the experience space; an MR glasses device, configured to display a virtual image and spatial interaction, the MR glasses device having a three-degree-of-freedom tracking function and a gesture recognition function; a space positioning module, configured to generate a space model and form a plurality of NFC tag identifiers for position recognition of the MR glasses device in the space model; a control module, configured to receive position information of the MR glasses in the projection space, and control a playing screen of the projection module and / or the MR glasses device according to the position information, so that the screen of the virtual image is aligned and synchronized with the screen of the holographic projection scene.
2. The mixed-reality overlay projection-based immersive scene experience system of claim 1, wherein, The space positioning module comprises: a scanning unit, configured to perform three-dimensional scanning on the experience space and generate a space model; a calibration unit, configured to import the space model into an editing environment for manual position calibration, form a plurality of space anchor points, and store a plurality of space anchor point information and different NFC tag identifiers that can be recognized by the MR glasses device in one-to-one correspondence.
3. The immersive scene experience system based on mixed reality superimposed projection according to claim 2, wherein when the MR glasses device equipment triggers an NFC tag, the corresponding NFC tag identifier is sent to the control module; the control module determines the position of the MR glasses device in the projection space according to the NFC tag identifier; when the control module determines the position of the MR glasses device in the projection space, the control module corrects the position of the virtual image according to the position information of the MR glasses device, so that the screen of the virtual image is aligned with the screen of the holographic projection scene; the position information comprises the position of the MR glasses device in the projection space and the orientation information of the MR glasses device.
4. The mixed-reality overlay projection-based immersive scene experience system of claim 3, wherein, The MR glasses device calculates the rotation posture of the MR glasses device according to the position information of the user, and rotates the MR glasses device, so that the screen of the virtual image is consistent with the screen of the holographic projection scene.
5. The mixed-reality overlay projection-based immersive scene experience system of claim 1, wherein, Further comprising an infrared light supplementing module, configured to supplement light to a gesture recognition area of the MR glasses device.
6. The mixed-reality overlay projection-based immersive scene experience system of claim 1, wherein, Further comprising a virtual content layered rendering module; The virtual content layered rendering module is configured to perform layered rendering on the holographic projection scene, and dynamically adjust the rendering order according to the viewing angle of the MR glasses device.
7. The mixed-reality overlay projection-based immersive scene experience system of claim 1, wherein, Further comprising an environment perception feedback module; The environment perception feedback module is configured to collect environment information in the experience space, and adjust parameters of the MR glasses device according to the environment information; the environment information comprises illumination, temperature and motion state information; the parameters comprise brightness, color and animation parameters.
8. A method for immersive scene experience based on mixed reality overlay projection, implemented by using the immersive scene experience system based on mixed reality overlay projection according to any one of claims 1-7, characterized in that, The system comprises the following steps: S1: establishing an experience space, surrounded by a plurality of projection curtains; S2: generating a space model by three-dimensional scanning in the experience space; the space model includes a plurality of identifiable space anchor points, and a plurality of space anchor point information are stored in one-to-one correspondence with different NFC tag identifiers; S3: when the user wears the MR glasses device and starts the APP, the MR glasses device is signal-connected with the control module and acquires a virtual scene and a control instruction; S4: after the user reaches a preset experience point according to the guidance of the control instruction, the MR glasses device sends a current NFC tag identifier to the control module through an NFC sensing tag, and the control module acquires space position information corresponding to the MR glasses device according to the NFC tag identifier; S5: the control module adjusts the position and angle of the virtual image according to the space position information of the MR glasses device, so that the picture of the virtual image is aligned with the picture of the holographic projection scene; S6: the control module plays the picture of the holographic projection scene and the picture of the virtual image by using a timeline scheme.
9. The mixed-reality overlay projection-based immersive scene experience method of claim 8, wherein, In the step S6, when the user triggers a virtual event and / or operates a virtual object in the scene by a gesture, the control system dynamically adjusts the picture of the virtual image in real time according to the position information of the MR glasses device, so that the picture of the virtual image is kept synchronous with the user action, and the picture of the virtual image is kept consistent with the picture of the holographic projection scene.
10. The mixed-reality overlay projection-based immersive scene experience system of claim 9, wherein, In the step S6, an infrared light supplementing module is used to supplement light to a gesture recognition area of the MR glasses device, so as to ensure the stability when the user triggers a virtual event and / or operates a virtual object in the scene by a gesture.