A display interaction method and device based on mixed reality

Abstract

The present invention relates to a display interaction method and device based on mixed reality. It allows users to roam in a virtual scene, and at the same time, the display racks and 3D printed rough models in the real world are mapped to the virtual scene in the correct posture, and presented in the user's field of vision as exquisite digital models. Users can also perform operations such as scene switching and brightness adjustment through gesture interaction, and at the same time, the material of the digital model corresponding to each rough model in the virtual scene can be switched. For some cultural relics in the device, some interactive operations can also be performed, such as pouring wine from a wine jar, stamping with a jade seal, lighting a palace lantern, etc., and corresponding real-life special effects will be presented in the scene. Compared with other virtual display devices and applications, the advantage of this device is that it realizes the dual perception of vision and touch, allowing users to have a more immersive experience, and allows users to interact with different cultural relics to "make cultural relics alive."

Description

Technical Field

[0001] The present invention relates to the field of mixed reality, and in particular to a display interaction method and device based on mixed reality. Background Art

[0002] As an extension of physical museums in the digital field, the construction of smart display projects in museums has been widely carried out at home and abroad. Such projects can expand the problem that museums and their collections of cultural relics are unable to provide cultural services outside of opening hours or outside the areas where museums and exhibits are located due to their own time and space locations, so that more people can learn scientific and cultural knowledge and improve their ideological and moral qualities. Existing display projects are generally implemented based on web pages or APPs, and there are problems such as insufficient user immersion and insufficient interactivity. In response to these problems, we proposed a mixed reality display interaction device, which uses mixed reality technology to realize users' free roaming in virtual scenes, real-time mapping between real 3D printed cultural relics rough models and virtual digital models, intelligent cultural relics interaction based on gesture recognition, and special effects simulation based on realistic drawing graphics. The device can realize the dual perception of vision and touch, allowing users to have a more immersive experience, and allows users to interact with different cultural relics to "make cultural relics alive." Summary of the invention

[0003] The technical problem solved by the present invention is: to realize a novel display interaction method and device based on mixed reality, which can realize the free roaming of users in virtual scenes, real-time mapping between real 3D printed cultural relics rough models and virtual digital models, intelligent cultural relics interaction based on gesture recognition, and special effects simulation based on realistic drawing graphics.

[0004] The technical solution adopted by the present invention is: a display interaction method based on mixed reality, comprising the following steps:

[0005] Step 1: Virtual scene roaming: Provide a virtual scene, and the user will immerse himself in the virtual scene, calibrate the position of the real exhibition stand and the position of the virtual exhibition stand so that the two coincide;

[0006] Step 2: Virtual-real mapping of cultural relics: first, collect red, green and blue depth image data through a depth camera, then analyze the sparse depth map obtained by the virtual cultural relic model, obtain the pose of the real cultural relic rough model relative to the depth camera, and combine the pose of the depth camera to obtain the real pose of the real cultural relic rough model in the world coordinate system, and set the pose of the virtual cultural relic model in the graphics engine to be consistent with the real pose of the real cultural relic rough model relative to the world coordinate system, so as to realize the virtual-real mapping of cultural relics;

[0007] Step 3: Intelligent gesture interaction: The binocular camera of the mixed reality head-mounted display collects the user's hand movements and left and right eye visual images. The gesture recognition method based on binocular vision uses the parallax information of the collected left and right eye visual images to generate a depth image based on the stereo vision algorithm, where each pixel in the image represents the distance from the corresponding point of the object to the camera. Then, through joint point extraction, the gesture is converted into a digital feature to obtain the spatial position of the hand joint point, and then different joint point combinations are used to track and identify specific hand gestures; the stereo vision algorithm includes five steps: image acquisition, feature extraction, feature matching, parallax calculation, and depth map generation;

[0008] Step 4: Special effects simulation. The different particle systems corresponding to the special effects of smoke emitted from the jade seal of the Qing Dynasty, flames of the Changxin Palace lanterns, and ignition flames were processed so that the smoke could produce a hazy visual effect and the flames could be seen when they were ignited. The interaction between smoke and other objects and users was optimized, and collision effects were added to the smoke to improve the authenticity of the interaction. Based on the content of pouring wine from the wine jar in the virtual and real scenes, special effects processing based on fluid simulation was performed on some container exhibits, achieving a water flow simulation effect close to reality.

[0009] Furthermore, in step one, three virtual scenes of different styles are provided for users to view, including a classical Chinese exhibition hall scene, a modern Chinese exhibition hall scene, and a Western exhibition hall scene.

[0010] Further, the specific implementation method in step 2 is:

[0011] First, the depth camera is used to collect red, green and blue depth image data, and then the data is converted into a common format according to the camera parameters. At the same time, the sparse depth map obtained by the virtual cultural relic model is analyzed. After histogram statistics, depth screening and probability function optimization, the pose of the real cultural relic rough model relative to the depth camera is obtained.

[0012] Furthermore, the position and posture of the depth camera in the world coordinate system; the tracker is fixed to the depth camera, and the depth camera position and posture are read in the graphics engine;

[0013] By combining the above two postures, we can get the real posture of the rough model of the real cultural relic relative to the world coordinate system. By setting the posture of the virtual cultural relic model in the graphics engine to be consistent with the real posture of the rough model of the real cultural relic relative to the world coordinate system, we can realize the virtual-real mapping of the cultural relic.

[0014] Furthermore, the gesture recognition method based on binocular vision specifically includes the following five steps:

[0015] 1) Use a temporal sliding window to extract action clips;

[0016] 2) Use the scale-invariant feature transformation method to extract key information from each segment;

[0017] 3) Divide the key information features of each segment into dynamic and static ones as the input of the classifier;

[0018] 4) For each classification operation input, effectively classify it and optimize the starting frame;

[0019] 5) Use non-maximum suppression method to remove repeated fragments.

[0020] Furthermore, the special effect simulation includes a particle simulation method and a fluid simulation method, wherein the particle simulation method includes the following steps:

[0021] 1) During initialization, various properties of each particle are initialized, including the particle's speed and coordinates. An index mechanism is established to allow new particles to point to the index mechanism and add new particles to the stack of active particles.

[0022] 2) The properties of each particle are transmitted to the graphics processing unit (GPU); the data is stored in the graphics card. The two important properties of the particle, speed and coordinates, are stored in the cache of the graphics card. They are grouped together and a backup cache of the same size is created so that they can exchange data with each other.

[0023] 3) Update the properties of each particle; during the processing, the speed and coordinate data of the particle are read from the cache area of ​​the graphics card, which stores the two important properties of the particle. According to the parameter changes of each frame, the particle's coordinates and speed are continuously updated with reference to the particle's motion equation. The updated new coordinates and speed properties are stored in the backup cache area, and data is exchanged between them. This realizes the update step, so that the particles can move according to a certain rule.

[0024] 4) Store the particle data in the vertex buffer object VBO and prepare for rendering;

[0025] 5) If the particle life cycle ends, the index mechanism will be reclaimed;

[0026] The fluid simulation method specifically includes the following steps:

[0027] Step 1: Establish a model; Based on the particle method, the fluid is regarded as a collection of a large number of particles, each particle represents a small part of the fluid, and the behavior of the fluid is simulated by modeling the movement and interaction of the particles;

[0028] Step 2: Numerical solution: Based on the Euler method, for each particle, first calculate the acceleration of the next time step according to the current position and velocity as well as the external force field, then update the velocity and position of the particle according to the calculated acceleration, and then repeat the above steps to continuously iterate and update the state of the particle to simulate the change in time;

[0029] Step 3: Particle rendering based on point sprites; each particle is represented as a point on the screen and rendered according to the particle's properties, including position, color, and size.

[0030] The present invention also proposes a display interaction device based on mixed reality, comprising the following modules:

[0031] Virtual scene roaming module: It is used for users to immerse themselves in the virtual scene and obtain a realistic exhibition hall tour experience; the position of the real exhibition stand is calibrated with the position of the virtual exhibition stand so that the two overlap. When the user picks up the 3D printed rough model or puts the rough model back on the real exhibition stand, the virtual cultural relics will also appear in the corresponding position on the virtual exhibition stand; three different styles of virtual scenes are also provided for users to view;

[0032] Cultural relic virtual-to-real mapping module: used to achieve virtual-to-real mapping of cultural relics; firstly, the depth camera is used to collect red, green and blue depth image data, and then the sparse depth map obtained by the virtual cultural relic model is analyzed to obtain the pose of the real cultural relic rough model relative to the depth camera. Combined with the pose of the depth camera, the real pose of the real cultural relic rough model in the world coordinate system is obtained, and the pose of the virtual cultural relic model in the graphics engine is set to be consistent with the real pose of the real cultural relic rough model relative to the world coordinate system, so as to achieve virtual-to-real mapping of cultural relics;

[0033] The intelligent gesture interaction module maps the user's hand gesture in the real space to the corresponding virtual scene, and presents it in the mixed reality head-mounted display to achieve the intelligent gesture interaction effect; the binocular camera of the mixed reality head-mounted display collects the left and right eye visual images of the user's hand movements, and the gesture recognition method based on binocular vision uses the parallax information of the collected left and right eye visual images to generate a depth image through a stereo vision algorithm, in which each pixel in the image represents the distance from the corresponding point of the object to the camera, and then through joint point extraction, the gesture is converted into a digital feature to obtain the spatial position of the hand joint point, and then different joint point combinations are used to track and identify specific hand gestures. The stereo vision algorithm includes five steps: image acquisition, feature extraction, feature matching, parallax calculation, and depth map generation;

[0034] Special effects simulation module: used to achieve close to realistic smoke, flame and water flow simulation effects; based on the jade seals, Changxin Palace lanterns and ignition contents of the Qing Dynasty in the virtual and real scenes, this module processes different particle systems corresponding to the special effects of the smoke emitted from the jade seals of the Qing Dynasty, the flames of the Changxin Palace lanterns and the ignition flames, so that the smoke can produce a hazy visual effect and the flames can see the smoke produced when they are ignited; this module also optimizes the interaction between smoke and other objects and users, and adds collision effects to the smoke to improve the authenticity of the interaction; based on the wine jar pouring content in the virtual and real scenes, this module performs special effects processing based on fluid simulation on some container exhibits, achieving close to realistic water flow simulation effects.

[0035] The principle of the present invention is:

[0036] (1) In mixed reality, by calibrating and identifying the position and establishing the correct virtual-real mapping relationship, the virtual scene and cultural relic agent can be matched with the real scene and the real 3D printed cultural relic rough model;

[0037] (2) By generating a depth image through a binocular camera and a stereo vision algorithm including five steps: image acquisition, feature extraction, feature matching, disparity calculation, and depth map generation, the hand joints can be extracted and matched with the hand bones to identify the real-time hand posture, thereby mapping the user's hand posture in the real space to the corresponding virtual environment and presenting it in the head-mounted display to achieve intelligent gesture interaction effects;

[0038] (3) Through particle simulation and fluid simulation based on the graphics engine, realistic rendering of special effects such as smoke, fire, and flowing water can be achieved.

[0039] The advantages of the present invention compared with the prior art are:

[0040] The present invention uses mixed reality technology and rough models of real cultural relics to introduce tactile perception into the virtual display system, achieving dual perception of vision and touch, and enhancing the user's immersive viewing experience;

[0041] It enables intelligent interaction between users and cultural relics in different collections, enhancing users’ sense of participation in the viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a diagram of the device architecture of the present invention;

[0043] Figure 2 This is a virtual scene roaming effect diagram of the present invention;

[0044] Figure 3 This is the virtual-real mapping effect diagram of the cultural relics of the present invention;

[0045] Figure 4 This is a diagram showing the interaction effect of the intelligent gesture of the present invention;

[0046] Figure 5 This is a special effect simulation effect diagram of the present invention. DETAILED DESCRIPTION

[0047] 1. Device Architecture

[0048] like Figure 1As shown in , the architecture of this device is divided into three layers: the hardware layer is all the hardware used in the device, including physical display racks, multiple real 3D printed rough models of cultural relics and supporting trackers and positioning base stations, mixed reality head-mounted displays, and a graphics workstation; the driver layer is located above the hardware layer. Through the driver program and mixed reality runtime, the driver layer processes and transmits the data obtained by the hardware layer for use by the graphics engine; the application layer is above the driver layer, and the related programs of the application layer realize virtual scene roaming, virtual and real mapping of cultural relics, intelligent gesture interaction, special effects simulation and other functions, providing users with a unique viewing experience.

[0049] 2. Virtual scene roaming module

[0050] By using real-time rendering technology, users can get a very realistic exhibition hall tour experience in a virtual scene after wearing a mixed reality head-mounted display. Real-time rendering technology can quickly and accurately process a large amount of image data through advanced computing and graphics processing capabilities, and convert it into realistic visual effects in real time. This technology can not only show delicate light, color and texture, but also simulate physical effects in the real world, such as reflection, refraction and shadow, making the virtual scene closer to the real environment.

[0051] Through the graphical user interface of the mixed reality headset driver, the world coordinate origin position of the virtual space can be calibrated, thereby calibrating the position of the real display stand and the position of the virtual display stand so that the two coincide. When the user picks up the rough model of the real cultural relic or puts the rough model back on the real display stand, the virtual cultural relic model will also appear in the correct position on the virtual display stand.

[0052] This device provides three different styles of virtual exhibition hall scenes for users to visit: classical Chinese exhibition hall, modern Chinese exhibition hall, and Western exhibition hall. Figure 2 As shown, users can switch freely between different scenes without introducing additional hardware, overcoming the problem of single scene style and easy loss of freshness in traditional virtual display applications.

[0053] 3. Cultural Relics Virtual Reality Mapping Module

[0054] In the virtual-to-reality mapping module, the red, green and blue depth image data obtained based on the depth camera is used, and the simultaneous localization and mapping (SLAM) technology is used to obtain the pose of the rough model of the real cultural relic. Then, combined with the acquired pose information, the graphics engine is used for subsequent rendering operations to achieve virtual-to-reality mapping of the cultural relics.

[0055] First, the depth camera is used to collect red, green and blue depth image data, and then the data is converted into a common format according to the parameters of the camera, and the virtual cultural relic model is analyzed with the help of a stereoscopic vision algorithm. The stereoscopic vision algorithm includes five steps: image acquisition, feature extraction, feature matching, disparity calculation, and depth map generation. The image acquisition part uses two or more cameras or depth sensors with a certain distance in space to shoot the scene at the same time. These cameras or sensors have a certain distance in space and capture different perspectives of the same scene. The feature extraction part extracts key points or features from the image of each perspective, such as corners, edges or textures, which have significantly changing positions and feature descriptors in the image. Feature matching matches the features extracted from different perspectives to find corresponding feature points in multiple images. Disparity calculation infers the depth information of objects in the scene by calculating the disparity or disparity vector between matching feature points, that is, the horizontal offset of the feature point between the left and right views. Depth map generation converts disparity information into depth values ​​to generate a sparse depth map. The sparse depth map obtained by the stereoscopic vision algorithm is analyzed through histogram statistics, depth screening, and probability function optimization to obtain the pose of the real cultural relic rough model relative to the depth camera. Among them, the depth histogram is histogram-based according to the pixel values ​​of the depth image to obtain the distribution of depth in the scene; the depth screening selects the depth values ​​with higher frequency in the histogram as the effective depth range; the probability function optimization uses the gradient descent method to find the optimal pose parameters of the cultural relic model to match the pixels observed by the depth camera. In order to obtain the pose of the virtual cultural relic model in the world coordinate system of the scene, the pose of the depth camera in the world coordinate system is further calculated. This module uses a tracker to measure the pose of the depth camera in the world coordinate system, that is, the tracker is fixed to the depth camera, and the pose of the depth camera is read in the graphics engine. By adding the vectors corresponding to the above two poses, the real pose of the virtual cultural relic model relative to the world coordinate system can be obtained. The pose of the virtual cultural relic model in the graphics engine is set to be consistent with the real pose of the real cultural relic rough model relative to the world coordinate system, so that the mapping between the virtual cultural relic model and the real cultural relic rough model can be achieved, and the effect is as follows. Figure 3 shown.

[0056] 4. Intelligent Gesture Interaction Module

[0057] In the intelligent gesture interaction module, the device performs gesture detection based on the analysis of candidate action fragments (Action Proposal), which specifically includes the following five steps:

[0058] 1) Use a temporal sliding window to extract action clips;

[0059] 2) Use the scale-invariant feature transform (SIFT) method to extract key information from each segment;

[0060] 3) Divide the key information features of each segment into dynamic and static ones as the input of the classifier;

[0061] 4) For each classification operation input, effectively classify it and optimize the starting frame;

[0062] 5) Use methods such as Non-Maximum Suppression (NMS) to remove duplicate fragments.

[0063] The gesture interaction effect of this device can be seen in Figure 4 .

[0064] 5. Special effects simulation module

[0065] This module includes two parts: particle simulation and fluid simulation. The particle simulation method is as follows:

[0066] 1) During initialization, various properties of each particle are initialized. For example, the properties of a particle include its speed and coordinates. An index mechanism is established to point new particles to the index mechanism and add new particles to the stack of active particles.

[0067] 2) The properties of each particle are transmitted to the graphics processing unit (GPU). Data storage is completed in the graphics card. The two important properties of the particle speed and coordinates are stored in the cache of the graphics card. They are treated as a group and a backup cache of the same size is created so that they can exchange data with each other;

[0068] 3) Update the properties of each particle. During the processing, the speed and coordinate data of the particle are read from the cache area of ​​the graphics card, which stores the two important properties of the particle. According to the parameter changes of each frame, the coordinates and speed of the particle are continuously updated with reference to the particle's motion equation. The updated new coordinates and speed properties are stored in the backup cache area, and data is exchanged between them. This achieves the update step, so that the particles can move according to a certain rule.

[0069] 4) Store the particle data in the vertex buffer object (VBO) and prepare for rendering;

[0070] 5) If the particle life cycle ends, the index mechanism will be reclaimed.

[0071] The fluid simulation method includes three steps: step 1, model building, step 2, numerical solution of the model, and step 3, fluid rendering. The implementation of these three steps determines the final effect of fluid simulation. Among them, the method of building the model in step 1 is based on the particle method, which regards the fluid as a collection of a large number of particles, each particle represents a small part of the fluid, and simulates the behavior of the fluid by modeling the movement and interaction of the particles; the numerical solution in step 2 is based on the Euler method. First, for each particle, according to the current position and velocity and the external force field, the acceleration of the next time step is calculated, and then the velocity and position of the particle are updated according to the calculated acceleration. After that, the above steps are repeated, and the state of the particle is continuously iterated to simulate the change in time; step 3 is based on particle rendering of point sprites, each particle is represented as a point on the screen, and rendered according to the properties of the particle (position, color, size, etc.).

[0072] Among these three steps, the establishment of the fluid model is the basis of the latter two steps. The numerical solution method determines the complexity of the calculation and the detail of the simulation, and the selection of a suitable rendering method directly determines the final simulation effect of the fluid. In the previous model establishment and numerical solution process, the state parameters of the fluid motion are obtained. If you want to visualize these data, you need to choose a suitable rendering method to present these data in a graphical way. These data are reflected in the form of fields. For these field data, traditional geometric description methods are not suitable, and volume rendering technology can achieve the required rendering effect. Therefore, in this device, the volume data of fluid simulation is obtained through simulation calculation, and then the volume rendering is realized according to the following three steps:

[0073] 1) Data preprocessing, which involves interpolation, data simplification, and resampling of the input raw data to improve accuracy and reliability. The interpolation operation infers the values ​​at these locations based on the missing values ​​between known data points or the data points between discrete data points. Data simplification is achieved by deleting redundant points or using fewer representation points. Resampling increases or decreases the number of data points and converts the data from one sampling rate to another to meet the analysis requirements of the conditional judgment part.

[0074] 2) Condition judgment: by obtaining the processed data, judging according to the fluid velocity, physical properties, environmental conditions, etc., different fluid rendering effects can be obtained. For example, according to the direction and magnitude of gravity, the direction and form of fluid flow are adjusted, and the velocity of the fluid outflow is calculated. When the fluid velocity is higher than the threshold, the turbulence effect is selected, and when it is lower than the threshold, the fluid static or slow flow effect is selected;

[0075] 3) Drawing and rendering, representing the fluid simulation results as a three-dimensional voxel grid, and using the voxel properties (density, color) for rendering and visualization to produce realistic fluid effects.

[0076] The special effects simulation effect of this device can be found in Figure 5 .

[0077] The hardware configuration of the graphics workstation used in the device is Intel i9-13900K processor, Nvidia GeForce3080Ti graphics card, 32G memory, running on 64-bit Windows 11 system. The mixed reality head-mounted display used in the system is Varjo XR-3, the tracker is HTC Vive Tracker 3.0, the base station is HTC Vive Base Station 2.0, the depth camera is Microsoft Azure Kinect DK, the rough model is made by 3D printing, and the physical display frame is composed of two ordinary cheap iron frames.

[0078] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

[0079] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A display interaction method based on mixed reality, characterized in that: The following steps are involved: Step 1: Virtual scene roaming; Provide a virtual scene, in which users can immerse themselves and calibrate the position of the real exhibition stand with the position of the virtual exhibition stand so that the two coincide. Step 2: Virtual-real mapping of cultural relics: first, collect red, green and blue depth image data through a depth camera, then analyze the sparse depth map obtained by the virtual cultural relic model, obtain the pose of the real cultural relic rough model relative to the depth camera, and combine the pose of the depth camera to obtain the real pose of the real cultural relic rough model in the world coordinate system, and set the pose of the virtual cultural relic model in the graphics engine to be consistent with the real pose of the real cultural relic rough model relative to the world coordinate system, so as to realize the virtual-real mapping of cultural relics; Step 3: Intelligent gesture interaction: The binocular camera of the mixed reality head-mounted display collects the user's hand movements and left and right eye visual images. The gesture recognition method based on binocular vision uses the parallax information of the collected left and right eye visual images to generate a depth image based on the stereo vision algorithm, where each pixel in the image represents the distance from the corresponding point of the object to the camera. Then, through joint point extraction, the gesture is converted into a digital feature to obtain the spatial position of the hand joint point, and then different joint point combinations are used to track and identify specific hand gestures; the stereo vision algorithm includes five steps: image acquisition, feature extraction, feature matching, parallax calculation, and depth map generation; Step 4: Special effects simulation; process the different particle systems corresponding to the smoke emitted by the jade seal of the Qing Dynasty, the flame of the Changxin Palace lantern, and the ignition flame special effects, so that the smoke can produce a hazy visual effect, and the flame can see the smoke produced when it is ignited; optimize the interaction between smoke and other objects and users, and add collision effects to the smoke to improve the authenticity of the interaction; based on the content of pouring wine from the wine jar in the virtual and real scenes, perform special effects processing based on fluid simulation on some container exhibits, and achieve a water flow simulation effect close to reality; The special effect simulation includes a particle simulation method, wherein the particle simulation method includes the following steps: 1) During initialization, various properties of each particle are initialized, including the particle's speed and coordinates. An index mechanism is established to allow new particles to point to the index mechanism and add new particles to the stack of active particles. 2) The properties of each particle are transmitted to the graphics processing unit (GPU); the data is stored in the graphics card. The two important properties of the particle, speed and coordinates, are stored in the cache of the graphics card. They are grouped together and a backup cache of the same size is created so that they can exchange data with each other. 3) Update the properties of each particle; during the processing, the speed and coordinate data of the particle are read from the cache area of ​​the graphics card, which stores the two important properties of the particle. According to the parameter changes of each frame, the coordinates and speed of the particle are continuously updated with reference to the particle's motion equation. The updated new coordinates and speed properties are stored in the backup cache area, and data is exchanged between them. This realizes the update step, so that the particles can move according to a certain rule. 4) Store the particle data in the vertex buffer object VBO and prepare for rendering; 5) If the particle life cycle ends, the index mechanism will be reclaimed.

2. The method according to claim 1, characterized in that In step one, three virtual scenes of different styles are provided for users to view, including a classical Chinese exhibition hall scene, a modern Chinese exhibition hall scene, and a Western exhibition hall scene.

3. The method according to claim 1, characterized in that The specific implementation method in step 2 is: First, the depth camera is used to collect red, green and blue depth image data, and then the data is converted into a common format according to the camera parameters. At the same time, the sparse depth map obtained by the virtual cultural relic model is analyzed. After histogram statistics, depth screening and probability function optimization, the pose of the real cultural relic rough model relative to the depth camera is obtained. Further obtain the position and posture of the depth camera in the world coordinate system; fix the tracker to the depth camera, and read the depth camera position and posture in the graphics engine; By combining the above two postures, we can get the real posture of the rough model of the real cultural relic relative to the world coordinate system. By setting the posture of the virtual cultural relic model in the graphics engine to be consistent with the real posture of the rough model of the real cultural relic relative to the world coordinate system, we can realize the virtual-real mapping of the cultural relic.

4. The method according to claim 1, characterized in that: The gesture recognition method based on binocular vision includes the following five steps: 1) Use a temporal sliding window to extract action clips; 2) Use scale-invariant feature transformation method to extract key information from each segment; 3) Divide the key information features of each segment into dynamic and static ones as the input of the classifier; 4) For each classification operation input, effectively classify it and optimize the starting frame; 5) Use non-maximum suppression method to remove repeated fragments.

5. The method according to claim 1, characterized in that: The special effects simulation also includes a fluid simulation method, which specifically includes the following steps: Step 1: Establish a model; Based on the particle method, the fluid is regarded as a collection of a large number of particles, each particle represents a small part of the fluid, and the behavior of the fluid is simulated by modeling the movement and interaction of the particles; Step 2: Numerical solution: Based on the Euler method, for each particle, first calculate the acceleration of the next time step according to the current position and velocity as well as the external force field, then update the velocity and position of the particle according to the calculated acceleration, and then repeat the above steps to continuously iterate and update the state of the particle to simulate the change in time; Step 3: Particle rendering based on point sprites; each particle is represented as a point on the screen and rendered according to the particle's properties, including position, color, and size.

6. A mixed reality based display interaction device using the method according to any one of claims 1 to 5, characterized in that: Includes the following modules: Virtual scene roaming module: It is used for users to immerse themselves in the virtual scene and obtain a realistic exhibition hall tour experience; the position of the real exhibition stand is calibrated with the position of the virtual exhibition stand so that the two overlap. When the user picks up the 3D printed rough model or puts the rough model back on the real exhibition stand, the virtual cultural relics will also appear in the corresponding position on the virtual exhibition stand; three different styles of virtual scenes are also provided for users to view; Cultural relic virtual-to-real mapping module: used to achieve virtual-to-real mapping of cultural relics; firstly, the depth camera is used to collect red, green and blue depth image data, and then the sparse depth map obtained by the virtual cultural relic model is analyzed to obtain the pose of the real cultural relic rough model relative to the depth camera. Combined with the pose of the depth camera, the real pose of the real cultural relic rough model in the world coordinate system is obtained, and the pose of the virtual cultural relic model in the graphics engine is set to be consistent with the real pose of the real cultural relic rough model relative to the world coordinate system, so as to achieve virtual-to-real mapping of cultural relics; The intelligent gesture interaction module maps the user's hand gesture in the real space to the corresponding virtual scene, and presents it in the mixed reality head-mounted display to achieve the intelligent gesture interaction effect; the binocular camera of the mixed reality head-mounted display collects the left and right eye visual images of the user's hand movements, and the gesture recognition method based on binocular vision uses the parallax information of the collected left and right eye visual images to generate a depth image through a stereo vision algorithm, in which each pixel in the image represents the distance from the corresponding point of the object to the camera, and then through joint point extraction, the gesture is converted into a digital feature to obtain the spatial position of the hand joint point, and then different joint point combinations are used to track and identify specific hand gestures. The stereo vision algorithm includes five steps: image acquisition, feature extraction, feature matching, parallax calculation, and depth map generation; Special effects simulation module: used to achieve close to realistic smoke, flame and water flow simulation effects; based on the jade seals, Changxin Palace lanterns and ignition contents of the Qing Dynasty in the virtual and real scenes, this module processes different particle systems corresponding to the special effects of the smoke emitted from the jade seals of the Qing Dynasty, the flames of the Changxin Palace lanterns and the ignition flames, so that the smoke can produce a hazy visual effect and the flames can see the smoke produced when they are ignited; this module also optimizes the interaction between smoke and other objects and users, and adds collision effects to the smoke to improve the authenticity of the interaction; based on the wine jar pouring content in the virtual and real scenes, this module performs special effects processing based on fluid simulation on some container exhibits, achieving close to realistic water flow simulation effects.

Images