Microscopic biological three-dimensional space mapping and multi-modal interaction method based on mixed reality

Abstract

The application discloses a microorganism three-dimensional space mapping and multi-modal interaction method based on mixed reality, relates to the field of computer graphics and virtual reality, and comprises the following steps: generating a parameterized virtual cell model with behavior labels and physical attributes based on cell biology characteristic data, and outputting model data; establishing a space anchor mapping relationship based on model data and biological activity habits of target cells, and adaptively anchoring the virtual cell model in a horizontal plane or a suspended space of a real environment; outputting a differentiated interaction instruction based on cell behavior attributes according to scene data after anchoring and user gesture recognition results; and in response to the differentiated interaction instruction and user viewpoint data, real-time calculation and output of visual rendering data and physical feedback data. The application restores biological characteristics through parameterized modeling, realizes adaptive layout of virtual objects in a real space, and efficiently realizes dynamic interaction feedback conforming to biological principles by defining a multi-modal gesture instruction set.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and virtual reality, and in particular to a method for microscopic biological three-dimensional spatial mapping and multimodal interaction based on mixed reality. Background Technology

[0002] In the field of life sciences and health education, traditional methods of displaying microscopic biological structures such as immune cells mainly rely on two-dimensional images, videos, or static three-dimensional models, which are insufficient to intuitively present their dynamic behavior and complex functions. In recent years, with the development of extended reality (XR) technology, virtual reality (VR) and mixed reality (MR) have been gradually introduced into science education applications. For example, virtual and real fusion scenes are constructed based on devices such as the Microsoft HoloLens 2. Through spatial positioning, holographic rendering, and basic gesture tracking, the visualization and overlay of microscopic models in the real environment are realized, initially enhancing the user's sense of immersion.

[0003] However, existing technologies still have significant shortcomings: First, the interaction logic is simplistic, typically supporting only general operations such as grasping and translation, lacking differentiated interaction mechanisms tailored to the biological characteristics of immune cells (such as phagocytosis, deformation, and targeted attack), making it difficult to convey their core functions through operational feedback; second, the spatial mapping mechanism is simplistic, with microscopic objects often suspended in physical space as isolated holograms, failing to establish semantic connections or intelligent mapping between the microscopic biological environment and the macroscopic user scenario; third, the visualization presentation lacks biological dynamism, with most models employing rigid geometric structures, unable to simulate the flexible deformation and activity response of cell membranes in real physiological processes, weakening the authenticity and educational effect of scientific expression.

[0004] The aforementioned shortcomings limit the application value of XR technology in in-depth popularization of life sciences, and there is an urgent need for a new interactive display method that can integrate biological behavioral logic, dynamic deformation modeling and intelligent spatial mapping. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] Therefore, this invention aims to solve the problems of single interactive methods, lack of biological characteristic feedback, and rigid spatial mapping in existing microscopic biological science popularization displays, and proposes a method for three-dimensional spatial mapping and multimodal interaction of microscopic organisms based on mixed reality.

[0007] Another objective of this invention is to propose a microscopic biological three-dimensional spatial mapping and multimodal interaction system based on mixed reality.

[0008] The third objective of this invention is to provide a computer device.

[0009] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0010] To achieve the above objectives, this invention proposes a method for microscopic biological three-dimensional spatial mapping and multimodal interaction based on mixed reality, comprising: Generate parameterized virtual cell models with behavioral labels and physical attributes based on cell biological feature data, and output model data; Based on the model data and the biological activity habits of the target cells, a spatial anchor point mapping relationship is established, and the virtual cell model is adaptively anchored to the horizontal plane or suspended space of the real environment. Based on the anchored scene data and user gesture recognition results, output differentiated interaction commands based on cell behavior attributes; Responding to differentiated interaction commands and user viewpoint data, it calculates and outputs visual rendering data and physical feedback data in real time.

[0011] The microscopic biological three-dimensional spatial mapping and multimodal interaction method based on mixed reality in this invention embodiment may also have the following additional technical features: In one embodiment of the present invention, a parameterized virtual cell model with behavioral labels and physical attributes is generated based on cell biological feature data, and the model data is output, including: A basic geometric model of the cell was constructed using 3D modeling software, and the cell nucleus, cell membrane, and staining features were extracted as key feature points. Based on the key feature points, the cell membrane material is parameterized, and the shader is used to simulate the translucent flow and edge lighting effects of the cell membrane, and the physical collision volume of the model is defined. A biological behavior tag library is established, and corresponding behavioral attribute tags are bound to different cell types based on the biological behavior tag library. Finally, complete virtual cell model data containing geometric structure, material parameters, collision volume and behavioral tags is output.

[0012] In one embodiment of the present invention, a spatial anchor mapping relationship is established based on the model data and the biological activity habits of the target cells, and the virtual cell model is adaptively anchored to the horizontal plane or suspended space of the real environment, including: Based on virtual cell model data and real-world environment grid data collected by mixed reality devices, horizontal and vertical planes are identified to generate structured physical space semantic information. Based on the physical space semantic information and the biological activity habit tags carried in the model, the initial anchoring position and orientation of the cell model in the real space are calculated and output through spatial anchor point and full space placement algorithms. Based on the anchoring position and real-time user gaze direction and distance data, the hovering pose of the cell model is dynamically adjusted, and optimized interactive comfort zone positioning data is output.

[0013] In one embodiment of the present invention, based on the anchored scene data and the user gesture recognition result, a differentiated interaction command based on cell behavior attributes is output, including: Input anchor scene data and hand skeleton data output by gesture recognition interface model, parse the common gesture type currently performed by the user, and generate preliminary interaction intent signals; Based on the preliminary interaction intent signal and the behavioral attribute labels of the selected cell model, combined with its relative spatial relationship with other objects in the environment, it is determined whether the specific biological interaction conditions are met, and semantically enhanced interaction context data is generated. Based on the interaction context data, the general operation logic and biological behavior rules are integrated to output the final differentiated interaction instructions, which are used to drive the rendering and execution of the corresponding visual or physical responses.

[0014] In one embodiment of the present invention, the input anchoring scene data and the hand skeleton data output by the gesture recognition interface model are used to parse the common gesture type currently being performed by the user to generate a preliminary interaction intent signal, including: The gesture recognition interface model is initialized based on the Mixed Reality Development Kit (MRTK), and standardized gesture recognition capability configuration data is output. The interaction logic script loads and calls the gesture recognition capability configuration data, subscribes to the hand skeleton data stream provided by MRTK in real time, and outputs raw input data containing joint coordinates and gesture state. Based on the anchored scene data and the original input data, the user's current gesture type and operation intention are analyzed to generate structured gesture interaction signals.

[0015] In one embodiment of the present invention, in response to differentiated interaction commands and user viewpoint data, visual rendering data and physical feedback data are calculated and output in real time, including: Receive differentiated interaction commands and cell model state data to generate material update parameters and physical force signals; Based on the material update parameters, and combined with user viewpoint data, the shader and particle system are driven to adjust the visual appearance of the cell model in real time, and output the corresponding rendering frame data; and, Using the physical force signals and the collision volume and mass attributes of the model, the physics engine is invoked to calculate the motion response and output position, velocity and deformation data that conform to physical laws, which are used to synchronously update the visual presentation and the user's tactile / motion perception feedback.

[0016] In one embodiment of the present invention, the method further includes utilizing HoloLens 2's spatial positioning and eye-tracking capabilities to achieve full-range adaptive placement and hovering of the model in physical space, including: Create a UWP project in Unity, set the target platform to HoloLens 2, the architecture to ARM64, and ensure that the system version is compatible with the space awareness function; Import Mixed Reality Toolkit to automatically configure MixedRealityToolkit and Playspace, enabling spatial anchor points, gesture recognition, and environmental understanding capabilities; Import the parameterized 3D model constructed based on the biological characteristics of immune cells, and perform lightweight optimization to adapt to device performance; HoloLens 2's spatial scanning function collects environmental meshes in real time, identifies horizontal and vertical planes, and generates semantic spatial structures that can be used to place virtual objects. Combining the biological activity habits of cell models, and using spatial anchoring and full-space placement algorithms, the model is intelligently anchored to a desktop, wall, or in the air, and its hovering position and orientation are dynamically adjusted based on eye tracking and distance detection. Test the model's stability, visibility, and interactive comfort in physical space on real devices, and optimize the anchoring logic and rendering performance based on user feedback.

[0017] To achieve the above objectives, another aspect of the present invention proposes a microscopic biological three-dimensional spatial mapping and multimodal interaction system based on mixed reality, comprising: The parameterized model library construction module is used to generate parameterized virtual cell models with behavioral labels and physical attributes based on cell biological feature data, and output model data; The intelligent mapping module is used to establish a spatial anchor point mapping relationship based on the model data and the biological activity habits of the target cells, and to adaptively anchor the virtual cell model to the horizontal plane or suspended space of the real environment. The multimodal interaction control logic construction module is used to output differentiated interaction commands based on cell behavior attributes according to the anchored scene data and user gesture recognition results. The rendering and feedback output module is used to respond to differentiated interaction commands and user viewpoint data, and to calculate and output visual rendering data and physical feedback data in real time.

[0018] The present invention relates to a method and system for three-dimensional spatial mapping and multimodal interaction of microscopic organisms based on mixed reality. This method restores biological features through parametric modeling, uses spatial mapping algorithms to achieve adaptive layout of virtual objects in real space, and effectively realizes dynamic interactive feedback that conforms to biological principles by defining a multimodal gesture command set.

[0019] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the mixed reality-based microscopic biological three-dimensional spatial mapping and multimodal interaction method as described in the first aspect embodiment.

[0020] To achieve the above objectives, the fourth aspect of this application proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the mixed reality-based microscopic biological three-dimensional spatial mapping and multimodal interaction method as described in the first aspect embodiment.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a method for microscopic biological three-dimensional spatial mapping and multimodal interaction based on mixed reality according to an embodiment of the present invention; Figure 2 This is the first immune cell knowledge graph according to an embodiment of the present invention; Figure 3 This is a second immune cell knowledge graph according to an embodiment of the present invention; Figure 4 This is the third immune cell knowledge graph according to an embodiment of the present invention; Figure 5 These are HoloLens 2 screen demonstration images according to an embodiment of the present invention; Figure 6 This is a schematic diagram of gesture operation according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a microscopic biological three-dimensional spatial mapping and multimodal interaction system based on mixed reality according to an embodiment of the present invention; Figure 8 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] The following description, with reference to the accompanying drawings, describes a method, system, device, and storage medium for microscopic biological three-dimensional spatial mapping and multimodal interaction based on mixed reality, according to embodiments of the present invention.

[0026] Figure 1 This is a flowchart of a method for microscopic biological three-dimensional spatial mapping and multimodal interaction based on mixed reality according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes, but is not limited to, the following steps: S1 generates a parameterized virtual cell model with behavioral labels and physical attributes based on cell biological feature data, and outputs the model data.

[0027] Specifically, this invention constructs a parameterized model library based on biological features to build microscopic biological models in a 3D engine, and extracts key visual elements based on biological features for parameterized definition. Classification and modeling are performed based on the morphology of the cell nucleus (e.g., mononuclear, segmented nucleus), cell membrane texture (e.g., fluid, crystalline), and surface marker features of different immune cells.

[0028] This embodiment uses the fourth and sixth editions of *Cell Biology*, *Medical Cell Biology*, and the *Journal of Cell Biology* as primary references, such as... Figure 2 , Figure 3 and Figure 4 As shown, a total of 18 immune cell types were compiled, and their corresponding information, including "abbreviations and full names in Chinese and English," "cell family," "cell function," "marker markers," "staining color," "size," and "reference diagram," was also compiled, along with their cell relationships and evolutionary atlases. Furthermore, the content on immune cell types was further organized and summarized, resulting in several important categories.

[0029] Broadly speaking, the original source of immune cells is hematopoietic stem cells (considered as undifferentiated stem cells, or pluripotent stem cells), and the subsequent major branches are "stem cell series", "lymphocyte series" and "bone marrow cell series".

[0030] The lymphocyte series, led by lymphoprogenitor cells. Main functions: responsible for humoral and cellular immunity, fighting pathogens and regulating immune responses. Cell types included: B cells, T cells, NK cells, etc. Immune functions: specific immune responses, recognition of specific antigens, and memory responses.

[0031] Bone marrow cells are a series of cells primarily composed of bone marrow progenitor cells. Their main functions include oxygen transport, infection control, and blood clotting. Cell types included are: erythrocytes, neutrophils, eosinophils, basophils, and monocytes. Immune functions include: non-specific immune responses such as inflammation and phagocytosis.

[0032] Specifically, the information displayed, including the "abbreviations and full names in Chinese and English", "cell series", "cell function", "marker", "staining color", "size", and "reference diagram", was effectively sorted and summarized.

[0033] Figure 2 , Figure 3 and Figure 4 This work meticulously outlines the broad lineage of immune cells, providing a comprehensive knowledge framework supported by biomedical and cell biology research. It offers ample input and a solid knowledge base to guide subsequent public education on immune cells. Furthermore, this knowledge map ensures that the information delivered to the public is based on original scientific accuracy and traceability, closely aligned with research topics and cutting-edge advancements. It also provides a comprehensive understanding of the diverse roles of stem cells and specialized immune cells in health and disease (and cellular function).

[0034] One embodiment of this invention classifies cells into stem cell series and lymphocyte series, etc., describes in detail the functions of these cells in the immune system, and identifies the major and minor markers for each cell type. This information is extremely valuable to those outside the biomedical field, enabling the establishment of the fastest and most preliminary localization perspective. Furthermore, this atlas can help science communicators and subsequent practical and design outputs of this research to be more precise and traceable.

[0035] Specifically, in some implementations, generating parameterized virtual cell models with behavioral labels and physical attributes based on cell biological characteristic data is a core step in constructing a microscopic biological three-dimensional spatial mapping and multimodal interaction system in mixed reality (MR). This step integrates morphological, physiological, and behavioral data of cells to construct virtual cell models with realism and interactivity, providing basic data support for subsequent spatial anchoring and user interaction.

[0036] The process begins by constructing a basic geometric model of the cell using 3D modeling software (such as Blender, Maya, or ZBrush), focusing on extracting the cell nucleus, cell membrane, and staining features as key feature points. These feature points not only define the cell's geometric structure but also serve as reference benchmarks for subsequent behavioral modeling and physical property binding. Further, based on these key feature points, the cell membrane material is parametrically set, typically using physically based rendering (PBR) techniques combined with shaders to simulate the translucent flow and edge lighting effects of the cell membrane. For example, diffuse reflectance can be set to 0.2–0.4, specular reflectance to 0.6–0.8, and alpha opacity to 0.5–0.7 to enhance visual realism.

[0037] The model needs to define a collision volume, typically approximated using a bounding box or convex hull, to ensure that interactions with user gestures or other virtual objects in a mixed reality environment conform to physical laws. Furthermore, a biological behavior tag library is established, which can include behavioral attributes such as "phagocytosis," "chemotaxis," and "division," and is bound to corresponding behavioral tags based on cell type (e.g., T cells, B cells, macrophages). The final output model data should include geometric structure, material parameters, collision volume, and behavioral tags, forming a structured virtual cell dataset.

[0038] This step provides the data foundation for subsequent spatial anchoring and interactive control, and is particularly suitable for scenarios such as medical education, biological research, and virtual laboratories. Through parametric modeling, the system can quickly generate various cell models and achieve differentiated interactive responses under different user viewpoints and gesture inputs, thereby enhancing immersion and the naturalness of interaction.

[0039] Furthermore, S1 includes: S11 uses 3D modeling software to construct the basic geometric model of the cell and extracts the cell nucleus, cell membrane and staining features as key feature points.

[0040] Specifically, in some implementations, a basic geometric model of the cell is constructed using 3D modeling software, and the cell nucleus, cell membrane, and staining features are extracted as key feature points. This step, based on morphological data from cell biology, uses modeling tools (such as Blender, Maya, ZBrush, or 3D Slicer) to digitally reconstruct the macroscopic structure of the cell, thereby providing a geometric basis for subsequent virtual cell behavior modeling and physical property binding.

[0041] The modeling process typically includes the following steps: First, microscopic imaging data (such as confocal microscope images, electron microscope slides, or CT reconstruction data) is imported into 3D modeling software. Image segmentation algorithms (such as MarchingCubes or Voxelize) are then used to generate a surface mesh model of the cell. The model's topology must ensure that the number of triangular facets is within a certain range to balance visual accuracy and real-time rendering performance. Further, using manual or automatic annotation tools, the geometric center points and boundary contours of key structures such as the cell nucleus, cell membrane, and chromosomes are extracted to form a set of key feature points, which are used for subsequent binding of behavioral logic and physical attributes.

[0042] The extraction of key feature points needs to meet certain accuracy requirements. For example, the coordinate error of the center point of the cell nucleus should be less than [missing information]. The boundary contour of the cell membrane needs to be within Sampling is performed at a high resolution to ensure sufficient biological morphological realism in the model during interaction. Furthermore, the modeling software must support exporting to common 3D formats (such as FBX, OBJ, or GLTF) and be compatible with the import standards of mainstream mixed reality development platforms such as Unity or Unreal Engine.

[0043] This step is widely used in medical education, biological research, and mixed reality interactive systems. For example, in virtual anatomy teaching, teachers can interact with cell models through gestures to observe changes in their internal structures; in research environments, researchers can simulate the dynamic behavior of cells under different stimuli based on parametric models, thereby assisting in experimental design and data analysis.

[0044] This step, through high-precision geometric modeling and key feature point extraction, endows the virtual cell model with a structured and programmable morphological foundation, enabling subsequent operations such as material simulation, behavior binding, and spatial anchoring to have clear geometric references and logical support, thereby enhancing the biological realism and interactive immersion of the entire system.

[0045] S12, based on key feature points, parameterizes the cell membrane material, uses shaders to simulate the translucent flow and edge lighting effects of the cell membrane, and defines the physical collision volume of the model.

[0046] Specifically, in some implementations, parameterizing the cell membrane material based on key feature points and using shaders to simulate the translucent flow and edge lighting effects of the cell membrane, while defining the physical collision volume of the model, are key steps in constructing a high-fidelity virtual cell model.

[0047] In one embodiment, the goal is to design a system that allows for the visualization and differentiation of cells from the microscopic world to the macroscopic world through cellular feature points. This is achieved by focusing on three key feature points: the cell nucleus, the cell membrane, and the stained cell. First, different immune cells exhibit significant differences in their nuclei, ranging from single nuclei to multiple nuclei, and even bilobed or bivalve-shaped nuclei. This is a crucial characteristic marker. Second, their cell membranes vary significantly, exhibiting different structures such as soft, fluid textures, crystalline or coral-like appearances, and multifaceted extensions and dendritic branching. These are also significant differentiating features. Furthermore, in actual cell biology and observation, cells need to be stained before presentation. Observing their different morphologies and states allows for the identification of different staining results for each cell type, sometimes displaying different colors. This process can also be artistically manipulated and magnified to extract and construct cellular features. The parameterization process primarily involves referencing the size and positional relationships of the cell nucleus, cell membrane, and cytoplasm, using the Blender software for modeling.

[0048] This step extracts cell nuclei, cell membranes, and staining features as key feature points, endowing cell membrane materials with adjustable optical and physical properties, thereby achieving realistic visual performance and natural interactive feedback in mixed reality (MR) environments.

[0049] Parametric settings for cell membrane materials are typically based on the Shader Graph in Unity or the Material Graph in Unreal Engine. By defining parameters such as diffuse reflectance, specular reflectance, opacity, and subsurface scattering, the translucent properties of the cell membrane can be simulated. For example, diffuse reflectance can be set to 0.3–0.6 to reflect the cell membrane's ability to scatter light; opacity is dynamically adjusted between 0.4 and 0.8 based on the cell membrane thickness and the characteristics of the biological tissue. Furthermore, edge lighting is calculated using the angle between the normal direction and the viewing direction, as shown in the formula: ,in The angle between the line of sight and the surface normal is used to enhance the visual prominence of the cell edge and improve the user's perception of three-dimensionality and dynamism.

[0050] Furthermore, the physical collision volume is defined using a bounding box or convex hull to ensure that the interaction between the model and user gestures or other virtual / real objects in MR space conforms to physical laws. In Unity, the `Mesh Collider` or `Capsule Collider` components are typically used, combined with the geometry of the cell membrane, to set reasonable collision radii and mass parameters, such as mass... Collision radius To adapt to the motion response of virtual cells in space.

[0051] This step is of great significance in practical applications, especially in medical education, biological research, and MR-assisted surgical training. Through precise material simulation and physical collision modeling, users can perform operations such as grasping, rotating, and scaling virtual cells, while obtaining visual and tactile feedback that conforms to the characteristics of biological tissues, thereby enhancing the realism and immersion of the interaction.

[0052] S13, establish a biological behavior tag library, and bind corresponding behavior attribute tags to different cell types based on the biological behavior tag library, and finally output complete virtual cell model data including geometric structure, material parameters, collision volume and behavior tags.

[0053] For example, a biological behavior tag library is established to mark different cells as objects with specific behavioral attributes, such as marking neutrophils as having phagocytic attributes and T cells as having aggressive attributes.

[0054] Specifically, a biological behavior tag library oriented towards the functional characteristics of immune cells is established. Different types of immune cells are structurally and semantically labeled, defining them as intelligent interactive objects with specific biological behavioral attributes. Specifically, based on the actual physiological functions of cells in the immune response, one or more standardized behavioral tags are assigned to each cell type: for example, neutrophils are labeled with attributes of "phagocytosis," "chemotaxis and migration," and "degranulation"; cytotoxic T cells (CTLs) are labeled with attributes of "target recognition," "attack and killing," and "perforin release"; macrophages are labeled with attributes of "antigen presentation," "phagocytic clearance," and "inflammatory cytokine secretion"; and B cells are labeled with attributes such as "antibody secretion" and "memory formation." This tag library adopts an extensible ontology structure design, supporting the addition, deletion, and combination of behavioral attributes, and is bound to the cell's three-dimensional morphological model, dynamic animation clips, and physical response rules. During mixed reality interaction, the system can call up the corresponding visual feedback (such as membrane deformation, pseudopodia extension), spatial animation (such as chemotactic movement path), or interactive logic (such as the disappearance of the target pathogen and the playback of endocytosis animation after triggering "phagocytosis") associated with the current user operation intention and scene context in real time. This enables differentiated and semantic human-computer interaction based on real biological mechanisms, significantly improving the scientific nature and immersiveness of popular science content.

[0055] S2 establishes a spatial anchor mapping relationship based on model data and the biological activity habits of target cells, and adaptively anchors the virtual cell model to the horizontal plane or suspended space of the real environment.

[0056] Specifically, in some implementations, establishing a spatial anchor mapping relationship based on the model data and the biological activity habits of the target cells, and adaptively anchoring the virtual cell model to the horizontal plane or suspended space of the real environment through a full-space placement algorithm, is a core step in realizing the visualization and interaction of microscopic organisms in mixed reality (MR). The technical implementation principle of this step is based on spatial semantic understanding and an adaptive positioning mechanism driven by biological behavior.

[0057] This step first relies on the environmental perception capabilities of mixed reality devices (such as HoloLens 2). Through its spatial scanning function, it collects real-time mesh data of the real environment and uses the Unity engine in conjunction with the Mixed Reality Toolkit (MRTK) for spatial semantic segmentation, identifying horizontal surfaces (such as desktops and floors) and vertical surfaces (such as walls and device surfaces). Subsequently, based on biological activity tags (such as movement patterns, levitation tendencies, and attachment capabilities) carried in the virtual cell model, and combined with spatial semantic information, the system calls the Full-Space Placement Algorithm to calculate the anchoring position and orientation. This algorithm is typically based on a spatial availability assessment model, calculating a stability index for candidate placement locations. ,in The score represents the availability of spatial anchor points. Indicates the distance from the user's viewpoint. To prevent the division by zero of tiny constants, the most suitable anchor point is selected in physical space.

[0058] Establishing spatial anchor point mapping relationships requires consideration of several key parameters, including but not limited to the anchor point confidence threshold (default 0.85), minimum anchoring area (recommended ≥0.05 m²), user viewpoint distance threshold (typically set to 1.5~3.0 m), and the normal angle between the model and the environment surface (recommended ≤15° to ensure stable attachment). Furthermore, the full-space placement algorithm needs to be dynamically adjusted based on the physical properties of the cell model (such as mass and collision volume) to ensure the model's natural representation in different spatial structures.

[0059] This step is widely used in medical education, biological research, and virtual laboratories. For example, in a teaching environment, the system can automatically anchor suspended cell models in the air in front of the user's line of sight, while attached cells are preferentially placed on the desktop or petri dish surface, thereby enhancing the immersiveness of the interaction and the intuitiveness of the operation.

[0060] The technical effect of this step is that by combining biological behavioral characteristics with spatial semantic information, it enables intelligent and adaptive anchoring of the virtual cell model in a mixed reality environment, significantly improving the stability of the model and the comfort of user interaction, and providing a reliable spatial foundation for subsequent multimodal interaction.

[0061] One embodiment of this invention uses HoloLens 2 to display three-dimensional models of immune cells and allows users to interact with these cells through gestures, achieving an edutainment purpose. The user interface is simplified as much as possible, using the most intuitive and user-friendly interface to conform to user habits and enable natural interaction with virtual cells, including but not limited to pinching, rotating, zooming, and other gesture operations. Furthermore, based on the HoloLens 2's global positioning system hardware and software support, it enables placement, hovering, and observation within the entire visible range. The previously created three-dimensional models of immune cells further ensure the scientific accuracy and good visual appeal of these cell models.

[0062] To deepen the design of scene space, materials, and processes, the first consideration should be how to accurately reproduce the complex structure and dynamic function of immune cells using precise 3D modeling techniques. Employing advanced 3D rendering technology and high-resolution texture materials is crucial, as it significantly enhances the realism and detail of the model, making the simulated immune cells not only scientifically accurate but also visually vivid. The interim goal is to leverage technological advantages and innovative popular science methods to transform this complex scientific research into knowledge that the public can easily access and understand.

[0063] Simultaneously, the selected display materials and technological implementation processes must support the technical characteristics of HoloLens 2. For example, considering the device's tracking and response speed, the design should use optimized models and textures, avoiding overly complex graphics processing to ensure smooth interaction and responsiveness. Through these in-depth technical processes and material selections, the potential of virtual reality technology in education and popular science can be maximized, making the project more effective and engaging in practice.

[0064] In summary, to achieve the ultimate goal and present a perfect effect, the development process not only focused on the development of technical prototypes and user experience testing, but also strived to find new ways to express popular science content. This aims to showcase the mysteries of life sciences and stimulate public interest and curiosity in scientific exploration. The specific development process is as follows: (1) Development environment setup: First, create a new 3D project in Unity specifically for HoloLens. Set the project to Universal Windows Platform, select HoloLens as the target device, and ensure that the platform version is at least 10.0.18362.0 and the architecture is adjusted to ARM64 to be compatible with the processing power of HoloLens 2.

[0065] (2) Importing and Configuring Tools: Next, import version 2.4.0 of the Microsoft.MixedReality.Toolkit.Unity.Foundation package into the project. This toolkit provides the necessary infrastructure and presets for HoloLens applications. After importing, select "Add to Scene and Configure" under the Mixed Reality Toolkit menu in Unity to automatically configure the Mixed Reality Toolkit and Mixed Reality Playspace in the scene, paving the way for subsequent development.

[0066] (3) Scene and Model Design: Design a virtual environment in Unity that reflects the operational background of the immune system and serves as the main scene for user interaction. Import immune cell models obtained from professional 3D resource libraries or designed by ourselves, and adjust their size and texture to ensure optimal visual effects on HoloLens 2.

[0067] (4) Programming and Interaction Logic: The core of the development includes writing the interaction logic, using Unity's C# API and HoloLens 2 SDK to implement spatial positioning and gesture recognition. This allows users to interact with the 3D immune cell model through intuitive gestures such as pinching, rotating, and swiping. In addition, the Mixed Reality Toolkit is used to implement interactive feedback between the user's gestures and the virtual cells, such as haptic, sound, and light effects, to enhance the immersive experience.

[0068] (5) Testing and Optimization: During development, the application is regularly deployed and tested on HoloLens 2 physical devices or simulators to ensure that all functions work properly and that the interaction flow meets design expectations. Feedback is collected through user testing, and the user interface and interaction design are adjusted according to actual operation to optimize performance and ensure that the application runs smoothly on HoloLens 2.

[0069] (6) Deployment and Release: Finally, the application will be packaged using Unity and deployed to HoloLens 2 for final testing to verify functional completeness and performance stability. After successful testing, the application can be submitted to the Microsoft Store or distributed through other channels to enable the target user group to access and use this educational tool.

[0070] Through the above steps, this part of the development process was completed, enabling the importing cells to be placed, hovered, and observed within the entire visible space. This project transforms the idea into a concrete, interactive virtual reality experience, which not only helps improve public understanding of immune cells but also creates an educational and entertaining mixed reality application, effectively enhancing public understanding and interest in the workings of the immune system. In this process, leveraging Unity's powerful 3D and interactive development capabilities, combined with HoloLens 2's unique spatial positioning and gesture recognition technology, advanced learning tools can be implemented, providing the project's audience with an unprecedented science education experience.

[0071] HoloLens2 screen demonstration of this invention embodiment is as follows: Figure 5 As shown.

[0072] Furthermore, S2 includes: S21, based on virtual cell model data and real-world environment grid data collected by mixed reality devices, identifies horizontal and vertical planes and generates structured physical space semantic information.

[0073] Specifically, this step aims to identify horizontal and vertical planes based on virtual cell model data and real-world environmental grid data collected by mixed reality (MR) devices, and generate structured physical spatial semantic information. This process is fundamental to enabling virtual cells to intelligently anchor and interact in real-world space, and has significant technological value and application implications.

[0074] This step first relies on the environmental perception capabilities of mixed reality devices (such as HoloLens 2), using their built-in depth cameras and spatial scanning modules to collect real-time 3D mesh data of the real environment. This mesh data is typically represented as point clouds or triangular patches, containing the geometric contours and surface normal information of objects in space. Subsequently, the system uses plane detection algorithms (such as RANSAC or plane segmentation methods based on normal clustering) to process the mesh data, identifying horizontal planes (such as desktops and floors) and vertical planes (such as walls and columns) with significant normal consistency. During the identification process, normal angle thresholds are typically set (e.g., angles less than 10° with the direction of gravity are considered horizontal planes, and angles greater than 80° are considered vertical planes), and plane area and continuity indicators (e.g., minimum area threshold of 0.1 m², continuity error less than 5%) are used for filtering and optimization.

[0075] The system classifies and structures the identified planes using spatial semantic labels. For example, horizontal planes are labeled `PlaneType.Horizontal` and vertical planes are labeled `PlaneType.Vertical`, and key parameters such as their position, normal direction, and bounding box are recorded. Furthermore, the system evaluates the planes' stability and interactivity, for example, by using indicators such as surface curvature (`curvature<0.05`) and surface material reflectance (`reflectance>0.3`) to determine whether they are suitable for anchoring virtual cell models.

[0076] This process is widely used in biological teaching, scientific research visualization, and medical training. For example, in a laboratory environment, users can use MR devices to anchor virtual cell models onto a lab bench or the inner wall of an incubator, enabling immersive observation and manipulation of cell structure and behavior. In medical training, virtual cells can be anchored onto the surface of anatomical models to help doctors understand the distribution and activity patterns of cells in tissues.

[0077] By leveraging structured physical spatial semantic information, a precise spatial reference framework is provided for subsequent virtual cell anchoring and interaction, thereby enhancing the system's spatial perception capabilities and the naturalness of interaction. Its innovation lies in the fusion of biological semantics and physical spatial semantics, enabling virtual objects to adaptively arrange and dynamically respond in the real-world environment.

[0078] S22, based on physical space semantic information and biological activity habit tags carried in the model, calculates and outputs the initial anchoring position and orientation of the cell model in real space through spatial anchor point and full space placement algorithm.

[0079] Specifically, in some implementations, based on the physical space semantic information and the biological activity habit tags carried in the model, the initial anchoring position and orientation of the cell model in real space are calculated and output through spatial anchoring and full-space placement algorithms. This is a key step in achieving natural, stable, and biologically-behavioral-compliant positioning of the virtual cell model in a mixed reality environment. This step integrates a multimodal decision-making mechanism that combines spatial perception, semantic understanding, and behavior-driven approaches, ensuring that the placement of the virtual cell model in real space conforms to both the structural characteristics of the physical environment and its biological behavioral characteristics.

[0080] The process first relies on environmental grid data collected by mixed reality devices (such as HoloLens 2) through spatial scanning to identify horizontal surfaces (e.g., desktops, floors) and vertical surfaces (e.g., walls, columns), and construct structured physical spatial semantic information. Then, combining pre-set biological activity habit tags in the virtual cell model (e.g., whether it has swimming ability, whether it depends on an attachment surface), the system invokes the Full Spatial Placement Algorithm to intelligently decide the model's initial anchoring position and orientation. This algorithm typically evaluates candidate placement positions based on a spatial availability scoring model, calculating their spatial relationships with the user's viewpoint, gesture operation area, and environmental obstacles to ensure the model's visibility and interactive accessibility.

[0081] The calculation of spatial anchor points typically involves the following key parameters: the normal vector angle threshold of the anchoring region (generally set to...). To ensure the model is stably attached to the horizontal plane), the distance threshold between the user's viewpoint and the model center (usually set to...). To ensure comfortable interaction), and the minimum safe distance between the model and environmental obstacles (generally 1000 meters). (To avoid visual occlusion and physical collisions). In addition, the initial orientation of the model is usually optimized by calculating the angle difference between the user's viewpoint direction and the model's default viewing direction, using a strategy that minimizes the viewpoint offset.

[0082] This step is widely used in medical education, biological research, and virtual experimental environments. For example, in teaching scenarios, teachers can use gestures to anchor specific types of immune cell models on a table or in the air. The system automatically determines whether to use a suspending placement strategy based on whether the model is mobile, and adjusts its orientation to face the user, thereby enhancing the intuitiveness and interactivity of the teaching demonstration.

[0083] This step effectively solves the problems of unnatural positioning and unintuitive interaction of virtual biological models in complex real spaces. By combining biological behavior and spatial semantic information, it achieves intelligent and adaptive anchoring of the model, significantly improving the immersiveness and scientific nature of mixed reality systems in microscopic biological visualization and interaction.

[0084] S23, based on the anchor position and real-time acquired user gaze direction and distance data, dynamically adjusts the hovering pose of the cell model and outputs optimized interactive comfort zone positioning data.

[0085] Preferably, a gaze tracking and distance detection mechanism is introduced to adjust the hovering position and orientation of the model within the visible range in real time, ensuring that the key model is always in the user's comfortable interaction zone.

[0086] Specifically, a gaze tracking and spatial distance detection mechanism is introduced to construct an adaptive model layout system oriented towards user perceived comfort. Specifically, the eye-tracking sensor built into the head-mounted XR device acquires the user's gaze direction and focal area in real time, and calculates the three-dimensional spatial distance between the user's head and the virtual model using a depth camera or SLAM (Simultaneous Localization and Mapping) module. Based on the aforementioned multimodal perception data, the system dynamically evaluates the visibility, interactive accessibility, and visual focus status of the current model within the user's field of vision.

[0087] Building upon this foundation, the system optimizes the real-time position and orientation of key immune cell models within the visible range (such as cells performing phagocytosis or attack). When a model deviates from the user's primary viewpoint or is too far / too close, exceeding the comfortable interaction range (e.g., the recommended operating distance of 0.5–1.2 meters), the system automatically fine-tunes its hovering position, smoothly moving it into the optimal observation and operation area. Simultaneously, it intelligently rotates the model's orientation according to the user's line of sight, ensuring that its functional structural surfaces (such as the TCR receptor region of T cells and the pseudopodia of neutrophils) are directly facing the user's viewpoint. This mechanism not only avoids the fatigue of frequent head turning or manual model adjustments but also ensures that key biological behaviors are clearly presented from the optimal perspective, significantly enhancing the naturalness, fluency, and cognitive efficiency of the immersive science popularization experience.

[0088] S3 outputs differentiated interaction commands based on cell behavior attributes, according to the anchored scene data and user gesture recognition results.

[0089] Specifically, in some implementations, the core step in this method to achieve semantic, biologically driven interactive control between the user and the virtual cell model is to output differentiated interaction commands based on cell behavior attributes, according to the anchored scene data and user gesture recognition results. This step achieves accurate parsing of user intent and dynamic generation of interactive responses by integrating spatial perception, gesture recognition, and biological behavior rules.

[0090] This step first receives scene data from the spatial anchoring module, including the virtual cell model's position, orientation, and relative spatial relationship with environmental objects in real space. Simultaneously, it subscribes to a real-time hand skeleton data stream from a gesture recognition interface model (such as MRTK) to obtain raw input data containing the 3D coordinates of each joint, gesture state, and motion trajectory. In the interaction logic script, a gesture classification algorithm (such as Euclidean distance and angle threshold judgment based on skeletal points) is used to parse the user's currently executed common gesture type, such as "grab," "swipe," or "zoom," and generates a structured preliminary interaction intent signal.

[0091] Furthermore, the system matches the initial interaction intent signal with the behavioral attribute tags of the selected virtual cell model. The behavioral attribute tag library defines the response rules for different cell types (such as T cells and B cells) under specific interaction conditions, such as whether to allow being "grabbed" or whether to respond to "slide" to simulate cell migration. At the same time, the system combines the relative spatial relationship between the virtual cell and environmental objects (such as organ models and cell culture dishes) to determine whether specific biological interaction conditions are met, such as whether it is within the cell membrane contact range or on the cell movement path.

[0092] This step is widely used in medical education, biological experiment simulation, and scientific research visualization systems. For example, in virtual anatomy teaching, users can use gestures to "grab" T cell models and "drag" them to the surface of specific organs. The system determines whether to allow the cell to "attach" or "migrate" based on its behavioral attributes and spatial relationships, and generates corresponding visual and tactile feedback.

[0093] This step significantly enhances the biological plausibility of the interaction and the user's immersion through a semantically enhanced interactive context data generation mechanism. By integrating general operational logic with biological behavior rules, the virtual cell model not only possesses interactive responsiveness but can also simulate the biological behavior of real cells, thereby enhancing its teaching and research value in mixed reality environments.

[0094] The gesture operation diagram of the present invention is as follows: Figure 6 As shown.

[0095] Furthermore, S3 includes: S31, input anchored scene data and hand skeleton data output by gesture recognition interface model, parse the general gesture type currently performed by the user, and generate preliminary interaction intent signals.

[0096] Specifically, in some implementations, inputting anchored scene data and hand skeleton data output by the gesture recognition interface model, and parsing the common gesture type currently being performed by the user to generate preliminary interaction intent signals, is a key step in enabling natural interaction between users and virtual cell models in a mixed reality environment. This step, based on the gesture recognition capabilities provided by the Mixed Reality Development Kit (MRTK) and combined with scene space anchoring information, performs real-time recognition and intent inference of the user's gesture behavior, thus providing a foundation for the subsequent generation of differentiated interaction commands.

[0097] This step first initializes the gesture recognition interface model using MRTK, configuring gesture recognition capabilities, including but not limited to standard gesture types such as "grab," "pinch," "spread," and "click." In the Unity interaction logic script, the system loads and calls this configuration data, subscribes to the hand skeleton data stream provided by MRTK in real time, and outputs the 3D coordinates of each joint. The system acquires raw input data including the virtual cell model's position, orientation, and relative distance to the user's viewpoint in real space, as well as the gesture status (e.g., "Start", "In Progress", "End"). Simultaneously, it obtains anchoring scene data generated by a spatial anchor mapping algorithm, including the virtual cell model's position, orientation, and relative distance to the user's viewpoint in real space.

[0098] Hand skeleton data is typically updated at a frequency of 60 frames per second (FPS), with joint coordinate accuracy down to the millimeter level (±1mm). Gesture recognition latency is controlled within 50ms to ensure real-time and natural interaction. Anchored scene data includes semantic information about the spatial grid, such as plane type (desktop, wall, air), normal direction (…). ) and distance threshold ( ), used to assist in the contextual determination of gesture intentions.

[0099] This step is widely used in virtual laboratories, medical teaching, and biological research visualization systems. For example, users can use a "pinch" gesture to select a specific cell model, a "spread" gesture to simulate cell division, and a "grab" gesture to move or rotate the cell model to observe its three-dimensional structure. By parsing these gestures and combining them with the anchoring state of the cell models in the scene, the system generates structured interactive signals such as "select," "operate," and "release."

[0100] This step enables the efficient conversion from raw sensor data to semantic interactive intents, providing clear input signals for subsequent biological behavior rule matching and physical feedback calculation. Its value lies in improving the precision and efficiency of user manipulation of microscopic biological models in mixed reality environments, enhancing the system's immersiveness and scientific rigor.

[0101] S32, based on the initial interaction intent signal and the behavioral attribute labels of the selected cell model, combined with its relative spatial relationship with other objects in the environment, determines whether the specific biological interaction conditions are met, and generates semantically enhanced interaction context data.

[0102] Specifically, in some implementations, this step involves parsing the user's gesture recognition results and the behavioral attribute labels of the virtual cell model, combined with its relative spatial relationship with other objects in the environment, to determine whether specific biological interaction conditions are met, thereby generating semantically enhanced interaction context data. The core of this process lies in semantically aligning the user's intent with the biological behavioral logic of the cell model to ensure the biological rationality and immersiveness of the interaction.

[0103] This step first relies on the gesture recognition interface model provided by the Mixed Reality Development Kit (MRTK), which obtains the user's hand joint coordinates and gesture state information by subscribing to the hand skeleton data stream. In the Unity engine, the gesture recognition module is typically implemented as a skeleton tracker, whose output includes the three-dimensional coordinates of each joint. The system then matches these raw input data with the virtual cell model in the anchored scene, taking into account parameters such as distance thresholds (e.g., less than 0.3 meters is considered an interaction trigger) and relative orientation (e.g., the angle between the hand orientation and the cell surface normal is less than 30 degrees).

[0104] Furthermore, the system determines whether the user's gesture conforms to the biological interaction rules of the cell based on the biological behavior tag library (such as phagocytosis, division, chemotaxis, etc.) bound to the cell model. For example, when the user performs a "grab" gesture, the system will check whether the target cell has the behavioral attribute tag of "can be grasped" or "can be moved". If the conditions are met, semantically enhanced interaction context data is generated, including interaction type, target object, interaction timestamp, user viewpoint direction (…). Meta-information such as )

[0105] The confidence threshold for gesture recognition is typically set above 0.7 to ensure the reliability of the interaction; the Euclidean distance formula is used in spatial relationship calculation. Assess the distance between the user's hand and the cell model; introduce a time window mechanism into the interaction judgment logic (e.g., continuously meeting the conditions within 200ms is considered a valid interaction) to improve the stability and response speed of the interaction.

[0106] This step achieves a semantic mapping between user intent and cellular behavior, ensuring that the interaction conforms to biological logic. It also improves the system's interaction accuracy and response efficiency, providing a structured and semantic context for subsequent visual and physical feedback. This enhances the realism and educational value of microscopic biological interactions in a mixed reality environment. S33, based on interaction context data, integrates general operation logic and biological behavior rules to output the final differentiated interaction instructions, which are used to drive rendering and execute corresponding visual or physical responses.

[0107] For example, specific biological interaction feedback is defined. The system triggers different dynamic feedback based on the biological behavior tags of the manipulated object. For instance, when a user manipulates a cell model with aggressive properties to approach a target object, a particle release effect is triggered; when a user squeezes a cell model with soft properties, an elastic deformation animation of the cell membrane is triggered based on calculations by the physics engine.

[0108] Specifically, after receiving user input (such as gestures, gaze, voice, or controller operations), the system first parses the current interaction context, including: the identity of the object being operated on (identified through its biometric tags), the type of the target object, its spatial relative position, and multi-dimensional parameters such as the force and speed of the operation. Then, it performs joint matching of this context with preset general XR operation logic (such as grasping, rotating, and scaling) and a cell-specific biological behavior rule base to generate semantically clear and biologically reasonable interaction instructions. For example, the system defines a series of specific biological interaction feedback mechanisms: When a user manipulates a cytotoxic T cell model labeled with the "attack" attribute and moves it near the target cell (within a preset threshold), the system automatically triggers the "immune synapse formation" animation and plays particle effects of perforin and granzyme release, accompanied by sound effects and slight vibration feedback to enhance functional cognition. When a user applies a squeezing gesture to a neutrophil model labeled with the "soft" or "highly plastic" attribute, the system calls a flexible body physics engine based on the finite element or mass-spring model to calculate the local stress distribution of the cell membrane in real time and drive the mesh vertices to generate realistic elastic deformation and pseudopodia retraction animations. If a user attempts to perform a "phagocytosis" operation (such as wrapping a pathogen in a macrophage model), the system will determine the rationality of the operation based on the phagocytosis efficiency model and then play the multi-stage dynamic process of membrane wrapping, endocytic vesicle formation, and lysosomal fusion in sequence.

[0109] The aforementioned differentiated interaction commands not only control the geometric transformation of the 3D model, but also synchronously schedule the particle system, material shader, audio module and haptic feedback unit to achieve a multi-channel, high-fidelity, biologically consistent immersive response, enabling users to intuitively understand the core principles of the microscopic immune mechanism through natural interaction.

[0110] S4 responds to differentiated interaction commands and user viewpoint data, and calculates and outputs visual rendering data and physical feedback data in real time.

[0111] Specifically, in the step of responding to differentiated interaction commands and user viewpoint data, the system realizes the user's multimodal interactive experience of the virtual cell model in a mixed reality (MR) environment by calculating and outputting visual rendering data and physical feedback data in real time. This step is a key link in the interaction loop of the entire system, and its technical implementation relies on the coordinated scheduling of the graphics rendering engine and the physics engine, as well as the high-precision perception of the user's viewpoint and gesture intentions.

[0112] In some implementations, the system first receives differentiated interaction commands from the multimodal interaction control logic building module. These commands are generated based on the matching relationship between user gestures and cellular behavior attributes, and have a clear semantic context. Simultaneously, the system acquires user viewpoint data, including gaze direction, eye position, and distance to virtual objects. This data is typically provided by the HoloLens 2's depth camera and eye-tracking sensor, achieving sub-millimeter accuracy and a refresh rate of over 60Hz to ensure real-time and natural interaction.

[0113] Furthermore, based on interaction commands and viewpoint data, the system drives the graphics rendering engine (such as Unity's URP pipeline) to dynamically update the materials of the virtual cell model. For example, when the user performs a "grab" gesture, the system adjusts the cell membrane's transparency parameter according to preset interaction rules. ), and activate edge lighting to enhance visual feedback. Simultaneously, the particle system can simulate the dynamic deformation or secretion behavior of cell membranes, with its particle emission rate ( The force of the user's operation is positively correlated, thus achieving a visual mapping of behavioral characteristics.

[0114] In terms of physics feedback, the system invokes a physics engine (such as NVIDIA PhysX or Unity's PhysicsSystem) based on the physical force signals contained in the interaction commands. The motion response is calculated using the collision volume and mass property of the model. The output includes the model's position. ),speed( ) and deformation state ( And through haptic feedback devices (such as HaptX gloves or force feedback controllers), physical responses are converted into haptic signals that users can perceive, enabling synchronous updates of visual and physical perception.

[0115] This step has wide applicability in practical applications such as medical education, biological research, and virtual experiments. For example, in cell biology teaching, students can "pinch" the cell membrane with gestures, and the system renders the deformation of the cell membrane in real time, simulating its elasticity and viscosity through tactile feedback, thereby enhancing the immersive learning experience. The technical value of this step lies in achieving a natural mapping and interaction of microscopic biological behavior in macroscopic space through high-precision viewpoint tracking and physical feedback mechanisms, improving the scientific rigor and practicality of mixed reality systems in biological visualization and interaction design. Furthermore, S4 includes: S41 receives differentiated interaction commands and cell model state data to generate material update parameters and physical force signals.

[0116] Specifically, in the step of responding to differentiated interaction commands and user viewpoint data, receiving differentiated interaction commands and cell model state data to generate material update parameters and physical force signals is the core link in realizing multimodal interactive feedback. This step, by integrating user intent and the dynamic state of the virtual cell model, drives the real-time response of the visual and tactile feedback systems, thereby enhancing the realism and immersion of the interaction.

[0117] This step first receives differentiated interaction commands from the multimodal interaction control logic building module. These commands are typically represented in structured data form, including the user's gesture type, interaction intent, the target cell model's identity, and behavioral attribute tags. Simultaneously, the system acquires the current cell model's state data, including its position, orientation, deformation state, and material parameters. In the Unity engine, this data is usually encapsulated and passed through GameObject and Rigidbody components.

[0118] Furthermore, based on interactive commands and model states, the system generates material update parameters through a material system and a particle system. These parameters include, but are not limited to, diffuse color, alpha transparency, normal map offset, and rimlight intensity. The update logic depends on the matching relationship between the user's action type and the cell's behavioral attributes. For example, when a user performs an "activate" gesture, the system can dynamically adjust the surface luminescence intensity based on the cell type (such as T cells or macrophages) to simulate fluorescence changes during cell activation.

[0119] In generating physical force signals, the system uses the contact point information and physical properties of the cell model (such as collision volume, mass, and elastic coefficients) contained in the interaction commands to call a physics engine (such as Unity's PhysX) to calculate the force vector. Force signals are typically represented in three-dimensional vector form, such as... In conjunction with the input interface of haptic feedback devices (such as HaptX or Leap Motion), force feedback signals are transmitted to the user in real time to achieve synchronous response of haptic perception.

[0120] At the parameter level, the update frequency of material update parameters is typically set to 30-60Hz to match the visual refresh perception threshold of the human eye; the calculation delay of physical force signals should be controlled within 50ms to ensure the real-time performance and naturalness of tactile feedback. Furthermore, the system needs to support dynamic interpolation of various material parameters, such as linear interpolation (Lerp) or spherical interpolation (Slerp), to achieve a smooth visual transition effect.

[0121] This process is widely used in fields such as medical education, biological research, and mixed reality-assisted surgery. For example, in virtual anatomy teaching, users can interact with virtual cell models through gestures. The system adjusts the material properties and tactile feedback in real time according to the cell type and the user's intention, thereby enhancing the learning experience and the realism of the operation.

[0122] By fusing user intent with the dynamic state of the cell model, highly realistic multimodal interactive feedback is achieved, effectively improving the interactive quality and immersive experience of microscopic biological simulation in mixed reality environments.

[0123] S42, based on the material update parameters, combined with the user viewpoint data to drive the shader and particle system, adjust the visual performance of the cell model in real time, and output the corresponding rendering frame data.

[0124] Specifically, in some implementations, based on the material update parameters, the shader and particle system are driven by user viewpoint data to adjust the visual appearance of the cell model in real time and output the corresponding rendering frame data. This step achieves high-fidelity, real-time visual rendering of the virtual cell model in a mixed reality environment by dynamically responding to interactive commands and changes in user viewpoint, thereby enhancing the user's immersive perception of microscopic biological structures and behaviors.

[0125] This step first receives differentiated interaction commands from the interaction control logic module, as well as user viewpoint data provided by the HoloLens 2 device, including gaze direction, eye position, and head pose. This data is obtained through the `GazeProvider` component in the Unity engine and represented as a three-dimensional vector, such as the gaze direction vector. Meanwhile, material update parameters are triggered by changes in model state (such as cell membrane tension, chromosome activity, etc.), and typically include key parameters such as diffuse coefficient, alpha value, and normal perturbation strength.

[0126] Furthermore, the shader system dynamically adjusts the lighting model and surface properties based on the aforementioned parameters. For example, using a physically based rendering (PBR) model, parameters such as metallicity, smoothness, and reflectivity are adjusted to simulate the translucent flow and edge lighting effects of cell membranes. The particle system is used to represent dynamic processes inside cells, such as mitochondrial movement and cytoplasmic flow; its particle emission rate, lifetime, and trajectory parameters are adaptively adjusted based on the user's viewpoint distance and interaction status.

[0127] The user viewpoint distance threshold is typically set between 0.5 and 3 meters to ensure that model details are clearly visible within the visual focus area. Material update frequency should ideally be controlled at 60 frames per second or higher to maintain visual smoothness. The maximum number of particles in the particle system should ideally not exceed 2000 to balance rendering quality and device performance.

[0128] S43 utilizes physical force signals and the collision volume and mass attributes of the model to call the physics engine to calculate motion response and output position, velocity and deformation data that conform to physical laws, which are used to synchronously update visual presentation and user tactile / motion perception feedback.

[0129] First, the system dynamically drives the rendering pipeline and multimodal feedback module of the mixed reality scene based on the user's interactive input (including gestures, gaze, voice and spatial displacement) and its real-time position changes in the physical environment, so as to achieve a low-latency and highly immersive science popularization interactive experience.

[0130] Specifically, upon startup, the system automatically identifies and locks a default origin coordinate system using a spatial anchor mechanism. This origin is strictly aligned with the position of the physical display panels in the on-site setup, ensuring that virtual content (such as immune cell populations, pathogen models, etc.) has a stable and reproducible layout reference in physical space. As the user moves while wearing an XR device (such as HoloLens 2 or Meta Quest Pro), the system continuously tracks the six-degree-of-freedom pose of their head using a SLAM algorithm and adjusts the projection matrix and occlusion relationships of virtual objects accordingly, ensuring geometric consistency in the fusion of virtual and real elements.

[0131] Based on this, the system provides multi-level, multi-sensory feedback output: Visual Feedback: The system dynamically adjusts the material properties, lighting response, and special effects of the cell model based on the current interaction state. For example, when a user selects an immune cell, its nucleus area is highlighted with a semi-transparent, high-brightness color (such as a cyan-blue glow); when "phagocytosis" or "attack" behavior is triggered, the system simultaneously activates the particle system, generating lysosomal fusion light effects, perforin release trajectories, or chemokine diffusion ripples; at the same time, based on the biological behavior tag library, different cell types adopt differentiated color coding and outline tracing strategies to enhance functional recognition.

[0132] Auditory feedback: Integrate directional audio or spatial sound effect modules into the on-site environment to play matching sound effects based on the type of interactive event. For example, a slight "lock" prompt sound is played when a T cell recognizes a target cell, a low-frequency "endocytosis" sound effect is triggered when phagocytosis is completed, and a faint fluid friction sound accompanies cell movement, enhancing the sense of presence and rhythm of the operation.

[0133] Physics Feedback: Integrating a lightweight physics engine (such as NVIDIA PhysX or Unity DOTS Physics) applies mechanical properties (such as mass, elastic coefficient, and damping ratio) to virtual cells that conform to their biological characteristics. When users drag, push, pull, or throw cell models with gestures, the system calculates collision detection, momentum transfer, and air resistance in real time, enabling the models to exhibit motion trajectories and inertial responses that conform to physical laws. For cells marked as "soft" (such as neutrophils), local deformation animations are triggered when subjected to force, further enhancing the realism of the interaction.

[0134] Physics and motion perception feedback: Utilizing the physical force signals applied by the user (such as gesture push force, grasping force) and the physical properties such as collision volume, mass, and elastic modulus preset by the virtual cell model, a lightweight physics engine (such as PhysX or Unity DOTS Physics) is called to calculate its motion response in real time and output position, velocity, angular momentum and local deformation data that conform to the laws of classical mechanics.

[0135] The calculation results of this invention are used, on the one hand, to drive the real-time updating of the 3D mesh—for example, to produce realistic membrane deformation animations of soft cells under pressure, and to make rigid pathogens bounce along the reflection direction after a collision; on the other hand, through the vibration intensity, resistance simulation, or visual motion inertial feedback of the gesture controller, the "feel" of physical interaction is indirectly conveyed, allowing users to obtain intuitive motion perception when dragging, throwing, or squeezing virtual cells. For cells marked as "highly plastic" (such as neutrophils), the system also combines a finite element approximation model to dynamically adjust the vertex displacement in the deformation region, achieving a biologically plausible flexible body response.

[0136] Through the aforementioned synergistic feedback mechanism of vision, hearing, and touch (implicit in the perception of operational resistance), the system not only achieves vivid visualization of the microscopic immune process, but also constructs an immersive science education environment that is semantically consistent, perceptually natural, and intuitively operational, significantly enhancing the public's understanding of complex life phenomena and their interest in participation.

[0137] According to the present invention, the method for three-dimensional spatial mapping and multimodal interaction of microorganisms based on mixed reality enables microorganisms to achieve adaptive spatial anchoring and multimodal interaction that conforms to biological characteristics in a mixed reality environment, thereby improving the accuracy of three-dimensional visualization and the immersiveness and naturalness of the interaction.

[0138] To achieve the above embodiments, such as Figure 7 As shown, this embodiment also provides a microscopic biological three-dimensional spatial mapping and multimodal interaction system 10 based on mixed reality. The system 10 includes: The parameterized model library construction module 100 is used to generate parameterized virtual cell models with behavioral labels and physical attributes based on cell biological feature data, and output model data; The intelligent mapping module 200 is used to establish a spatial anchor point mapping relationship based on the model data and the biological activity habits of the target cell, and to adaptively anchor the virtual cell model to the horizontal plane or suspended space of the real environment. The multimodal interaction control logic construction module 300 is used to output differentiated interaction commands based on cell behavior attributes according to the anchored scene data and user gesture recognition results. The rendering and feedback output module 400 is used to respond to differentiated interaction commands and user viewpoint data, and to calculate and output visual rendering data and physical feedback data in real time.

[0139] According to embodiments of the present invention, a microscopic biological three-dimensional spatial mapping and multimodal interaction system based on mixed reality enables microscopic organisms to achieve adaptive spatial anchoring and biologically consistent multimodal interaction in a mixed reality environment, thereby improving the accuracy of three-dimensional visualization and the immersiveness and naturalness of the interaction.

[0140] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 8 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0141] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0142] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0143] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for microscopic biological three-dimensional spatial mapping and multimodal interaction based on mixed reality, characterized in that, include: Generate parameterized virtual cell models with behavioral labels and physical attributes based on cell biological feature data, and output model data; Based on the model data and the biological activity habits of the target cells, a spatial anchor point mapping relationship is established, and the virtual cell model is adaptively anchored to the horizontal plane or suspended space of the real environment. Based on the anchored scene data and user gesture recognition results, output differentiated interaction commands based on cell behavior attributes; Responding to differentiated interaction commands and user viewpoint data, it calculates and outputs visual rendering data and physical feedback data in real time.

2. The method as described in claim 1, characterized in that, Generate parameterized virtual cell models with behavioral labels and physical attributes based on cell biological feature data, and output model data, including: A basic geometric model of the cell was constructed using 3D modeling software, and the cell nucleus, cell membrane, and staining features were extracted as key feature points. Based on the key feature points, the cell membrane material is parameterized, and the shader is used to simulate the translucent flow and edge lighting effects of the cell membrane, and the physical collision volume of the model is defined. A biological behavior tag library is established, and corresponding behavioral attribute tags are bound to different cell types based on the biological behavior tag library. Finally, complete virtual cell model data containing geometric structure, material parameters, collision volume and behavioral tags is output.

3. The method as described in claim 1, characterized in that, Based on the model data and the biological activity habits of the target cells, a spatial anchor point mapping relationship is established, and the virtual cell model is adaptively anchored to the horizontal plane or suspended space of the real environment, including: Based on virtual cell model data and real-world environment grid data collected by mixed reality devices, horizontal and vertical planes are identified to generate structured physical space semantic information. Based on the physical space semantic information and the biological activity habit tags carried in the model, the initial anchoring position and orientation of the cell model in the real space are calculated and output through spatial anchor point and full space placement algorithms. Based on the anchoring position and real-time user gaze direction and distance data, the hovering pose of the cell model is dynamically adjusted, and optimized interactive comfort zone positioning data is output.

4. The method as described in claim 1, characterized in that, Based on the anchored scene data and user gesture recognition results, differentiated interaction commands based on cell behavior attributes are output, including: Input anchor scene data and hand skeleton data output by gesture recognition interface model, parse the common gesture type currently performed by the user, and generate preliminary interaction intent signals; Based on the preliminary interaction intent signal and the behavioral attribute labels of the selected cell model, combined with its relative spatial relationship with other objects in the environment, it is determined whether the specific biological interaction conditions are met, and semantically enhanced interaction context data is generated. Based on the interaction context data, the general operation logic and biological behavior rules are integrated to output the final differentiated interaction instructions, which are used to drive the rendering and execution of the corresponding visual or physical responses.

5. The method as described in claim 4, characterized in that, The input anchored scene data and the hand skeleton data output by the gesture recognition interface model are used to parse the common gesture type currently being performed by the user to generate a preliminary interaction intent signal, including: Initialize the gesture recognition interface model based on the Mixed Reality Development Kit (MRTK) and output standardized gesture recognition capability configuration data; The interaction logic script loads and calls the gesture recognition capability configuration data, subscribes to the hand skeleton data stream provided by MRTK in real time, and outputs raw input data containing joint coordinates and gesture state. Based on the anchored scene data and the original input data, the user's current gesture type and operation intention are analyzed to generate structured gesture interaction signals.

6. The method as described in claim 1, characterized in that, Responding to differentiated interaction commands and user viewpoint data, it calculates and outputs visual rendering data and physical feedback data in real time, including: Receive differentiated interaction commands and cell model state data to generate material update parameters and physical force signals; Based on the aforementioned material update parameters, and combined with user viewpoint data, the shader and particle system are driven to adjust the visual appearance of the cell model in real time, and output the corresponding rendering frame data; and, Using the physical force signals and the collision volume and mass attributes of the model, the physics engine is invoked to calculate the motion response and output position, velocity and deformation data that conform to physical laws, which are used to synchronously update the visual presentation and the user's tactile / motion perception feedback.

7. The method as described in claim 3, characterized in that, The method also includes utilizing HoloLens 2's spatial localization and gaze tracking capabilities to achieve full-range adaptive placement and hovering of the model in physical space, including: Create a UWP project in Unity, set the target platform to HoloLens 2, the architecture to ARM64, and ensure that the system version is compatible with the space awareness function; Import Mixed Reality Toolkit to automatically configure MixedRealityToolkit and Playspace, enabling spatial anchor points, gesture recognition, and environmental understanding capabilities; Import the parameterized 3D model constructed based on the biological characteristics of immune cells, and perform lightweight optimization to adapt to device performance; HoloLens 2's spatial scanning function collects environmental meshes in real time, identifies horizontal and vertical planes, and generates semantic spatial structures that can be used to place virtual objects. Combining the biological activity habits of cell models, and using spatial anchoring and full-space placement algorithms, the model is intelligently anchored to a desktop, wall, or in the air, and its hovering position and orientation are dynamically adjusted based on eye tracking and distance detection. Test the model's stability, visibility, and interactive comfort in physical space on real devices, and optimize the anchoring logic and rendering performance based on user feedback.

8. A microscopic biological three-dimensional spatial mapping and multimodal interaction system based on mixed reality, characterized in that, include: The parameterized model library construction module is used to generate parameterized virtual cell models with behavioral labels and physical attributes based on cell biological feature data, and output model data; The intelligent mapping module is used to establish a spatial anchor point mapping relationship based on the model data and the biological activity habits of the target cells, and to adaptively anchor the virtual cell model to the horizontal plane or suspended space of the real environment. The multimodal interaction control logic construction module is used to output differentiated interaction commands based on cell behavior attributes according to the anchored scene data and user gesture recognition results. The rendering and feedback output module is used to respond to differentiated interaction commands and user viewpoint data, and to calculate and output visual rendering data and physical feedback data in real time.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the microscopic biological three-dimensional spatial mapping and multimodal interaction method based on mixed reality as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for three-dimensional spatial mapping and multimodal interaction of microscopic organisms based on mixed reality as described in any one of claims 1-7.

Images