Mixed reality space perception method and system

Abstract

The invention discloses a mixed reality space perception method and system, which is executed under the architecture of cooperative work of mixed reality equipment and a cloud server, and realizes object-level three-dimensional reconstruction and semantic understanding through two-dimensional semantic segmentation, two-dimensional to three-dimensional mapping, multi-modal interaction and visual language model reasoning. And the requirement of the mixed reality application for understanding the accurate and efficient three-dimensional scene is met.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of augmented reality / mixed reality human-computer interaction technology, and in particular to a mixed reality spatial perception method and system. Background Technology

[0002] In Mixed Reality (MR) applications, systems need to spatially integrate virtual information with the real environment, enabling users to observe, understand, and interact in an intuitive and natural way in the real world. Among these capabilities, 3D scene understanding is a key foundational technology for achieving immersive experiences and high-quality human-computer interaction. To support users in recognizing, querying, selecting, and manipulating objects in the real environment, the system not only needs to acquire the 3D geometry of the environment but also needs to understand the spatial relationships between objects, the shape features of objects, and their corresponding semantic information. These capabilities place high demands on real-time performance, accuracy, and generalization, especially in dynamic, open, and complex real-world usage environments. Summary of the Invention

[0003] This application provides a mixed reality spatial perception method and system that can meet the needs of mixed reality applications for accurate and efficient 3D scene understanding.

[0004] This invention provides a hybrid reality spatial perception method, comprising: Acquire scene images and perform two-dimensional segmentation of objects in the scene images based on a zero-shot semantic segmentation model; Based on the results of two-dimensional segmentation, a two-dimensional to three-dimensional spatial mapping is performed to generate a three-dimensional point cloud of the corresponding object; The 3D point cloud of the corresponding object is filtered and reconstructed into a mesh to generate an object-level 3D mesh model. Based on the query information from the user, multimodal semantic reasoning is performed on the visual language model to obtain the structured response output by the visual language model; The structured response output by the visual language model is analyzed and fed back into the mixed reality scene.

[0005] In one exemplary instance, after generating the object-level 3D mesh model, the method further includes: When it is necessary to determine the user's focus and query intent, the user's focus and query intent should be determined based on the multimodal interaction method.

[0006] In one exemplary instance, the multimodal interaction method includes: the user's eye movements, and / or the user's gestures, and / or the user's voice input.

[0007] In one exemplary instance, acquiring a scene image and performing two-dimensional segmentation of objects in the scene image based on a zero-shot semantic segmentation model includes: Acquire scene images captured by the mixed reality device and obtain the camera parameters corresponding to the scene images. The camera parameters include projection parameters and pose parameters used to establish the geometric relationship between image pixels and three-dimensional space. The scene image is input into the zero-shot semantic segmentation model to perform object-level two-dimensional segmentation on the objects in the scene image, and obtain a two-dimensional segmentation mask corresponding to at least one object.

[0008] In one exemplary instance, generating the 3D point cloud of the corresponding object includes: Based on the two-dimensional segmentation mask and the camera parameters, the pixels in the two-dimensional segmentation mask are mapped to three-dimensional space to obtain a three-dimensional point set corresponding to the object.

[0009] In one exemplary instance, generating the object-level 3D mesh model includes: For the point cloud data of each of the objects, spatial filtering is performed on the point cloud to remove noise points and outliers; Triangulation is performed based on the two-dimensional projection of the filtered point cloud onto the image plane to generate a set of triangle indices. The generated set of triangle indices is then combined with the corresponding three-dimensional point cloud to construct the object-level three-dimensional mesh model.

[0010] In one exemplary instance, obtaining the structured response output by the visual language model includes: The system obtains query information input by the user and combines it with the object-level 3D mesh model to construct input information for the visual language model. The input information includes image information and object identification information corresponding to the object. The input information is provided to the visual language model for multimodal semantic reasoning to obtain the structured response output by the visual language model.

[0011] In one exemplary instance, the process of parsing the structured response output by the visual language model and feeding it back to the mixed reality scene includes: The semantic information output by the visual language model is associated with the corresponding object-level 3D mesh model, so that the object-level 3D mesh model has the object category, name or semantic attributes obtained by the visual language model, and the object-level semantic understanding results are output in the mixed reality scene.

[0012] This application also provides a computer-readable storage medium storing computer-executable instructions for executing any of the above-described mixed reality spatial perception methods.

[0013] This application provides another mixed reality spatial perception system, including: a spatial segmentation and fusion module and a visual language model integration module; wherein... The spatial segmentation and fusion module includes a zero-sample segmentation submodule, a 2D-to-3D mapping submodule, and a mesh optimization submodule; among them, The zero-shot segmentation submodule is configured to acquire scene images and perform two-dimensional segmentation of objects in the scene images based on a zero-shot semantic segmentation model. The 2D to 3D mapping submodule is configured to perform 2D to 3D spatial mapping based on the results of 2D segmentation to generate 3D point clouds of the corresponding objects; The mesh optimization submodule is configured to filter and reconstruct the 3D point cloud of the corresponding object, generating an object-level 3D mesh model.

[0014] The visual language model integration module is configured to perform multimodal semantic reasoning based on the visual language model according to the query information from the user, obtain the structured response output by the visual language model, parse the structured response output by the visual language model and feed it back to the mixed reality scene. In one exemplary instance, the visual language model integration module includes: an object-level encoding submodule, a visual language model processing submodule, and an object-level decoding submodule; wherein, The object-level encoding submodule is configured to encode the object-level 3D mesh model generated by the spatial segmentation and fusion module, construct an object-level input representation that can be understood by the visual language model; and encapsulate the object set information, the object's 2D projection information, and related image information into a structured data format and transmit them to the visual language model processing submodule. The visual language model processing submodule is set to annotate objects in the scene image based on the two-dimensional projection position of the object in the image, so as to clarify the correspondence between the object identifier and the image region. The object-level decoding submodule is configured to parse the structured response output by the visual language model processing submodule and map the parsing results back to the corresponding object-level 3D mesh model, so as to realize the presentation and interaction of object-level semantic information in mixed reality scenes.

[0015] In one exemplary instance, a multimodal interaction module is also included, configured to determine the user's focus and query intent based on the multimodal interaction method when it is necessary to determine the user's focus and query intent.

[0016] In one exemplary instance, the multimodal interaction module includes: an eye-tracking submodule, and / or a gesture interaction submodule, and / or a speech recognition submodule; wherein, The eye-tracking submodule is configured to acquire the user's gaze direction in real time and determine the object the user is currently focusing on based on the spatial relationship between the gaze ray and the object-level 3D mesh model. The gesture interaction submodule is set up to allow users to explicitly select objects using their hand gestures, serving as a supplement or alternative to eye-tracking input. The speech recognition submodule is configured to collect the user's voice input and convert the voice signal into text to obtain the user's query content or interaction instructions.

[0017] The mixed reality spatial perception method provided in this application is executed in an architecture where mixed reality devices and cloud servers work together. Through two-dimensional semantic segmentation, two-dimensional to three-dimensional mapping, multimodal interaction, and visual language model reasoning, it realizes object-level three-dimensional reconstruction and semantic understanding, meeting the needs of mixed reality applications for accurate and efficient three-dimensional scene understanding.

[0018] Furthermore, this embodiment of the application introduces a technical feature to determine the user's focus and query intent based on multimodal interaction when the object-level 3D mesh is generated and a user-initiated interactive query request is detected. This involves associating the user's natural interactive inputs, such as eye movements, gestures, and / or voice, with the specific object-level 3D mesh. This accurately determines the user's focus and query context without requiring the user to explicitly describe the target object. Consequently, it provides explicit object-level contextual information for subsequent multimodal semantic reasoning based on a visual language model, avoiding inference based solely on plain text or 2D image information. This improves the intuitiveness of object selection and the accuracy of semantic understanding, further enhancing the naturalness and reliability of human-computer interaction in mixed reality scenarios.

[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0020] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0021] Figure 1 This is a flowchart illustrating the mixed reality spatial perception method in the embodiments of this application; Figure 2 This is a schematic diagram of the composition structure of the mixed reality spatial perception system in the embodiments of this application; Figure 3This is a schematic diagram of the overall system architecture of an example of a mixed reality spatial perception system in this application. Figure 4 This is an example processing flowchart of the spatial segmentation and fusion module in an embodiment of this application; Figure 5 This is a schematic diagram illustrating an example of the human-computer interaction interface of the system in this application embodiment; Figure 6 This is a schematic diagram illustrating an encoding and decoding example of the visual language model integration module in this application. Figure 7 These are examples of different visualization modes in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be arbitrarily combined with each other.

[0023] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0025] It is understood that the terms "first" and "second" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0026] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.

[0027] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0028] The steps illustrated in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in a different order than that presented here.

[0029] Most 3D scene understanding technologies are based on 3D sensors or multi-view scanning devices. They acquire point cloud, depth map, or voxel data and combine them with supervised learning models to achieve object recognition and semantic segmentation. These methods typically rely on large-scale, finely labeled datasets and train models for specific scenes or a limited number of categories. In practical mixed reality applications, these methods suffer from high data acquisition and annotation costs, limited model generalization ability, and difficulty adapting to new scenes and objects. Furthermore, they often require additional 3D scanning equipment, increasing system complexity and deployment costs.

[0030] With the development of Large Language Models (LLMs), some studies have attempted to introduce LLM-based methods for semantic parsing of user commands or scene descriptions to improve the system's understanding and interaction capabilities. However, LLMs primarily process textual information and lack the ability to directly perceive visual content and 3D space; their output relies on structured visual input provided by external modules. Therefore, methods solely based on LLMs struggle to locate objects, perform geometric modeling, and understand spatial relationships in real-world environments, failing to meet the demands for 3D scene understanding in mixed reality applications.

[0031] Visual language models (VLMs) have made some progress in image understanding and cross-modal reasoning by fusing visual and linguistic information. VLMs can perform object recognition and semantic inference under zero-shot or few-shot conditions, reducing reliance on manually labeled data. However, these methods primarily use two-dimensional images as input, focusing mainly on image-level or region-level semantic understanding, lacking the ability to model real three-dimensional spatial structures. Furthermore, VLMs struggle to directly generate object-level three-dimensional representations or mesh models, and the lack of an effective mapping mechanism between two-dimensional semantic results and three-dimensional geometry limits their direct application in mixed reality scenarios.

[0032] Zero-shot segmentation models (such as the Segment Anything Model, or SAM) that have emerged in recent years can segment any object in a 2D image without predefined categories, providing a new approach to visual understanding in open environments. However, the segmentation results of these methods are still limited to the 2D image space, and a mature integration scheme with the 3D perception process of mixed reality devices has not yet been formed. At the same time, there is a lack of effective methods to automatically convert 2D segmentation results into object-level 3D meshes or interactive entities, making it difficult to directly support the spatial interaction needs in mixed reality.

[0033] In practical mixed reality devices, the main-view color camera and depth sensor are typically physically separated, with different intrinsic and extrinsic parameter configurations. This hardware structure makes it difficult to directly apply traditional point cloud fusion and semantic mapping methods that rely on precise sensor alignment. Semantic segmentation results in 2D images are difficult to accurately map to 3D point clouds or meshes, thus affecting the accuracy and stability of 3D scene understanding and object-level modeling. Furthermore, mixed reality systems in related technologies often rely on users to provide explicit and complete object names or complex semantic descriptions for object selection and query interaction, or only support operations on predefined object categories. This interaction method does not conform to users' intuitive operating habits, increases the learning cost, and makes it difficult to achieve a natural interactive experience based on WYSIWYG (What You See Is What You Get).

[0034] In summary, existing mixed reality 3D scene understanding technologies still have significant shortcomings in terms of data dependence, 3D semantic modeling, 2D-to-3D mapping, device adaptability, and natural interaction. Therefore, to achieve zero-shot object recognition, object-level 3D representation generation, and natural interaction capabilities with deep integration with multimodal models in the real environment without requiring additional 3D scanning equipment and large-scale labeled data, thus meeting the needs of mixed reality applications for accurate and efficient 3D scene understanding, this application proposes a mixed reality spatial perception method. This method is based on zero-shot segmentation and a visual language model to achieve mixed reality scene understanding.

[0035] Figure 1 This is a flowchart illustrating the mixed reality spatial perception method in the embodiments of this application, as shown below. Figure 1 As shown, it may include: Step 100: Acquire scene images and perform two-dimensional segmentation of objects in the scene images based on a zero-shot semantic segmentation model.

[0036] In one exemplary instance, step 100 may include: Acquire scene images captured by a mixed reality device, and acquire camera parameters corresponding to the scene images. The camera parameters include at least projection parameters and pose parameters used to establish the geometric relationship between image pixels and three-dimensional space. The scene image is input into the zero-sample semantic segmentation model, which performs object-level two-dimensional segmentation on the objects in the scene image without pre-defining object categories, and obtains a two-dimensional segmentation mask corresponding to at least one object.

[0037] In one embodiment, during operation, the mixed reality device can use a front-facing color camera, such as that of a head-mounted display device, to capture scene images from a first-person perspective (e.g., with a resolution of 1920×1080 pixels), and record the image sequence number corresponding to the image while capturing the image. Transformation matrix from camera to world coordinate system and camera projection matrix The acquired image data is converted into a byte stream format and sent to a cloud server via network communication (such as the WebSocket protocol). In this embodiment, the mixed reality device refers to a computing device capable of acquiring images of the real environment and overlaying virtual information onto the real environment. It can be implemented using a head-mounted display device, glasses-type augmented reality device, or other terminal devices with mixed reality functionality.

[0038] After receiving image data, the cloud server calls a pre-loaded zero-shot semantic segmentation model to perform object-level segmentation of the image. Without needing to predefine object categories, it generates segmentation results containing multiple segmentation region masks. Each segmented region corresponds to a potential object in the image, and the segmentation results serve as the input basis for subsequent 2D-to-3D mapping and object-level modeling.

[0039] Step 101: Based on the results of the two-dimensional segmentation, perform a two-dimensional to three-dimensional spatial mapping to generate a three-dimensional point cloud of the corresponding object.

[0040] In one exemplary instance, step 101 may include: Based on a 2D segmentation mask and camera parameters, pixels in the 2D segmentation mask are mapped to 3D space to obtain a 3D point set corresponding to the object. In one embodiment, the 3D point set can be obtained by backprojecting image pixels into 3D rays and combining this with depth information or scene geometry models in the real environment to determine the corresponding position of the rays in 3D space.

[0041] In one exemplary instance, mapping pixels from a two-dimensional segmentation mask to a three-dimensional space based on a two-dimensional segmentation mask and camera parameters to obtain a three-dimensional point set corresponding to the object can include: The cloud server will split the results The set of pixel coordinates corresponding to each object and its corresponding image number. The system returns to the mixed reality device. Upon receiving this information, the mixed reality device performs a 2D-to-3D mapping on each pixel in each segmented region based on the camera parameters of the current image. Specifically, for any pixel, the pixel coordinates are first normalized according to the camera resolution and mapped to projection space coordinates. Then, the inverse transformation of the projection matrix is ​​used to transform it back to the camera coordinate system, and the corresponding world space ray direction is obtained by combining the camera-to-world coordinate system transformation matrix. Next, a ray is emitted from the camera position along the ray direction towards the scene mesh, and the intersection of the ray and the scene mesh is obtained as the pixel's corresponding position in 3D space. Through this process, multiple pixels within the same segmented region are mapped to a set of 3D space points, forming the initial point cloud data for the corresponding object.

[0042] In one embodiment, based on the segmentation result Generate a set of mesh key points The two-dimensional pixel coordinates are mapped to a three-dimensional point cloud through ray projection. This can include: pixel coordinates (u, v) are first normalized to the range [0, 1], then mapped to the projection space [-1, 1], and then projected using a projection matrix. Inverse transformation to camera space, and finally through transformation matrix Convert to world space coordinates; project a ray along the ray direction from the camera position and find the intersection with the scene mesh to obtain the 3D contact point.

[0043] Step 102: Filter and reconstruct the 3D point cloud of the corresponding object to generate an object-level 3D mesh model.

[0044] In one exemplary instance, step 102 may include: For the point cloud data of each object, spatial filtering is performed on the point cloud to remove noise points and outliers; in one embodiment, the filtering process can use spatial hashing and neighborhood search to retain only points with a sufficient number of neighbors within a set spatial range. Triangulation is performed based on the 2D projection of the filtered point cloud onto the image plane to generate a set of triangle indices. These indices are then combined with the corresponding 3D point clouds to construct an object-level 3D mesh model. The generated 3D mesh model is instantiated as an interactive object in a mixed reality scene and assigned a unique identifier.

[0045] In one embodiment, point cloud filtering can be performed using a spatial hashing method (e.g., a grid size of 0.004m) to retain vertices with a sufficient number of neighbors within a neighborhood radius (e.g., 0.012 × ray length). Finally, the filtered 2D projection set is subjected to Delaunay triangulation to generate the 3D point cloud of the corresponding object.

[0046] In this embodiment, the above segmentation and mapping process can be repeated from different perspectives or at different times to supplement and update the three-dimensional geometric information of the object, and the generated object-level mesh is persistently stored for subsequent context-related interactions and queries.

[0047] Step 103: Based on the query information from the user, perform multimodal semantic reasoning based on the visual language model to obtain the structured response output by the visual language model.

[0048] In one exemplary instance, step 103 may include: Obtain the query information input by the user, and combine it with the object-level 3D mesh model to construct the input information of the visual language model. The input information includes at least the image information and object identification information corresponding to the object. The input information is provided to the visual language model for multimodal semantic reasoning, and the structured response output by the visual language model is obtained, namely the semantic information related to the object (i.e. the object recognition result) and the text response to the user query.

[0049] In one embodiment, step 103 may specifically include: First, the segmented and reconstructed objects are encapsulated into structured object data. This may include: each segmented object... (Including the spatial transformation information of the objects and the three-dimensional coordinates of the center of the object mesh) are encapsulated into a unified data structure to form an object collection. Based on the camera projection matrix corresponding to the current scene image. and the camera's extrinsic matrix Calculate the two-dimensional projected coordinates of each object on the image plane. This is used to establish the correspondence between 3D objects and 2D images. The object collection and its projection information are serialized into a structured data format (such as JSON) and transmitted to a cloud server.

[0050] Then, based on the two-dimensional projected coordinates of the objects, each object is labeled in the scene image. In one embodiment, a numerical labeling strategy can be used, labeling the objects at their corresponding pixel positions. Different colored number labels are superimposed on the area. This annotation method uses a unique identifier to indicate the object. While minimizing pixel-level visual interference, it preserves the object's original color and texture information, thus facilitating accurate reasoning by the visual language model.

[0051] Simultaneously, structured text prompts are automatically generated based on the object set information and the user's query. The generated structured text prompts are used to clearly inform the visual language model of the number of objects in the current image, the identification number of each object, and the relevance between the objects and the user's query. The output format of the visual language model is also specified to be a structured data format to ensure the machine readability and consistency of the inference results.

[0052] In one embodiment, when a user poses a context-dependent or multi-turn query question, the system appends previously labeled images, historical user questions, and corresponding visual language model responses as contextual information to the current query, enabling the visual language model to reason based on historical information, thereby supporting semantic understanding with context-dependent or multi-view features.

[0053] Subsequently, the labeled images and generated structured text prompts are fed as input to the visual language model for multimodal semantic reasoning. The visual language model outputs a structured response, which includes at least the identifier, category, and name information of the identified objects, as well as the text answer generated for the user query.

[0054] In this step, after receiving the query information from the user, the cloud server encodes the objects based on the object-level mesh information already generated in the current scene to construct an object set description. Based on the object's projection position in the image, it overlays digital identifiers onto each object in the image to establish a one-to-one correspondence between image pixels and 3D objects. Simultaneously, it generates structured text prompts based on the object set and the user's query content to clearly inform the visual language model of the number of objects, the relationship between object identifiers and the query. The labeled image and the generated text prompts are then sent to the visual language model as input for inference processing to obtain a structured response containing object recognition results and the user's query answer.

[0055] Step 104: Analyze the structured response output by the visual language model and feed it back into the mixed reality scene.

[0056] In one exemplary instance, step 104 may include: The semantic information output by the visual language model is associated with the corresponding object-level 3D mesh model, enabling the object-level 3D mesh model to possess the object category, name, or semantic attributes parsed by the visual language model. The object-level semantic understanding results are then output in a mixed reality scene, thereby achieving object-level 3D perception and semantic interaction for real-world environments. Specifically, this can include: The structured response output by the visual language model is parsed, the object identifiers are mapped to the corresponding 3D mesh model, and the object category, name and semantic relationship information inferred from the 3D mesh model are assigned to the corresponding object. The parsed results are presented in a visual manner through mixed reality devices, enabling the object-level 3D information and textual semantic results to be presented synchronously in the real scene, thereby completing mixed reality spatial perception and object-level semantic understanding.

[0057] In one embodiment, step 104 may specifically include: First, the structured response output by the visual language model is parsed to extract the object identification information contained therein, and the object identification is mapped to the corresponding object-level 3D mesh model. Then, the object category, name, and semantic relationships obtained from the visual language model are assigned to the corresponding object-level 3D mesh model, giving the 3D mesh model the corresponding semantic attributes.

[0058] Next, the interactive capabilities of the object-level 3D mesh model are updated in the physics engine so that it can respond to user interaction commands and perform corresponding display or interactive operations based on the semantic attributes parsed from the visual language model.

[0059] Finally, the parsed results are presented visually using mixed reality devices. Visualization methods can include, but are not limited to, object highlighting, label display, direction indicators, or boundary display, enabling the object-level 3D geometric information and the textual semantic results output by the visual language model to be presented synchronously in the real-world scene, thereby achieving object-level 3D perception and semantic interaction for a real-world environment.

[0060] The mixed reality spatial perception method provided in this application is executed in an architecture where mixed reality devices and cloud servers work together. Through two-dimensional semantic segmentation, two-dimensional to three-dimensional mapping, multimodal interaction, and visual language model reasoning, it realizes object-level three-dimensional reconstruction and semantic understanding, meeting the needs of mixed reality applications for accurate and efficient three-dimensional scene understanding.

[0061] In one exemplary instance, after step 102, when it is necessary to determine the user's focus and query intent, the mixed reality spatial perception method provided in this application embodiment may further include: Determine the user's focus and query intent based on multimodal interaction methods.

[0062] In one exemplary instance, determining the user's focus and query intent based on multimodal interaction methods may include: After generating the object-level 3D mesh, when a user initiates an interactive query request, the device acquires the user's object selection and query input through multimodal interaction. The mixed reality device utilizes eye tracking to detect the user's gaze direction in real time and determines the currently focused object through collision detection between the gaze ray and the object mesh. Furthermore, it supports explicit object selection via gesture rays and pinch gestures as a supplement to eye-tracking input, improving the accuracy and flexibility of object selection.

[0063] When user voice input is detected, the process also includes: acquiring the user's voice signal and converting it into text using a voice recognition service. The voice text is then associated with the identifier information of the selected object, the object's location information in three-dimensional space, and its corresponding image number, and uniformly encapsulated into a structured data format. The encapsulated structured data is then sent to a cloud server as input for constructing a visual language model and executing subsequent multimodal semantic reasoning.

[0064] In this embodiment, when the object-level 3D mesh is generated and the user initiates an interactive query request, the user's focus object and query intent are determined based on the multimodal interaction method. This realizes the association between the user's multimodal natural interactive input and the specific object, thereby providing object-level contextual information for subsequent semantic reasoning based on the visual language model.

[0065] In this embodiment, a further technical feature is introduced to determine the user's focus object and query intent based on multimodal interaction methods when the object-level 3D mesh is generated and a user-initiated interactive query request is detected. Multimodal interaction methods may include, but are not limited to, user eye movements, and / or user gestures, and / or user-inputted speech. These natural interactive inputs are associated with the specific object-level 3D mesh, thereby accurately determining the user's focus object and query context without requiring the user to explicitly describe the target object. This provides clear object-level contextual information for subsequent multimodal semantic reasoning based on a visual language model, avoiding inference based solely on plain text or 2D image information. It improves the intuitiveness of object selection and the accuracy of semantic understanding, and further enhances the naturalness and reliability of human-computer interaction in mixed reality scenarios.

[0066] This application also provides a computer-readable storage medium storing computer-executable instructions for performing any of the mixed reality spatial perception methods described above.

[0067] This application further provides a processing system, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: steps for performing the mixed reality spatial perception method described in any of the preceding claims.

[0068] Figure 2 This is a schematic diagram of the composition structure of the mixed reality spatial perception system in the embodiments of this application, as shown below. Figure 2 As shown. It includes at least: a spatial segmentation and fusion module, and a visual language model integration module; wherein, The spatial segmentation and fusion module includes a zero-sample segmentation submodule, a 2D-to-3D mapping submodule, and a mesh optimization submodule; among them, The zero-shot segmentation submodule is configured to acquire scene images and perform two-dimensional segmentation of objects in the scene images based on a zero-shot semantic segmentation model. The 2D to 3D mapping submodule is configured to perform 2D to 3D spatial mapping based on the results of 2D segmentation to generate 3D point clouds of the corresponding objects; The mesh optimization submodule is configured to filter and reconstruct the 3D point cloud of the corresponding object, generating an object-level 3D mesh model.

[0069] The visual language model integration module is configured to perform multimodal semantic reasoning based on the visual language model according to the query information from the user, obtain the structured response output by the visual language model, parse the structured response output by the visual language model and feed it back to the mixed reality scene.

[0070] In one embodiment, the visual language model integration module realizes multimodal semantic interaction between object-level 3D information and the visual language model, including an object-level encoding submodule, a visual language model processing submodule, and an object-level decoding submodule; wherein... The object-level encoding submodule is configured to encode the object-level 3D mesh model generated by the spatial segmentation and fusion module, constructing an object-level input representation that can be understood by the visual language model. In one embodiment, the object-level encoding submodule encodes each segmented object... Encapsulated as structured object data, the object data includes at least the object's unique identifier, spatial transformation information, and the 3D coordinates of the geometric center of the object's mesh. The object-level encoding submodule calculates the 2D projection position of each object in the image plane based on the camera projection matrix and camera extrinsic matrix corresponding to the current scene image, which is used to establish a one-to-one correspondence between 3D objects and 2D images.

[0071] The object-level encoding submodule is also used to encapsulate object collection information, object 2D projection information, and related image information into a structured data format and transmit it to the visual language model processing submodule as the input basis for subsequent multimodal semantic reasoning. The visual language model processing submodule is configured to use the structured input information output by the object-level encoding submodule to call the visual language model to perform multimodal semantic reasoning, generating object-level semantic understanding results and text responses to user queries.

[0072] In one embodiment, the visual language model processing submodule annotates objects in a scene image based on their two-dimensional projection positions in the image, thereby clarifying the correspondence between object identifiers and image regions. The annotation method can employ numerical labels or other low-interference visual cues to provide clear object cues to the visual language model while preserving as much of the original image's appearance information as possible.

[0073] Meanwhile, the visual language model processing submodule automatically generates structured text prompts based on the object set information and user query content. These prompts clearly inform the visual language model of the number of objects, object identifiers, and their relationship with the user query. The output format of the visual language model is also defined as a structured data format.

[0074] The visual language model processing submodule inputs the labeled image information and generated text prompts into the visual language model to obtain a structured response containing object identifiers, object categories, object names, and text answers generated for user queries.

[0075] The object-level decoding submodule is configured to parse the structured response output by the visual language model processing submodule and map the parsing results back to the corresponding object-level 3D mesh model, thereby enabling the presentation and interaction of object-level semantic information in a mixed reality scene. In one embodiment, the object-level decoding submodule extracts object identification information from the structured response and matches the object identification with the corresponding object-level 3D mesh model; subsequently, the object category, name, or other semantic attributes inferred from the visual language model are assigned to the corresponding object-level 3D mesh model, giving the 3D mesh model the corresponding semantic attributes.

[0076] The object-level decoding submodule is also used to pass the parsed object-level semantic information to the mixed reality rendering and interaction system, enabling the object-level 3D mesh model to respond to the user's interaction commands and present the semantic information of the object in a visual way in the mixed reality scene, thereby realizing the fusion output of object-level 3D perception and semantic understanding.

[0077] The mixed reality spatial perception system provided in this application embodiment is executed under the architecture of mixed reality devices and cloud servers working together. Through two-dimensional semantic segmentation, two-dimensional to three-dimensional mapping, multimodal interaction and visual language model reasoning, it realizes object-level three-dimensional reconstruction and semantic understanding, which meets the needs of mixed reality applications for accurate and efficient three-dimensional scene understanding.

[0078] In one exemplary instance, a multimodal interaction module may also be included, configured to determine the user's focus and query intent based on the multimodal interaction method when it is necessary to determine the user's focus and query intent.

[0079] In one embodiment, the multimodal interaction module includes: an eye-tracking submodule, and / or a gesture interaction submodule, and / or a speech recognition submodule; wherein, The eye-tracking submodule is configured to acquire the user's gaze direction in real time and determine the object the user is currently focusing on based on the spatial relationship between the gaze ray and the object-level 3D mesh model. In one embodiment, the eye-tracking submodule generates the user's gaze ray using the eye-tracking sensor of the mixed reality device and detects the collision or intersection relationship between the gaze ray and the object-level 3D mesh model. When an intersection between the gaze ray and an object-level 3D mesh model is detected, that object is identified as the target object the user is currently focusing on, and the corresponding object identification information is output. The eye-tracking submodule can also provide feedback information to the rendering module for highlighting or other visual cues of the object being focused on.

[0080] The gesture interaction submodule is configured to allow users to explicitly select objects using their hand gestures, serving as a supplement or alternative to eye-tracking input. In one embodiment, the gesture interaction submodule utilizes the hand tracking function of the mixed reality device to generate gesture rays and detects the spatial relationship between the gesture rays and the object-level 3D mesh model. When it detects that the user has selected a target object using a preset gesture (such as a pinch gesture), the object is identified as the user-selected target object, and the corresponding object identification information is output. The gesture interaction submodule is suitable for interaction scenarios requiring precise object selection or simultaneous selection of multiple objects.

[0081] The speech recognition submodule is configured to collect the user's speech input and convert the speech signal into text to obtain the user's query content or interaction instructions. In one embodiment, the speech recognition submodule collects the user's speech signal through the microphone of a mixed reality device and sends the speech signal to a speech recognition service for processing to generate corresponding text information. The text information is used to represent the user's query intent or interaction needs and is associated with a target object determined by the eye-tracking submodule and / or the gesture interaction submodule.

[0082] In one embodiment, the multimodal interaction module is further used to encapsulate the user's voice text, target object identification information, and object spatial location information into a structured data format and send it to the visual language model integration module for constructing the input of the visual language model and performing subsequent multimodal semantic reasoning.

[0083] The method of the present invention will now be described with reference to an example.

[0084] like Figure 3 As shown in the example, the mixed reality spatial perception system based on zero-shot segmentation and visual language models proposed in this application consists of three parts: a spatial segmentation and fusion module, a multimodal interaction module, and a visual language model integration module. In this example, a mixed reality device such as HoloLens 2 is used as the application platform, a cloud server, Unity3D, Visual Studio, and the OpenXR framework to develop and implement the complete system. The goal is to acquire scene images in real time, generate object-level 3D meshes through zero-shot segmentation, and achieve semantic understanding by combining a visual language model. Unity3D and Visual Studio are used to create MR applications that support multiple head-mounted display devices, the OpenXR framework is used to ensure cross-device compatibility, and the WebSocket protocol is used to achieve real-time communication between the device and the cloud server. Specific implementation steps may include: First, construct an object-level spatial segmentation module.

[0085] The spatial segmentation and fusion module, based on a zero-shot semantic segmentation model and 2D-to-3D mapping technology, achieves the conversion from HMD-acquired images to object-level 3D meshes. Among these, First-person perspective images were captured using the front-facing camera of a head-mounted display device, with a resolution of 1920×1080 pixels. The following key parameters were recorded during image acquisition: image number. Transformation matrix from camera to world (4×4 matrix), projection matrix (4×4 matrix). The image is converted to byte format and transmitted to the cloud server via the WebSocket protocol. The cloud server stores the images in the order of transmission to ensure data synchronization. The average transmission time in this example is 1.6 seconds (range 1.4-2.5 seconds).

[0086] The cloud server preloads a zero-shot semantic segmentation model. In this example, the parameters are configured as follows: confidence threshold ( ): 0.6, Crossover Union (CUC) threshold ( ): 0.2. After receiving the image, the segmentation model generates a segmentation result containing mask information for multiple regions. Based on the segmentation results Initialize the fast sampling prompt and use the global prompt (everything prompt) to generate segmentation labels for all regions. For each segmented region （ ） Extract the corresponding mask And generate grid sampling points (such as Figure 4 (As shown). The set of all grid points forms the region. List. The average processing time in this example is 1.3 seconds (range 1.1-1.6 seconds).

[0087] like Figure 4 As shown, a two-dimensional mask is converted into a three-dimensional point cloud. The cloud server will... Coordinates and corresponding image number Return to the head-mounted display. Upon receiving a new message, clear all previously generated meshes and point clouds, and begin processing the segmentation results of the current image.

[0088] For a given image sequence number Search data It contains a list of pixel coordinate sets corresponding to multiple segmented regions. Calculate the total number of points across all regions. Initial capacity is Point cloud prefabricated components. To visually distinguish regions, each region is... Assign a unique color This forms a color set. For each region... , will pixels Convert pixels to spatial points within the scene mesh using the following steps: 1) Map pixels to world space: pixels First, normalize based on camera resolution. scope: ,in and These represent the width and height of the camera resolution, respectively. Normalized position mapping to projection space. : Use with images Associated projection matrix The position is projected back into the camera space. This is the assumed depth value (usually set to 1). ： Ultimately, this is achieved through the transformation matrix from camera to world. Convert to world spatial location: .

[0089] Raycasting to find scene intersections: acquiring images Camera position Used to calculate ray direction: ;from along Directional ray emission, maximum distance When a ray intersects with the scene mesh, record the intersection point. As pixels The corresponding position.

[0090] Point cloud construction: each intersection point With area-specific colors Pair and append to the vertex list This method effectively converts a 2D mask into a spatial point cloud, providing a foundation for subsequent object-level mesh generation. The average point cloud generation time is 0.1 seconds (range 0.04-0.35 seconds).

[0091] For each region In this example, coroutines based on a vertex list can be used. and Generate region-specific meshes. For optimization... Triangulation is performed, followed by a filtering step to reduce noise and ensure mesh quality. The filtering process is as follows: using a spatial hashing method, mesh size... For each vertex In radius Perform a neighborhood search within the range, where To and Associated ray length; retain at least The vertices form a filtered vertex set. and corresponding two-dimensional projection set ;right Perform Delaunay triangulation to generate a set of triangle indices T. Then combine... Construct the final mesh with T Then, normals and boundaries are calculated for rendering. Each mesh is assigned a unique color. The mesh is then instantiated as an object in the AR scene. The average mesh generation time is 4.5 seconds (range 1.6-11.7 seconds).

[0092] Outliers may appear due to deviations in scene mesh reconstruction. This application's embodiment optimizes the mesh, preserving a more reasonable mesh structure, successfully isolating outliers, and reducing the impact of deviations on the generated mesh. This application's embodiment supports multi-view analysis using a frame-based method, allowing users to change the viewpoint or position and re-execute the segmentation pipeline to capture new object geometry. A persistent mesh storage mechanism is implemented, retaining the object mesh for each processed request, facilitating context-aware queries of objects across different viewpoints and time instances.

[0093] Then, construct the multimodal interaction module.

[0094] Intuitive object selection and query functions are achieved through eye tracking, gesture interaction, and voice recognition. In this example, function switches can be designed as a menu that appears when the user flips their hand. The user can summon the menu by raising their palm and then use gestures to click function buttons to access the functions on the menu (such as…). Figure 5 (As shown).

[0095] In one instance, object selection can be achieved using hand tracking and eye tracking capabilities provided by the Mixed Reality Toolkit (MRTK3). Eye tracking: The user indicates the specific object they are currently focusing on using an eye gaze ray. Each object in the environment is equipped with a mesh collider that detects and responds to intersections with the eye gaze ray. This setup ensures the system can continuously identify the object the user is observing. Furthermore, visual feedback (such as object highlighting) informs the user whether the system has accurately identified their gaze target. Gesture interaction: The system uses hand rays as an alternative target selection method for gesture-based interaction. The user can aim at an object with the hand ray and perform a pinch gesture to select the target. This method serves as a supplementary input mode to eye gaze, providing greater flexibility, especially when selecting multiple objects. Voice interaction is achieved through a cloud-based speech recognition service. Audio captured by the head-mounted display is transmitted in real-time to the cloud for processing and conversion into text. The system maintains up-to-date, complete speech-to-text records. Voice input can be queries of varying specificity: it supports fuzzy queries such as "What is this?" or "Describe it."; it supports precise questions related to specific objects, such as "What is the relationship between the first and second objects?" and "Where is the computer located in space?"; in the absence of such contextual information, the VLM must infer the target object by supplementing descriptive details to ensure accurate interpretation of the user's intent. This application addresses this problem by implicitly conveying the object of interest through object-level spatial mesh segmentation (integrated with eye tracking and gesture input).

[0096] User query and interaction data can be formatted as standard JSON and transmitted to a cloud server via a WebSocket connection. The JSON data includes: the ID and 3D coordinates of the selected object, the text content of the user's voice query, and the sequence number and timestamp of the current image.

[0097] Next, we will build a visual language model integration module.

[0098] Semantic understanding of 3D scenes can be achieved through object-level encoding, VLM processing, and object-level decoding (e.g., Figure 6 (As shown).

[0099] Numerical labeling can be used for object annotation in VLM analysis. Compared to alternatives such as masks or contours, numerical labels minimize pixel-level interference with object appearance, providing clear cues for VLM while preserving original color and texture information. After receiving data from the HMD, the cloud server processes the captured image by labeling each object. Specifically, different colored numbers are superimposed on the center of the object mask. This annotation serves as a visual cue for the VLM, ensuring a one-to-one correspondence between the segmented mesh and the annotated objects in the image. In addition to image annotations, structured cues are automatically generated based on object information. The cues are designed to be both logical and practical, clearly indicating the number of VLM objects and their relevance to the user's question. The structured JSON output format ensures machine readability and consistency in the VLM response, facilitating downstream processing.

[0100] The following is an example of a prompt template:

[0101] The image annotation and generated prompt input are processed by a visual language multimodal model (VLM). This example uses Qwen-VL-Max, a state-of-the-art visual language model that excels in multimodal reasoning and object recognition tasks, particularly adept at handling complex visual inputs paired with text prompts. The VLM outputs a structured response containing the identified object (ID, category, and name) and provides a textual answer to the user's query. The average VLM response time in this example is 6.9 seconds (range 2.8–11.8 seconds).

[0102] The JSON file generated by VLM contains information about the collection of objects. Given the use of a standardized sequential encoding method, the object information from the VLM response can be mapped to the corresponding object transformations and mesh representations. In the physics engine, these meshes are not only given the ability to respond to user interaction commands but also possess inherent properties derived from VLM analysis, such as object categories and names. To enhance user interaction, the system also implements a hand gesture menu to control the mesh and point cloud display. This menu also provides a convenient interface to view user-input questions and the corresponding text responses from VLM. By integrating this functionality into the AR scene, object-level information and text answers are seamlessly presented to the user, enhancing usability and immersion.

[0103] like Figure 7As shown, the system features multiple visualization modes, allowing users to choose the most suitable display effect based on different usage scenarios and personal preferences. For example: the colored grid display mode overlays a semi-transparent colored grid onto the segmented object, providing clear boundary identification. This method is suitable for scenarios requiring precise segmentation analysis or managing densely packed objects. The directional arrow mode generates directional arrows based on the geometric center of the object's grid, indicating the target object's location while minimizing obscuring the actual object's surface details. This mode is particularly suitable for indicating object locations or when users need to observe object surface information. The bounding box mode generates wireframe cubes around the object, providing an intuitive display of the object's spatial extent while maintaining full visibility of the object itself.

[0104] To enhance the user experience, corresponding label information can be directly displayed on objects to ensure the immediate visibility of object recognition results. Users can then access the system menu via a palm-flipping gesture, switching between different visualization modes and adjusting the display effects according to the current task requirements. Furthermore, the system can be designed with a dedicated user interface to display voice input and VLM response results, providing a complete multimodal interactive experience.

[0105] Based on the experimental setup, the performance metrics of the system shown in this example under various scenarios are as follows: segmentation accuracy is 94.9% for clean scenes and 93.5% for cluttered scenes; recognition accuracy is 94.9% for clean scenes and 89.1% for cluttered scenes; the average processing time is: image acquisition and transmission 1.6 seconds, point cloud generation 0.1 seconds, mesh generation 4.5 seconds, image processing 1.3 seconds, VLM response 6.9 seconds, totaling approximately 14.4 seconds (excluding human-computer interaction time). Under extreme conditions, this example can process 45-71 target objects, with a segmentation accuracy of 75.6%-87.3% and a recognition accuracy of 72.9%-82.5%.

[0106] Although the embodiments disclosed in this application are as described above, the content described is merely for the purpose of understanding this application and is not intended to limit this application. Any person skilled in the art to which this application pertains may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application; however, the scope of patent protection of this application shall still be determined by the scope defined in the appended claims.

Claims

1. A hybrid reality spatial perception method, characterized in that, include: Acquire scene images and perform two-dimensional segmentation of objects in the scene images based on a zero-shot semantic segmentation model; Based on the results of two-dimensional segmentation, a two-dimensional to three-dimensional spatial mapping is performed to generate a three-dimensional point cloud of the corresponding object; The 3D point cloud of the corresponding object is filtered and reconstructed into a mesh to generate an object-level 3D mesh model. Based on the query information from the user, multimodal semantic reasoning is performed on the visual language model to obtain the structured response output by the visual language model; The structured response output by the visual language model is analyzed and fed back into the mixed reality scene.

2. The mixed reality spatial perception method according to claim 1, after generating the object-level 3D mesh model, further includes: When it is necessary to determine the user's focus and query intent, the user's focus and query intent should be determined based on the multimodal interaction method.

3. The mixed reality spatial perception method according to claim 2, wherein, The multimodal interaction methods include: user eye movements, and / or user gestures, and / or user-inputted voice.

4. The mixed reality spatial perception method according to claim 1 or 2, wherein, The step of acquiring a scene image and performing two-dimensional segmentation of objects in the scene image based on a zero-shot semantic segmentation model includes: Acquire scene images captured by the mixed reality device and obtain the camera parameters corresponding to the scene images. The camera parameters include projection parameters and pose parameters used to establish the geometric relationship between image pixels and three-dimensional space. The scene image is input into the zero-shot semantic segmentation model to perform object-level two-dimensional segmentation on the objects in the scene image, and obtain a two-dimensional segmentation mask corresponding to at least one object.

5. The mixed reality spatial perception method according to claim 4, wherein, The generation of the corresponding 3D point cloud includes: Based on the two-dimensional segmentation mask and the camera parameters, the pixels in the two-dimensional segmentation mask are mapped to three-dimensional space to obtain a three-dimensional point set corresponding to the object.

6. The mixed reality spatial perception method according to claim 1 or 2, wherein, The generated object-level 3D mesh model includes: For the point cloud data of each of the objects, spatial filtering is performed on the point cloud to remove noise points and outliers; Triangulation is performed based on the two-dimensional projection of the filtered point cloud onto the image plane to generate a set of triangle indices. The generated set of triangle indices is then combined with the corresponding three-dimensional point cloud to construct the object-level three-dimensional mesh model.

7. The mixed reality spatial perception method according to claim 1 or 2, wherein, The structured response obtained from the visual language model output includes: The system obtains query information input by the user and combines it with the object-level 3D mesh model to construct input information for the visual language model. The input information includes image information and object identification information corresponding to the object. The input information is provided to the visual language model for multimodal semantic reasoning to obtain the structured response output by the visual language model.

8. The mixed reality spatial perception method according to claim 1 or 2, wherein, The structured response output by the parsed visual language model is fed back to the mixed reality scene, including: The semantic information output by the visual language model is associated with the corresponding object-level 3D mesh model, so that the object-level 3D mesh model has the object category, name or semantic attributes obtained by the visual language model, and the object-level semantic understanding results are output in the mixed reality scene.

9. A computer-readable storage medium storing computer-executable instructions for performing the mixed reality spatial perception method according to any one of claims 1-8.

10. A mixed reality spatial perception system, characterized in that, include: Spatial segmentation and fusion module, visual language model integration module; among them. The spatial segmentation and fusion module includes a zero-sample segmentation submodule, a 2D-to-3D mapping submodule, and a mesh optimization submodule; among them, The zero-shot segmentation submodule is configured to acquire scene images and perform two-dimensional segmentation of objects in the scene images based on a zero-shot semantic segmentation model. The 2D to 3D mapping submodule is configured to perform 2D to 3D spatial mapping based on the results of 2D segmentation to generate 3D point clouds of the corresponding objects; The mesh optimization submodule is configured to filter and reconstruct the 3D point cloud of the corresponding object, generating an object-level 3D mesh model. The visual language model integration module is configured to perform multimodal semantic reasoning based on the visual language model according to the query information from the user, obtain the structured response output by the visual language model, parse the structured response output by the visual language model and feed it back to the mixed reality scene.

11. The mixed reality spatial perception system according to claim 10, wherein, The visual language model integration module includes: an object-level encoding submodule, a visual language model processing submodule, and an object-level decoding submodule; wherein... The object-level encoding submodule is configured to encode the object-level 3D mesh model generated by the spatial segmentation and fusion module, construct an object-level input representation that can be understood by the visual language model; and encapsulate the object set information, the object's 2D projection information, and related image information into a structured data format and transmit them to the visual language model processing submodule. The visual language model processing submodule is set to annotate objects in the scene image based on the two-dimensional projection position of the object in the image, so as to clarify the correspondence between the object identifier and the image region. The object-level decoding submodule is configured to parse the structured response output by the visual language model processing submodule and map the parsing results back to the corresponding object-level 3D mesh model, so as to realize the presentation and interaction of object-level semantic information in mixed reality scenes.

12. The mixed reality spatial perception system according to claim 10 or 11 further includes a multimodal interaction module, configured to determine the user's attention object and query intent based on a multimodal interaction method when it is necessary to determine the user's attention object and query intent.

13. The mixed reality spatial perception system according to claim 12, wherein, The multimodal interaction module includes: an eye-tracking submodule, and / or a gesture interaction submodule, and / or a speech recognition submodule; wherein, The eye-tracking submodule is configured to acquire the user's gaze direction in real time and determine the object the user is currently focusing on based on the spatial relationship between the gaze ray and the object-level 3D mesh model. The gesture interaction submodule is set up to allow users to explicitly select objects using their hand gestures, serving as a supplement or alternative to eye-tracking input. The speech recognition submodule is configured to collect the user's speech input and convert the speech signal into text to obtain the user's query content or interaction instructions.

Images