Multi-mode fusion mixed reality multi-person space alignment method and system

Abstract

The invention discloses a multi-mode fusion mixed reality multi-person space alignment method and system. The method comprises the following steps: step 1, system initialization; step 2, space alignment of the VPS mode; step 3, carrying out space alignment of the image tracking mode; step 4, carrying out space alignment of the object tracking mode; step 5, space alignment of the center cooperation mode; step 6, carrying out space alignment in a double-anchor mode; 7, real-time updating and dynamic adaptation are carried out; and step 8, multi-person synchronization. By combining various space alignment schemes, the precision and adaptability of space alignment can be effectively improved, and the method is suitable for various environments and use scenes. The system provided by the invention has high flexibility, can perform adaptive adjustment according to real-time environment change and user requirements, and ensures that the virtual object keeps a consistent alignment effect in a plurality of user viewing angles.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality (MR) technology, and in particular to a multi-mode fused mixed reality multi-person space alignment method and system, which are applied to scenes such as virtual reality and augmented reality. Background Art

[0002] With the development of mixed reality technology, especially its application in multi-person interactive scenarios, how to ensure accurate alignment of virtual content between different users has become an important technical challenge. Existing technologies mainly include VPS (visual positioning system), image tracking, object tracking and single anchor point system. However, existing technologies have the following problems or shortcomings: (1) The positioning accuracy of VPS decreases in environments with insufficient light or a lot of obstructions.

[0003] (2) Image tracking is limited by the availability of preset images, making it difficult to achieve effective alignment in environments without clear landmarks.

[0004] (3) Object tracking is highly complex to implement and has strict requirements on object features.

[0005] (4) The spatial alignment accuracy of a single anchor point is insufficient, which may cause misalignment of virtual objects in multi-person scenes. Summary of the invention

[0006] The present invention provides a multi-mode fusion mixed reality multi-person spatial alignment method and system, which aims to solve the problems of insufficient spatial alignment accuracy, poor environmental adaptability and high implementation complexity in multi-person MR scenarios in the prior art. The present invention combines multiple spatial alignment schemes to improve the flexibility and accuracy of the system, ensuring effective virtual content alignment in different environments.

[0007] The technical solution adopted by the present invention to solve its technical problem is: A multi-mode fusion mixed reality multi-person spatial alignment method includes the following steps: step 1. system initialization; step 2. spatial alignment of VPS mode; step 3. spatial alignment of image tracking mode; step 4. spatial alignment of object tracking mode; step 5. spatial alignment of center collaboration mode; step 6. spatial alignment of dual anchor point mode; step 7. real-time update and dynamic adaptation; step 8. multi-person synchronization.

[0008] A multi-mode fusion mixed reality multi-person space alignment system includes a VPS file, an image tracking module, an object tracking module, a center collaboration module and a dual anchor point module; wherein: the VPS file is responsible for loading and using the stored space file for precise positioning; the image tracking module recognizes and matches predefined images through a camera; the object tracking module is used to recognize physical target objects and perform space alignment; the center collaboration module allows users to manipulate virtual anchor points through gestures for space synchronization; the dual anchor point module creates two anchor points through a user's pinch gesture and calculates direction vectors for alignment.

[0009] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements a mixed reality multi-person space alignment method of multi-mode fusion when executing the computer program.

[0010] A storage medium stores a computer program, which, when executed by a processor, implements a mixed reality multi-person space alignment method for multi-mode fusion.

[0011] The present invention also has the following additional technical features: As a further specific optimization of the technical solution of the present invention: Step 1. System initialization includes the following steps: After the system is started, it enters the selection phase of the space alignment mode; the system checks and selects the most suitable space alignment mode in turn according to the following priority order: Step 1.1: The system will detect whether there is a VPS space file pre-stored in the device; if there is a pre-stored VPS file, the system will load the VPS file first, and perform spatial positioning and alignment through the three-dimensional environment data and visual features in the VPS file to achieve high-precision virtual content overlay; In step 1.2, if the VPS file is not found, the system will start an environmental scan and try to identify trackable images or objects through the device's camera in order to select other alignment methods; In step 1.3, when the VPS file, image or object cannot be detected, the system will enable the backup alignment scheme, and the user can enable the center collaboration module or the dual anchor mode through gesture interaction to perform basic spatial alignment.

[0012] As a further specific optimization of the technical solution of the present invention: Step 2. The spatial alignment of the VPS mode includes the following steps: When there is a VPS space file in the device, the system will use the file for spatial alignment. The VPS captures the visual feature points in the environment through the camera and matches them with the pre-stored three-dimensional map or visual feature database to ensure the accurate superposition of virtual content.

[0013] As a further specific optimization of the technical solution of the present invention: Step 3. Spatial alignment of the image tracking mode includes the following steps: If the system fails to find the VPS file, the image tracking mode will be started; the system uses the device's camera to try to identify a predefined image (the predefined image includes a poster or a QR code); if the target image is identified, the system will perform spatial alignment through the image tracking module to ensure that the virtual object is accurately aligned with the image.

[0014] As a further specific optimization of the technical solution of the present invention: Step 4. Spatial alignment of the object tracking mode includes the following steps: If no available VPS file or image is detected, the system will enter the object tracking mode; the device will scan the surrounding environment and identify physical target objects (physical target objects include furniture or equipment); if the physical target object can be identified, the system will align it through the object tracking module to ensure accurate synchronization between the virtual object and the physical target object.

[0015] As a further specific optimization of the technical solution of the present invention: Step 5. The spatial alignment of the center collaboration mode includes the following steps: When the system cannot detect the VPS file, image or object, the user can control the center collaboration module through gestures; the gesture-controlled center collaboration module allows the user to grab a virtual center collaboration object through gestures, and then place the virtual center collaboration object with other users in the same spatial position in reality, and all virtual scenes will be loaded around the center point to achieve basic spatial alignment.

[0016] As a further specific optimization of the technical solution of the present invention: Step 6. The spatial alignment of the dual anchor point mode includes the following steps: When the system cannot detect the VPS file, image or object, the user can also select the dual anchor point mode; the user creates two anchor points through a pinch gesture, and the system will calculate the direction vector between the two anchor points and align the virtual object in this direction to ensure that the virtual scene remains consistent in all user perspectives.

[0017] As a further specific optimization of the technical solution of the present invention: Step 7. Real-time update and dynamic adaptation includes the following steps: During the operation of the system, whether through VPS files, image tracking, object tracking, central collaboration module or dual anchor point mode, the system will update the position and orientation of virtual objects in real time; user interactions (interactions include gestures or head movements) will be reflected in real time on virtual objects, ensuring that all users see consistent content in a shared virtual environment.

[0018] As a further specific optimization of the technical solution of the present invention: Step 8. Multi-person synchronization includes the following steps: All users' devices are connected through the network, and the system synchronizes each user's operation and the state of the virtual object to ensure that all users in the same physical space can see the same virtual object position and state.

[0019] Compared with the prior art, the present invention has the following advantages: Advantage 1: By combining multiple spatial alignment schemes, the present invention can effectively improve the accuracy and adaptability of spatial alignment and is suitable for a variety of environments and usage scenarios.

[0020] Advantage 2: The system of the present invention is highly flexible and can be adaptively adjusted according to real-time environmental changes and user needs to ensure that virtual objects maintain a consistent alignment effect in multiple user perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0024] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0025] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects. Example

[0026] A multi-mode fusion mixed reality multi-person spatial alignment method includes the following steps: step 1. system initialization; step 2. spatial alignment of VPS mode; step 3. spatial alignment of image tracking mode; step 4. spatial alignment of object tracking mode; step 5. spatial alignment of center collaboration mode; step 6. spatial alignment of dual anchor point mode; step 7. real-time update and dynamic adaptation; step 8. multi-person synchronization.

[0027] Working principle of the mixed reality multi-person spatial alignment method for multi-modal fusion: The system of the present invention selects the optimal alignment scheme by detecting whether there is a VPS file in the device. If there is a VPS file, VPS is used first for spatial alignment; if not, it detects whether there are trackable images or objects in the environment in turn. When no other alignment methods are detected, the user can achieve basic spatial alignment through the gesture control center collaborative module or dual anchor points. Example

[0028] A multi-mode fusion mixed reality multi-person space alignment method includes the following steps: Step 1. System initialization.

[0029] After the system starts, it enters the space alignment mode selection phase. The system checks and selects the most suitable space alignment mode in the following priority order: In step 1.1, the system will detect whether there is a VPS space file pre-stored in the device. If there is a pre-stored VPS file, the system will load the VPS file first, and perform spatial positioning and alignment through the three-dimensional environment data and visual features in the VPS file to achieve high-precision virtual content overlay.

[0030] In step 1.2, if the VPS file is not found, the system will initiate an environment scan and try to identify trackable images or objects through the device's camera so that other alignment methods can be selected.

[0031] In step 1.3, when the VPS file, image or object cannot be detected, the system will enable the backup alignment scheme, and the user can enable the center collaboration module or the dual anchor mode through gesture interaction to perform basic spatial alignment.

[0032] Step 2. Spatial alignment of VPS mode.

[0033] When a VPS space file exists in the device, the system will use it for spatial alignment. VPS captures visual feature points in the environment through the camera and matches them with the pre-stored 3D map or visual feature database to ensure accurate superposition of virtual content.

[0034] Step 3. Spatial alignment of image tracking patterns.

[0035] If the system fails to find the VPS file, it will start the image tracking mode. The system uses the device's camera to try to recognize a predefined image (predefined images include posters or QR codes). If the target image is recognized, the system will perform spatial alignment through the image tracking module to ensure that the virtual object is accurately aligned with the image.

[0036] Step 4. Spatial alignment of object tracking patterns.

[0037] If no available VPS file or image is detected, the system will enter object tracking mode; the device will scan the surrounding environment and identify physical target objects (physical target objects include furniture or equipment). If the physical target object can be identified, the system will align it through the object tracking module to ensure accurate synchronization between the virtual object and the physical target object.

[0038] Step 5. Spatial alignment of central synergistic patterns.

[0039] When the system cannot detect VPS files, images or objects, users can use gestures to manipulate the center collaboration module. The gesture manipulation center collaboration module allows users to grab a virtual center collaboration object through gestures, and then place the virtual center collaboration object in the same spatial position in reality with other users. All virtual scenes will be loaded around the center point to achieve basic spatial alignment.

[0040] Step 6. Spatial alignment of dual anchor point mode.

[0041] When the system cannot detect VPS files, images or objects, users can also select dual anchor point mode. Users create two anchor points through a pinch gesture, and the system will calculate the direction vector between the two anchor points and align the virtual object in this direction to ensure that the virtual scene is consistent in all user perspectives.

[0042] Step 7. Real-time update and dynamic adaptation.

[0043] During the operation of the system, whether through VPS files, image tracking, object tracking, center collaboration module or dual anchor mode, the system will update the position and orientation of virtual objects in real time. User interactions (including gestures or head movements) will be reflected in real time on virtual objects, ensuring that all users see consistent content in a shared virtual environment.

[0044] Step 8. Multi-person synchronization.

[0045] All users' devices are connected through the network, and the system synchronizes each user's operations and the status of virtual objects to ensure that all users in the same physical space can see the same virtual object position and status. Example

[0046] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements a mixed reality multi-person space alignment method of multi-mode fusion when executing the computer program.

[0047] In one embodiment, the electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a mixed reality multi-person space alignment method of multi-mode fusion is implemented. Example

[0048] A storage medium stores a computer program, which, when executed by a processor, implements a mixed reality multi-person space alignment method for multi-mode fusion.

[0049] A storage medium having a computer program stored thereon, which, when executed by a processor, implements a mixed reality multi-person space alignment method for multi-mode fusion. A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0050] In one embodiment, any reference to memory, storage, database or other medium used may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0051] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0052] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-modal fusion mixed reality multi-person spatial alignment method, characterized by: The following steps are included: Step 1. System initialization; Step 2. Spatial alignment of VPS mode; Step 3. Spatial alignment of image tracking mode; Step 4. Spatial alignment of object tracking mode; Step 5. Spatial alignment of center collaboration mode; Step 6. Spatial alignment of dual anchor mode; Step 7. Real-time update and dynamic adaptation; Step 8. Multi-person synchronization.

2. The multi-modal fusion mixed reality multi-person space alignment method according to claim 1 is characterized by: Step 1. System initialization includes the following steps: After the system starts, it enters the space alignment mode selection phase; the system checks and selects the most suitable space alignment mode in the following priority order: Step 1.1: The system will detect whether there is a VPS space file pre-stored in the device; if there is a pre-stored VPS file, the system will load the VPS file first, and perform spatial positioning and alignment through the three-dimensional environment data and visual features in the VPS file to achieve high-precision virtual content overlay; In step 1.2, if the VPS file is not found, the system will start an environmental scan and try to identify trackable images or objects through the device's camera in order to select other alignment methods; In step 1.3, when the VPS file, image or object cannot be detected, the system will enable the backup alignment scheme, and the user can enable the center collaboration module or the dual anchor mode through gesture interaction to perform basic spatial alignment.

3. The multi-modal fusion mixed reality multi-person space alignment method according to claim 1, characterized in that: Step 2. VPS mode spatial alignment includes the following steps: When there is a VPS space file in the device, the system will use it for spatial alignment. VPS captures visual feature points in the environment through the camera and matches them with the pre-stored 3D map or visual feature database to ensure accurate superposition of virtual content; Step 3. Spatial alignment in image tracking mode includes the following steps: If the system fails to find the VPS file, it will start the image tracking mode. The system will use the device's camera to try to recognize the predefined image. If the target image is recognized, the system will use the image tracking module to perform spatial alignment to ensure that the virtual object is accurately aligned with the image. Step 4. Spatial alignment of object tracking mode includes the following steps: If no available VPS file or image is detected, the system will enter object tracking mode; the device will scan the surrounding environment and identify physical target objects; if the physical target object can be identified, the system will align it through the object tracking module to ensure accurate synchronization between the virtual object and the physical target object.

4. The multi-modal fusion mixed reality multi-person space alignment method according to claim 1, characterized in that: Step 5. The spatial alignment of the center coordination model includes the following steps: When the system cannot detect VPS files, images or objects, users can use gestures to control the center collaboration module; the gesture control center collaboration module allows users to grab a virtual center collaboration object through gestures, and then place the virtual center collaboration object in the same spatial position in reality with other users. All virtual scenes will be loaded around the center point to achieve basic spatial alignment.

5. The multi-modal fusion mixed reality multi-person space alignment method according to claim 1, characterized in that: Step 6. The spatial alignment of the dual anchor point mode includes the following steps: When the system cannot detect VPS files, images or objects, users can also select dual anchor point mode; users create two anchor points through a pinch gesture, and the system will calculate the direction vector between the two anchor points and align the virtual object in this direction to ensure that the virtual scene remains consistent from all user perspectives.

6. The multi-modal fusion mixed reality multi-person space alignment method according to claim 1, characterized in that: Step 7. Real-time updating and dynamic adaptation includes the following steps: During the operation of the system, whether through VPS files, image tracking, object tracking, central collaboration module or dual anchor point mode, the system will update the position and orientation of virtual objects in real time; user interactions will be reflected in virtual objects in real time, ensuring that all users see consistent content in a shared virtual environment.

7. The multi-modal fusion mixed reality multi-person space alignment method according to claim 1, characterized in that: Step 8. Multi-person synchronization includes the following steps: All users' devices are connected through the network, and the system synchronizes each user's operations and the status of virtual objects to ensure that all users in the same physical space can see the same virtual object position and status.

8. A multi-modal fusion mixed reality multi-person spatial alignment system, characterized by: It includes VPS file, image tracking module, object tracking module, center coordination module and dual anchor module; among them: The VPS file is responsible for loading and using the stored space files for precise positioning; The image tracking module recognizes and matches predefined images through the camera; The object tracking module is used to identify physical target objects and perform spatial alignment; The center collaboration module allows users to synchronize space by manipulating virtual anchor points through gestures; The dual anchor point module creates two anchor points through the user's pinch gesture and calculates the direction vector for alignment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-mode fusion mixed reality multi-person spatial alignment method as described in any one of claims 1 to 9 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-mode fusion mixed reality multi-person space alignment method as described in any one of claims 1 to 9 is implemented.

Images