Display method, device, equipment, medium and program

Abstract

The embodiment of the invention provides a display method, device, equipment, medium and program, XR equipment determines data of an environment map of a physical space and an association relationship between a coordinate system of the environment map and a coordinate system of a virtual scene, obtains a first environment image of the physical space, and displays the first environment image according to the first environment image and the data of the environment map. Determining a first pose of the XR equipment in a coordinate system of the environment map; calculating a second pose of the XR equipment in the coordinate system of the virtual scene according to the first pose and the association relationship; and displaying the virtual scene according to the second pose and the virtual scene. According to the method, the XR equipment can obtain the incidence relation between the coordinate system of the environment map and the coordinate system of the virtual scene through the XR equipment and share the incidence relation with other XR equipment, so that multi-equipment coordinate system sharing in a VR'large space 'scene is realized, additional equipment is not needed to assist in multi-equipment coordinate system sharing, the implementation is simple, and the implementation efficiency is improved. And the cost of sharing the coordinate system by multiple devices is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of electronic devices, and in particular, to a display method, apparatus, device, medium and program. Background Art

[0002] Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR is also a general term for multiple technologies such as Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR). It can bring an "immersive feeling" to the experiencer with seamless transitions between the virtual world and the real world.

[0003] With the diversification of XR applications, multi-person VR scene applications have emerged, for example, VR "large space" scenes, in which multiple users can use multiple VR devices in the same physical space to perform multi-person collaborative painting, design, multi-person games and other services. The basis for the implementation of VR "large space" is that multiple XR devices need to share the same coordinate system, that is, multiple XR devices obtain their own postures in the same coordinate system, and each XR device renders the picture based on its own posture information. In the prior art, multiple XR devices need to share the same coordinate system based on an external base station, that is, establish a coordinate system with the position of the base station as the origin, and measure their own posture relative to the base station.

[0004] The existing technology solution requires additional deployment of base stations, and the cost of multiple VR devices sharing the same coordinate system is high. Summary of the invention

[0005] Embodiments of the present application provide a display method, apparatus, device, medium, and program that can reduce the cost of sharing a coordinate system among multiple devices.

[0006] In a first aspect, an embodiment of the present application provides a display method, the method comprising:

[0007] Determine data of an environment map of a physical space, and an association relationship between a coordinate system of the environment map and a coordinate system of a virtual scene;

[0008] Acquire a first environment image of the physical space, and determine a first pose of the extended reality XR device in a coordinate system of the environment map according to the first environment image and data of the environment map;

[0009] Calculating a second posture of the XR device in the coordinate system of the virtual scene according to the first posture and the association relationship;

[0010] The virtual scene is displayed according to the second posture and the virtual scene.

[0011] In some optional implementations, before determining the data of the environment map of the physical space and the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene, the method further includes:

[0012] Collecting a plurality of second environment images of the physical space, and generating an environment map of the physical space according to the plurality of second environment images;

[0013] The association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to user input, or the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to a preset mark recognized in at least one second environment image.

[0014] In some optional implementations, determining the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene according to the user input includes:

[0015] The association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to the first point in the physical space determined by the user.

[0016] In some optional implementations, determining the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene according to the first point in the physical space determined by the user includes:

[0017] Determining an association relationship between a coordinate system of the environment map and a coordinate system of the virtual scene according to a first point and a direction in the physical space determined by a user;

[0018] Alternatively, the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to the first point and the second point in the physical space determined by the user.

[0019] In some optional implementations, a physical mark is provided at the first point in the physical space, or the first point in the physical space is a corner point.

[0020] In some optional implementations, the virtual scene matches the physical space.

[0021] In some optional implementations, after calculating the second pose of the XR device in the coordinate system of the virtual scene according to the first pose and the association relationship, the method further includes:

[0022] The second posture is sent so that other devices in the physical space display the virtual object corresponding to the XR device according to the second posture.

[0023] In some optional implementations, after determining the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene according to the user input, the method further includes:

[0024] The association relationship is sent to other devices in the physical space, so that the other devices determine their own positions in the coordinate system of the environmental map according to the association relationship and the environmental image of the physical space.

[0025] In some optional implementations, the data of the environment map includes three-dimensional coordinates of feature points in the environment map, pixel coordinates of the feature points, and descriptors of the feature points;

[0026] The determining, according to the first environment image and the data of the environment map, a first pose of the XR device in a coordinate system of the environment map includes:

[0027] Extracting feature points in the first environment image, and calculating descriptors of the feature points in the first environment image;

[0028] Matching the descriptors of the feature points in the first environment image with the descriptors of the feature points in the environment map;

[0029] According to the three-dimensional coordinates of the matched feature points, the pixel coordinates in the first environment image, and the pixel coordinates in the environment map, pose calculation is performed to obtain the first pose of the XR device in the coordinate system of the environment map.

[0030] In some optional implementations, the XR device uses a perspective n-point PnP algorithm to perform pose solution.

[0031] In some optional implementations, generating the environment map of the physical space according to the second environment image includes:

[0032] Extracting feature points in the second environment image, determining pixel coordinates of the feature points, and calculating descriptors of the feature points;

[0033] Calculating the three-dimensional coordinates of the feature point according to the pixel coordinates of the feature point;

[0034] The environment map is established according to the pixel coordinates, descriptors, and three-dimensional coordinates of the feature points.

[0035] On the other hand, an embodiment of the present application provides a display device, the device comprising:

[0036] A first determination module is used to determine data of an environment map of a physical space, and an association relationship between a coordinate system of the environment map and a coordinate system of a virtual scene;

[0037] A second determination module is used to obtain a first environment image of the physical space, and determine a first pose of the extended reality XR device in a coordinate system of the environment map according to the first environment image and data of the environment map;

[0038] A third determination module, configured to calculate a second posture of the XR device in the coordinate system of the virtual scene according to the first posture and the association relationship;

[0039] A display module is used to display the virtual scene according to the second posture and the virtual scene.

[0040] On the other hand, an embodiment of the present application provides an XR device, comprising: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute any of the methods described above.

[0041] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute any of the methods described above.

[0042] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the methods described above when executed by a processor.

[0043] The display method, device, equipment, medium and program provided in the embodiments of the present application are that the XR device determines the data of the environment map of the physical space, and the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene, obtains the first environment image of the physical space, and determines the first pose of the XR device in the coordinate system of the environment map based on the first environment image and the data of the environment map; calculates the second pose of the XR device in the coordinate system of the virtual scene based on the first pose and the association relationship; and displays the virtual scene based on the second pose and the virtual scene. In this method, the XR device can obtain the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene by itself, and share it with other XR devices, thereby realizing the sharing of coordinate systems by multiple devices in the VR "large space" scene, without the need for additional equipment to assist in the sharing of coordinate systems by multiple devices, which is simple to implement and reduces the cost of sharing coordinate systems by multiple devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of a multi-person VR scene applicable to an embodiment of the present application;

[0046] Figure 2 A flowchart of a display method provided in Embodiment 1 of the present application;

[0047] Figure 3 A signaling flow chart of the display method provided in Embodiment 2 of the present application;

[0048] Figure 4 A schematic diagram of the structure of a display device provided in Embodiment 3 of the present application;

[0049] Figure 5 A structural schematic diagram of the XR device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0052] An embodiment of the present application provides a display method, which can be applied to an XR device, including but not limited to a VR device, an AR device, or a MR device.

[0053] VR: Technology for creating and experiencing a virtual world. It generates a virtual environment by computation. It is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and can also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.). It realizes the simulation of the fusion and interactive three-dimensional dynamic vision and entity behavior of the virtual environment, immersing users in a simulated virtual reality environment, and realizing applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0054] AR: AR scenery refers to a simulated scenery in which at least one virtual object is superimposed on a physical scenery or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor for capturing images or videos of a physical scenery, which are representations of the physical scenery. The system combines the image or video with the virtual object and displays the combination on the opaque display. An individual uses the system to indirectly view the physical scenery via an image or video of the physical scenery and observes the virtual object superimposed on the physical scenery. When the system uses one or more image sensors to capture images of the physical scenery and uses those images to present the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, an electronic system for displaying an AR scenery may have a transparent or translucent display through which an individual can directly view the physical scenery. The system may display virtual objects on a transparent or translucent display so that an individual uses the system to observe virtual objects superimposed on the physical scenery. As another example, the system may include a projection system that projects virtual objects into a physical scenery. Virtual objects can be projected, for example, on a physical surface or as a hologram, so that individuals using the system observe virtual objects superimposed on a physical setting. Specifically, a technology that calculates the camera's posture parameters in the real world (or three-dimensional world, real world) in real time during the process of the camera capturing images, and adds virtual elements to the images captured by the camera based on the camera posture parameters. Virtual elements include, but are not limited to: images, videos, and three-dimensional models. The goal of AR technology is to integrate the virtual world into the real world on the screen for interaction.

[0055] MR: By presenting virtual scene information in the real scene, an interactive feedback information loop is established between the real world, the virtual world and the user to enhance the realism of the user experience. For example, computer-created sensory input (e.g., virtual objects) is integrated with sensory input from the physical scene or its representation in the simulated scene. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical scene. In addition, some electronic systems used to present MR scenes can monitor the orientation and / or position relative to the physical scene so that virtual objects can interact with real objects (i.e., physical elements from the physical scene or their representations). For example, the system can monitor movement so that virtual plants appear stationary relative to physical buildings.

[0056] Virtual reality devices refer to terminals that achieve virtual reality effects, which can usually be provided in the form of glasses, helmet-mounted displays (Head Mount Display, abbreviated as HMD), and contact lenses to achieve visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to this, and can be further miniaturized or enlarged according to actual needs.

[0057] Optionally, the virtual reality device (ie, XR device) described in the embodiments of the present application may include but is not limited to the following types:

[0058] (1) Mobile virtual reality devices support the configuration of mobile terminals (such as smart phones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations of virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0059] (2) All-in-one virtual reality devices have a processor for performing relevant calculations for virtual functions, and thus have independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal and have a high degree of freedom in use.

[0060] (3) Personal computer VR (PCVR) devices use the PC to perform relevant calculations and data output for virtual reality functions. External PC VR devices use the data output by the PC to achieve the effect of virtual reality.

[0061] The method provided in the embodiment of the present application can be applied in the "large space" scene of VR equipment. VR "large space" means that a relatively large space can accommodate multiple people to perform some VR services at the same time, so it is also called a multi-person VR scene. VR "large space" allows multiple users to experience VR in a physical space at the same time. The relative position information of the user in the physical space is transmitted to the virtual scene through positioning technology. The user can see the precise position of other users in the virtual scene, thereby achieving the effect of collaborative interaction of multiple users in the virtual scene.

[0062] Figure 1 Schematic diagram of a multi-person VR scene applicable to the embodiment of the present application, such as Figure 1 As shown, this scenario includes four users, each of whom uses an XR device. The four XR devices are located in the same physical space. The four XR devices can establish a multi-person service connection, which is used to jointly complete multi-person services, such as multi-person games, multi-person collaborative painting, multi-person collaborative design, etc., which can be completed by multiple users online at the same time. One of the users can initiate the establishment of a multi-person service connection, and after the multi-person service connection is established, the multi-person service can be started.

[0063] Taking a multiplayer VR game scenario as an example, multiple users each wear XR devices and team up to play games in the living room. One of the users can initiate an online invitation, and other users can accept the online invitation, thus establishing a multiplayer business connection.

[0064] In a multi-person VR scene, the relative spatial relationship of multiple users in the physical space is consistent with the relative spatial relationship of the virtual users corresponding to the multiple users in the 3D virtual scene, where the virtual users correspond one-to-one to the real users in the physical space, and the perspective of the virtual users is consistent with the perspective of the corresponding real users in the physical space. For example, a multi-person VR scene includes two users, and the two users are face to face in the physical space, so the virtual users corresponding to the two users are also face to face in the virtual scene. The two virtual users have different perspectives, and the two virtual users see different images of the virtual scene. Correspondingly, the images of the virtual scene seen by the users in the physical space through the XR device are also different.

[0065] In the VR "big space", the physical space where multiple users are located is relatively large, and the virtual scene is usually a virtual space generated by the designer. The structure and relative position relationship of the objects in the virtual space match the structure and relative position relationship of the objects in the physical space. After entering the virtual scene, each user needs to synchronize his or her posture in the physical space to the virtual scene. Among them, the posture of each user in the physical space is different, and each user needs to determine his or her posture in the physical space based on the same coordinate system. The same coordinate system is also called a shared coordinate system. A multi-user shared coordinate system means that multiple users use the same coordinate system, that is, multiple users determine their posture in the same coordinate system (that is, the same coordinate system). In actual scenarios, users and XR devices are bound, and a multi-user shared coordinate system can also be understood as a multi-device shared coordinate system. The posture of a user in the coordinate system refers to the posture of the device used by the user in the coordinate system.

[0066] In the prior art, the multi-device shared coordinate system needs to be realized with the help of other devices, which makes the cost of the multi-device shared coordinate system high. In order to solve the problem of the prior art, the embodiment of the present application provides a display method, which can realize the multi-device shared coordinate system without the need for other devices, thereby reducing the cost of the multi-device shared coordinate system.

[0067] Combination Figure 1 The scene shown is used to illustrate the display method provided in the embodiment of the present application. Figure 2 This is a flow chart of the display method provided in Embodiment 1 of the present application. The method of this embodiment can be executed by an XR device. The XR device can be any XR device in a multi-person VR scene, or a separate device independent of the multi-person VR scene. The XR device does not participate in multi-person services, such as Figure 2 As shown, the method provided in this embodiment includes the following steps:

[0068] S101, determining data of an environment map of a physical space, and an association relationship between a coordinate system of the environment map and a coordinate system of a virtual scene.

[0069] Taking the XR device as any one of the multiple XR devices in the VR "big space" as an example, before executing the method of this embodiment, a multi-person business connection can be established by the first XR device in the VR "big space", and the multiple XR devices corresponding to the multi-person business connection are located in the same physical space, and the first XR device is any one of the multiple XR devices.

[0070] The multi-person business connection includes multiple XR devices. In one implementation method, any one of the multiple XR devices can initiate an online invitation, and other XR devices accept the connection invitation, thereby completing the establishment of the multi-person business connection. The embodiment of the present application does not provide a detailed description of the process of establishing a multi-person business connection, and refers to the establishment of an existing multi-person business connection.

[0071] The multi-person service corresponds to a 3D virtual scene, which may be a VR scene, and multiple users share and interact with the virtual scene. Taking a game scene as an example, the virtual scene is a game scene, and users can play in the game scene, perform corresponding tasks, etc.

[0072] The environment map may be created by any one of the multiple XR devices and shared with other XR devices, and the environment map is used by the multiple XR devices to respectively determine their own positions in the coordinate system of the environment map.

[0073] After the multi-person service connection is established, a certain device may be designated to establish an environment map according to a rule, or a certain device may be configured in advance to establish an environment map, for example, the initiator of the multi-person service connection may be configured to establish an environment map.

[0074] Accordingly, the XR device can determine the data of the environment map of the physical space in the following ways:

[0075] Method 1: The XR device collects multiple second environment images of the physical space, and generates an environment map of the physical space based on the multiple second environment images of the physical space. After generating the environment map, the XR device stores the data of the environment map locally, and sends the data of the environment map to other XR devices in the multiple XR devices, so that multiple XR devices in the multi-person service can obtain the environment map.

[0076] The physical space is the real physical space (or real scene) where the multi-person business connection is located, which can be a room, living room, game room, etc. The user wears the XR device and moves in the physical space, and the camera of the XR device collects multiple second environment images of the physical space, and the multiple second environment images cover each position of the physical space. The XR device can establish an environment map corresponding to the physical space based on the posture information of the camera and the multiple second environment images captured by the camera.

[0077] Method 2: The XR device receives data of an environment map sent by other XR devices among the multiple XR devices. For example, the environment map is generated by a second XR device. After the second XR device generates the environment map, it sends the environment map to other XR devices among the multiple XR devices, so that multiple XR devices in the multi-person service can obtain the environment map.

[0078] The second XR device and the XR device use the same method to establish an environmental map. After the multi-person service is connected, the XR device or the second XR device can collect environmental images of the physical space in real time and generate an environmental map.

[0079] Method three: if the XR device or the second XR device has previously generated an environment map for the physical space, there is no need to generate the environment map again. The XR device or the second XR device only needs to retrieve the environment map and then use it.

[0080] Optionally, data can be transmitted between multiple XR devices via short-range communication, which includes but is not limited to Bluetooth, wireless fidelity (Wi-Fi), infrared data transmission, ZigBee, etc.

[0081] In one implementation, the XR device sends the environment map to other XR devices via the short-distance communication connection. In another implementation, the XR device that creates the environment map sends the environment map to the server, and multiple XR devices in the VR "large space" establish connections and communicate with the server. Other XR devices can actively request the environment map from the server, or the server, after receiving the environment map sent by the XR device that creates the environment map, actively sends the environment map to other XR devices. Optionally, the server can be a local server or a cloud server.

[0082] The data of the environment map includes visualization information of the physical space, and the visualization information includes pixel coordinates of feature points of each frame image on the map, 3D coordinates of feature points, descriptors of feature points, etc. Each frame image in the map is divided into a normal frame and a key frame.

[0083] The feature point of an image refers to a point where the grayscale value of the image changes dramatically or a point with a large curvature on the edge of the image (i.e., the intersection of two edges). Therefore, the feature point is a 2D point, and the position of the feature point can be represented by the pixel coordinates of the image where the feature point is located (i.e., the image from which the feature point is extracted).

[0084] The 3D coordinates of a feature point refer to the 3D coordinates of the feature point in the physical space, where feature points in different images may correspond to the same 3D point in the physical space. The 3D coordinates of a feature point can also be understood as a map point, which is a 3D point that comes from a real object in the physical space and can have a unique ID.

[0085] The descriptor of the feature point is used to describe the detected feature point. It is a binary coded descriptor. The descriptor can be used to describe the information around the feature point, such as describing the geometric features around the feature point. Commonly used descriptors include the binary robust independent elementary features (BRIEF) descriptor.

[0086] Optionally, XR devices can use Simultaneous Localization and Mapping (SLAM) algorithms to build environmental maps of physical spaces. SLAM algorithms use sensors to build maps and structures in unknown environments and locate the position and orientation of devices. Related steps include: sensors read / laser scan video images or point clouds and other data; the front-end visual odometry (VO) estimates the change in camera pose at two moments through feature matching, direct registration and other algorithms; the back-end receives the camera pose measured by the visual odometry at different moments, and optimizes it to obtain a globally consistent trajectory and map. The process involves loop detection, cumulative error processing and map creation.

[0087] VIO-SLAM, based on Visual-Inertial Odometry (VIO), also called Visual-inertial System (VINS), integrates camera and IMU data to achieve SLAM, which can improve the performance of SLAM algorithm.

[0088] Exemplarily, the XR device extracts feature points in the second environment image of the physical space, determines the pixel coordinates of the feature points, and calculates the descriptors of the feature points. Based on the pixel coordinates of the feature points, the 3D coordinates of the feature points are calculated, and the environment map is established based on the pixel coordinates, descriptors, and three-dimensional coordinates of the feature points.

[0089] The XR device can use a feature extractor to extract feature points from an image. The feature extractor can be a function that extracts features from an image, and the features extracted by the function are represented by feature vectors. The feature extractor can also be a neural network, which includes but is not limited to: Convolutional Neural Networks (CNN), Multi-layer perceptron neural networks (MLP), Transformer structure neural networks, etc.

[0090] For example, the position of the feature point can be detected using the FAST feature point detection algorithm or the Harris corner point detection algorithm or the Scale-invariant feature transform (SIFT), the Speeded Up Robust Features (SURF) and other algorithms. After the position of the feature point (i.e., the pixel coordinates of the feature point) is detected, the descriptor of the feature point is calculated.

[0091] After determining the pixel coordinates of the feature points, the 3D coordinates of the feature points can be calculated by triangulation method, and the 3D coordinates of the feature points can also be calculated by depth map. This is just an example, and the embodiment of the present application does not limit the method for calculating the 3D coordinates of the feature points.

[0092] In the process of building an environment map of the physical space, the coordinate system of the environment map is first established. The coordinate system of the environment map is uniquely determined by the origin of the environment map and the direction of the coordinate axis. The XR device can select a 3D point in the physical space as the origin of the coordinate system and specify the direction of the coordinate axis to generate the data of the environment map based on the established coordinate system of the environment map.

[0093] After determining the data of the environment map of the physical space, the XR device can determine the association between the coordinate system of the environment map and the coordinate system of the 3D virtual scene based on the user input. The association is the offset between the coordinate system of the virtual scene and the coordinate system of the environment map. The coordinate system of the virtual scene is determined when the virtual scene is established, while the coordinate system of the environment map is determined by the XR device itself when generating the environment map. The coordinate system of the environment map may not be the ideal coordinate system expected by the user. The ideal coordinate system can better present the content of the virtual scene. Therefore, in this embodiment, the coordinate system of the environment map is adjusted so that the adjusted coordinate system of the environment map is the ideal coordinate system expected by the user.

[0094] The offset between the coordinate system of the environment map and the coordinate system of the virtual scene includes rotation information and / or translation information of the coordinate system. The translation information refers to the movement distance of the coordinate system on the X, Y, and Z axes, and the rotation information refers to the rotation angle of the coordinate system on the X, Y, and Z axes.

[0095] The offset between the coordinate system of the environment map and the coordinate system of the virtual scene can be expressed by a matrix or a Euclidean transformation, or can be expressed in other forms, which is not limited in the embodiments of the present application.

[0096] Associating the coordinate system of the environment map with the coordinate system of the virtual scene can be understood as binding the coordinate system of the environment map with the coordinate system of the 3D virtual scene. By binding the coordinate system of the environment map with the coordinate system of the 3D virtual scene, the posture of the XR device in the physical space is synchronized with the posture of the virtual character corresponding to the XR device in the virtual scene. When the posture of the XR device in the physical space changes, the posture of the virtual character corresponding to the XR device in the virtual scene also changes accordingly, or the virtual character follows the movement of the XR device. The movement of the XR device includes movement and / or rotation. Therefore, the user can adjust the posture of the virtual character corresponding to the XR by adjusting the posture of the XR device.

[0097] In one implementation, the XR device determines the association between the coordinate system of the environment map and the coordinate system of the virtual scene based on the first point in the physical space determined by the user. Specifically, the first point determined by the user is associated with the coordinate origin of the virtual scene, that is, the first point in the environment map is used as the origin of the coordinate system of the virtual scene. The user can indicate the position of the first point through various input methods. For example, the ray emitted by the controller (such as a handle) of the XR device can be displayed, and the intersection of the ray and the ground at a certain moment is determined as the position of the first point, or the first point is determined by the position of the user's gaze combined with other operations, or the first point is determined by the user's voice input combined with semantic recognition of the environment image, etc. The embodiment of the present application does not limit the method of determining the first point.

[0098] In addition to the origin, the coordinate system also needs to define a direction. The direction can be pre-defined or determined by user input. For example, after the origin is selected, an adjustable arrow is displayed in a direction parallel to the ground (i.e., perpendicular to gravity). The direction of the arrow is adjusted based on the user's input, and the direction of the coordinate system is determined according to the direction of the adjusted arrow. The embodiment of the present application does not limit the method for determining the direction of the coordinate system.

[0099] Optionally, in the process of obtaining user input, the XR device can work in perspective mode, that is, obtaining the image of the physical space and displaying it on the XR device after processing, so that the user can see the physical space and input more conveniently.

[0100] Optionally, a physical mark is provided at the first point in the physical space, and the physical mark is used to prompt the user to determine the location of the physical mark as the first point. Alternatively, the first point in the physical space is a corner point, and the corner point can be understood as a wall corner point or a table corner point in the physical space, which can usually be the intersection of two or three straight lines perpendicular to each other in the space. There may be multiple corner points in the physical space, and the user can indicate the position of a corner point of the physical space through the controller of the XR device, and use the indicated corner point as the first point.

[0101] Optionally, the XR device determines the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene based on the first point in the physical space determined by the user. Specifically, this can be done in the following two ways: (1) determining the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene based on the first point and direction in the physical space determined by the user; (2) determining the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene based on the first point and second point in the physical space determined by the user.

[0102] In method (1), the user not only indicates the first point, but also indicates the direction. The first point and the direction are the origin and the direction of the coordinate system of the environmental map after adjustment. Then, based on the origin and the direction of the coordinate system of the environmental map after adjustment, and the origin and the direction of the coordinate system of the virtual scene, the offset between the coordinate system of the environmental map and the coordinate system of the virtual scene is determined.

[0103] In method (2), the user indicates the origin and direction of the coordinate system of the adjusted environment map by indicating two points, wherein the first point is the origin of the coordinate system of the adjusted environment map, and the direction from the first point to the second point is the direction of any coordinate axis of the coordinate system of the adjusted environment map.

[0104] Optionally, the XR device may also determine the association between the coordinate system of the environment map and the coordinate system of the virtual scene based on a preset marker identified in at least one second environment image. By identifying the preset marker, a position of an origin may be obtained, and optionally, a direction may also be obtained.

[0105] Exemplarily, the preset mark can be a T-shaped mark located on the ground in the physical space, or a device with a special light spot (like a VR handle with a light spot that can identify the position), and the position and direction of an origin are determined based on the optical recognition of the "T"-shaped mark or the light spot. For example, the "T"-shaped mark is recognized, and the connection point of the two strokes of the "T"-shaped mark is used as the origin, and the directions of the two strokes are used as the directions of the two coordinate axes of the coordinate system.

[0106] S102: Acquire a first environment image of the physical space, and determine a first position of the XR device in a coordinate system of the environment map according to the first environment image and data of the environment map.

[0107] The multiple XR devices corresponding to the multi-person business connection share the environment map, and the environment map is used for the multiple XR devices to respectively determine their own first pose in the coordinate system of the environment map. The multiple XR devices determine their own pose in the coordinate system of the environment map based on the same environment map, thereby realizing the sharing of the coordinate system by multiple devices. The multiple XR devices use the same method to determine the first pose.

[0108] The first environment image is the environment image captured by the XR device in real time through the camera. The first posture of the XR device is a 6-degree-of-freedom (6DOF) data, including the position and posture of the XR device.

[0109] Exemplarily, the XR device extracts feature points in the first environment image, calculates descriptors of the feature points in the first environment image, matches the descriptors of the feature points in the first environment image with the descriptors of the feature points in the environment map, and performs pose solution based on the 3D coordinates of the matched feature points, the pixel coordinates in the first environment image, and the pixel coordinates in the environment map to obtain the first pose of the XR device in the coordinate system of the environment map.

[0110] The XR device can use any existing feature point matching algorithm to match the feature points of the first environment image with the feature points of the environment map. Commonly used feature point matching algorithms include but are not limited to: brute force matching method, cross matching, random sampling consensus (RANdom SAmple Consensus, referred to as RANSAC), etc.

[0111] The XR device can use any existing posture solution method to determine the first posture of the XR device in the coordinate system of the environment map, and this embodiment is not limited to this.

[0112] Exemplarily, the XR device can use a perspective-n-Point (PnP) algorithm to solve the matching results. PnP is a method for solving 3D to 2D point pair motion. It describes how to estimate the camera's pose (i.e., the first pose of the XR device) when the coordinates of n 3D points and the pixel coordinates of these points are known, where the value of n is greater than or equal to 2. Exemplarily, the value of n is 3 or 5.

[0113] Multiple XR devices share the environment image of the physical space in which they are located. Each device determines its own first pose in the coordinate system of the environment map based on the shared environment map and the current environment image it has taken, thereby realizing a multi-device shared coordinate system. The method of multi-device shared coordinate system does not require additional equipment assistance and is simple to implement, thereby reducing the cost of multi-device shared coordinate system.

[0114] S103: Calculate a second pose of the XR device in the coordinate system of the virtual scene according to the first pose and the association relationship.

[0115] The association relationship may be an offset between a coordinate system of the environment map and a coordinate system of the virtual scene, where the offset includes rotation information and / or translation information of the coordinate system.

[0116] When the XR device creates an environment map, it generates the origin and direction of the coordinate system of the environment map, determines the association relationship based on the user input, and saves the association relationship. Subsequently, other XR devices will also obtain the association relationship when obtaining the environment map, and determine their second posture in the coordinate system of the virtual scene based on the association relationship.

[0117] Exemplarily, the second pose of the XR device in the coordinate system of the virtual scene is calculated by the following formula:

[0118]

[0119] Among them, T offset Indicates the association relationship (i.e., the offset between the coordinate system of the environment map and the coordinate system of the virtual scene), Indicates the first position of the XR device in the coordinate system of the environment map. It represents the second pose of the XR device in the coordinate system of the virtual scene. The first pose and the second pose are both 6DOf data, which contain the rotation and translation information of the XR device. It can be represented by Euclidean transformation or other forms.

[0120] S104: Displaying a virtual scene according to the second posture and the virtual scene.

[0121] In one implementation, the XR device renders the virtual scene according to the second posture to obtain a rendering result, and displays the rendering result of the virtual scene. In this method, the XR device performs local rendering through a rendering engine, and after rendering, the rendering result is displayed on the screen.

[0122] In another implementation, the XR device sends the second posture to the rendering device, and receives the rendering result of the virtual scene sent by the rendering device, where the rendering result is obtained by the rendering device rendering the virtual scene according to the second posture. In this method, the virtual scene is rendered by a remote server, which can be a cloud rendering server.

[0123] Optionally, after calculating the second posture, the XR device sends the second posture to other devices in the physical space, so that the other devices in the physical space display the virtual object corresponding to the XR device according to the second posture. The virtual object may be a virtual character corresponding to the XR device.

[0124] Optionally, the virtual scene matches the physical space. In the VR "big space", the virtual scene matches the physical space, which means that the internal structure and external outline of the virtual scene are the same or corresponding to the physical space, and the structure and position relationship of the virtual objects in the virtual scene can be the same as the structure and position relationship of the physical objects in the physical space. When each user moves in the physical space, the virtual character corresponding to each user moves accordingly in the virtual scene, and the relative position relationship of each user in the physical space is consistent with the relative position relationship of the virtual character corresponding to each user in the virtual scene.

[0125] In the method of this embodiment, the XR device determines the data of the environment map of the physical space, and the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene, obtains the first environment image of the physical space, and determines the first pose of the XR device in the coordinate system of the environment map based on the first environment image and the data of the environment map; calculates the second pose of the XR device in the coordinate system of the virtual scene based on the first pose and the association relationship; and displays the virtual scene based on the second pose and the virtual scene. In this method, the XR device can obtain the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene by itself, and share it with other XR devices, thereby realizing the sharing of coordinate systems by multiple devices in the VR "large space" scene. No additional equipment is required to assist in the sharing of coordinate systems by multiple devices, which is simple to implement and reduces the cost of sharing coordinate systems by multiple devices.

[0126] Based on the first embodiment, the second embodiment of the present application provides a display method. This embodiment is described by taking two devices as examples. Figure 3 The signaling flow chart of the display method provided in the second embodiment of the present application is as follows: Figure 3 As shown, the method provided in the embodiment includes the following steps.

[0127] S201. A first XR device and a second XR device establish a multi-person service connection.

[0128] The first XR device and the second XR device are located in the same physical space. It can be understood that a multi-person business connection can be established between the first XR device and the second XR device through one or more pieces of information.

[0129] S202. The first XR device collects multiple second environment images of the physical space, and generates an environment map using a SLAM algorithm based on the multiple second environment images.

[0130] The first XR device moves in the physical space and captures multiple frames of environmental images of the physical space through a camera, wherein the multiple frames of environmental images have different camera poses. The first XR device extracts feature points in the image of the physical space, determines the pixel coordinates of the feature points, and calculates the descriptors of the feature points. Based on the pixel coordinates of the feature points, the three-dimensional coordinates of the feature points are calculated, and an environmental map is established based on the pixel coordinates, descriptors, and three-dimensional coordinates of the feature points.

[0131] S203: The first XR device saves the environment map, and determines an association relationship between a coordinate system of the environment map and a coordinate system of the virtual scene according to a user input.

[0132] The virtual scene is a 3D virtual scene corresponding to a multi-person service, and the first XR device and the second XR device access the virtual scene through a multi-person service connection.

[0133] S204: The first XR device sends the data of the environment map and the association relationship to the second XR device.

[0134] The data of the environment map includes information of the coordinate system of the environment map, 3D coordinates of feature points in the environment map, pixel coordinates of the feature points, and descriptors of the feature points.

[0135] It should be noted that the execution order of step S201 can be before step S202 or after step S202. For example, after executing steps S202 and S203, the first device executes step S201 to establish a multi-person business connection with the second XR device, and then sends the data of the environmental map and the association relationship to the second XR device through the multi-person business connection.

[0136] In addition, the first XR device and the second XR device may establish a connection directly or through a server. When the first XR device and the second XR device establish a connection directly, the first XR device may send the data of the environment map and the association relationship through the directly established connection. When the first XR device and the second XR device establish a connection through a server, the first XR device sends the data of the environment map and the association relationship to the server, and the server sends the data of the environment map and the association relationship to the second XR device.

[0137] S205: The second XR device saves the environment map and the association relationship.

[0138] The second XR device receives the environment map and the association relationship sent by the first XR device, and saves the environment map and the association relationship. The environment map is used to subsequently determine its own position and posture in the coordinate system of the environment map.

[0139] S206: The first XR device captures a first environment image of the physical space, and determines a first pose of the first XR device in the coordinates of the environment map according to the first environment image and data of the environment map.

[0140] Exemplarily, the first XR device extracts feature points in the first environment image, calculates descriptors of the feature points in the first environment image, matches the descriptors of the feature points in the first environment image with the descriptors of the feature points in the environment map, and performs pose solution based on the 3D coordinates of the matched feature points, the pixel coordinates in the first environment image, and the pixel coordinates in the environment map to obtain the pose of the first XR device in the coordinate system of the environment map.

[0141] S207: The first XR device calculates a second posture of the first XR device in the coordinate system of the virtual scene according to the first posture and the association relationship, and displays the virtual scene according to the second posture and the virtual scene.

[0142] S206′: The second XR device collects a third environment image of the physical space, and determines a first position of the second XR device in the coordinate system of the environment map according to the third environment image and data of the environment map.

[0143] Exemplarily, the second XR device extracts feature points in the third environment image, calculates descriptors of the feature points in the third environment image, matches the descriptors of the feature points in the third environment image with the descriptors of the feature points in the environment map, and performs pose solution based on the 3D coordinates of the matched feature points, the pixel coordinates in the third environment image, and the pixel coordinates in the environment map to obtain the first pose of the second XR device in the coordinate system of the environment map.

[0144] S207': The second XR device calculates a second posture of the second XR device in the coordinate system of the virtual scene according to the first posture and the association relationship, and displays the virtual scene according to the second posture and the virtual scene.

[0145] It should be noted that steps S206 and S207 and steps S206 ′ and S207 ′ are performed in parallel, that is, the two XR devices locate their own postures in parallel.

[0146] The first environment image is an image captured by the camera of the first XR device, and the third environment image is an image captured by the camera of the second XR device. The first XR device and the second XR device have different postures, so the first environment image and the third environment image are different. The two XR devices each perform camera posture calculation based on the captured environment image and the environment map to obtain their first posture in the coordinate system of the environment map, and determine their second posture in the coordinate system of the virtual scene based on their first posture and the association relationship, and render the image of the virtual scene based on the second posture. Therefore, the relative position relationship of the two XR devices in the physical space is consistent with the relative position relationship of the virtual characters corresponding to the two XR devices in the virtual scene.

[0147] In order to better implement the display method of the embodiment of the present application, the embodiment of the present application also provides a display device. Figure 4 A schematic diagram of the structure of a display device provided in Embodiment 3 of the present application is shown in FIG. Figure 4 As shown, the display device 100 may include:

[0148] A first determination module 11 is used to determine data of an environment map of a physical space, and an association relationship between a coordinate system of the environment map and a coordinate system of a virtual scene;

[0149] A second determination module 12 is used to obtain a first environment image of the physical space, and determine a first pose of the extended reality XR device in a coordinate system of the environment map according to the first environment image and data of the environment map;

[0150] A third determination module 13, configured to calculate a second posture of the XR device in the coordinate system of the virtual scene according to the first posture and the association relationship;

[0151] The display module 14 is configured to display the virtual scene according to the second posture and the virtual scene.

[0152] In some optional implementations, the method further includes a fourth determining module, configured to:

[0153] Collecting a plurality of second environment images of the physical space, and generating an environment map of the physical space according to the plurality of second environment images;

[0154] The association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to user input, or the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to a preset mark recognized in at least one second environment image.

[0155] In some optional implementations, the fourth determination module is specifically used to determine the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene according to the first point in the physical space determined by the user.

[0156] In some optional implementations, the fourth determination module is specifically used to: determine the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene according to the first point and direction in the physical space determined by the user;

[0157] Alternatively, the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to the first point and the second point in the physical space determined by the user.

[0158] In some optional implementations, a physical mark is provided at the first point in the physical space, or the first point in the physical space is a corner point.

[0159] In some optional implementations, the virtual scene matches the physical space.

[0160] In some optional implementations, a sending module is also included, which is used to: send the second posture so that other devices in the physical space display the virtual object corresponding to the XR device according to the second posture.

[0161] In some optional implementations, a sending module is also included, which is used to: send the association relationship to other devices in the physical space, so that the other devices can determine their own position in the coordinate system of the environmental map based on the association relationship and the environmental image of the physical space.

[0162] In some optional implementations, the data of the environment map includes three-dimensional coordinates of feature points in the environment map, pixel coordinates of the feature points, and descriptors of the feature points;

[0163] The second determining module 12 is specifically used for:

[0164] Extracting feature points in the first environment image, and calculating descriptors of the feature points in the first environment image;

[0165] Matching the descriptors of the feature points in the first environment image with the descriptors of the feature points in the environment map;

[0166] According to the three-dimensional coordinates of the matched feature points, the pixel coordinates in the first environment image, and the pixel coordinates in the environment map, pose calculation is performed to obtain the first pose of the XR device in the coordinate system of the environment map.

[0167] In some optional implementations, the XR device uses a perspective n-point PnP algorithm to perform pose solution.

[0168] In some optional implementations, the fourth determining module is specifically configured to:

[0169] Extracting feature points in the second environment image, determining pixel coordinates of the feature points, and calculating descriptors of the feature points;

[0170] Calculating the three-dimensional coordinates of the feature point according to the pixel coordinates of the feature point;

[0171] The environment map is established according to the pixel coordinates, descriptors, and three-dimensional coordinates of the feature points.

[0172] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here.

[0173] The above describes the device 100 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0174] The embodiment of the present application also provides an XR device. Figure 5 A structural diagram of an XR device provided in Embodiment 4 of the present application, such as Figure 5 As shown, the XR device 200 may include:

[0175] The memory 21 and the processor 22, the memory 21 is used to store the computer program and transmit the program code to the processor 22. In other words, the processor 22 can call and run the computer program from the memory 21 to implement the method in the embodiment of the present application.

[0176] For example, the processor 22 may be configured to execute the above method embodiments according to instructions in the computer program.

[0177] In some embodiments of the present application, the processor 22 may include but is not limited to: a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0178] In some embodiments of the present application, the memory 21 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0179] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the XR device.

[0180] like Figure 5As shown, the XR device 200 may further include: a transceiver 23 , which may be connected to the processor 22 or the memory 21 .

[0181] The processor 22 may control the transceiver 23 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 23 may include a transmitter and a receiver. The transceiver 23 may further include an antenna, and the number of antennas may be one or more.

[0182] Understandable, although Figure 5 Not shown, the XR device 200 may also include a camera module, a wireless fidelity WIFI module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be repeated here.

[0183] It should be understood that the various components in the XR device are connected via a bus system, wherein the bus system includes, in addition to a data bus, a power bus, a control bus, and a status signal bus.

[0184] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0185] The present application also provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the XR device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the XR device executes the corresponding process in the method embodiment, which will not be repeated here for the sake of brevity.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0187] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0188] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A display method, characterized in that: include: Determine data of an environment map of a physical space, and an association relationship between a coordinate system of the environment map and a coordinate system of a virtual scene; Acquire a first environment image of the physical space, and determine a first position of the extended reality XR device in a coordinate system of the environment map according to the first environment image and data of the environment map; Calculating a second posture of the XR device in the coordinate system of the virtual scene according to the first posture and the association relationship; The virtual scene is displayed according to the second posture and the virtual scene.

2. The method according to claim 1, characterized in that Before determining the data of the environment map of the physical space and the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene, the method further includes: Collecting a plurality of second environment images of the physical space, and generating an environment map of the physical space according to the plurality of second environment images; The association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to user input, or the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to a preset mark recognized in at least one second environment image.

3. The method according to claim 2, characterized in that The determining, according to the user input, the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene comprises: The association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to the first point in the physical space determined by the user.

4. The method according to claim 3, characterized in that: The step of determining the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene according to the first point in the physical space determined by the user includes: Determining an association relationship between a coordinate system of the environment map and a coordinate system of the virtual scene according to a first point and a direction in the physical space determined by a user; Alternatively, the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene is determined according to the first point and the second point in the physical space determined by the user.

5. The method according to claim 3, characterized in that: A physical mark is set at the first point in the physical space, or the first point in the physical space is a corner point.

6. The method according to claim 1, characterized in that The virtual scene matches the physical space.

7. The method according to claim 1, characterized in that After calculating the second posture of the XR device in the coordinate system of the virtual scene according to the first posture and the association relationship, the method further includes: The second posture is sent so that other devices in the physical space display the virtual object corresponding to the XR device according to the second posture.

8. The method according to claim 2, characterized in that: After determining the association relationship between the coordinate system of the environment map and the coordinate system of the virtual scene according to the user input, the method further includes: The association relationship is sent to other devices in the physical space, so that the other devices determine their own positions in the coordinate system of the environmental map according to the association relationship and the environmental image of the physical space.

9. The method according to any one of claims 1 to 8, characterized in that: The data of the environment map includes three-dimensional coordinates of feature points in the environment map, pixel coordinates of feature points and descriptors of feature points; The determining, according to the first environment image and the data of the environment map, a first pose of the XR device in a coordinate system of the environment map includes: Extracting feature points in the first environment image, and calculating descriptors of the feature points in the first environment image; Matching the descriptors of the feature points in the first environment image with the descriptors of the feature points in the environment map; According to the three-dimensional coordinates of the matched feature points, the pixel coordinates in the first environment image, and the pixel coordinates in the environment map, pose calculation is performed to obtain the first pose of the XR device in the coordinate system of the environment map.

10. The method according to claim 9, characterized in that The XR device uses a perspective n-point PnP algorithm to perform pose calculation.

11. The method according to claim 2, characterized in that The step of generating the environment map of the physical space according to the second environment image comprises: Extracting feature points in the second environment image, determining pixel coordinates of the feature points, and calculating descriptors of the feature points; Calculating the three-dimensional coordinates of the feature point according to the pixel coordinates of the feature point; The environment map is established according to the pixel coordinates, descriptors, and three-dimensional coordinates of the feature points.

12. A display device, characterized in that: include: A first determination module is used to determine data of an environment map of a physical space, and an association relationship between a coordinate system of the environment map and a coordinate system of a virtual scene; A second determination module is used to obtain a first environment image of the physical space, and determine a first pose of the extended reality XR device in a coordinate system of the environment map according to the first environment image and data of the environment map; A third determination module, configured to calculate a second posture of the XR device in the coordinate system of the virtual scene according to the first posture and the association relationship; A display module is used to display the virtual scene according to the second posture and the virtual scene.

13. An extended reality device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images