Spatial calibration method, apparatus, device, medium, and program

Abstract

Embodiments of the present application provide a space calibration method, device, equipment, medium and program, the method comprises: acquiring the environment image of the space, determining the video perspective image of the space according to the environment image of the space;According to the position of the virtual ray emitted by the virtual object in the video perspective image, the ground and the ceiling in the space are calibrated;According to the calibration result of the environment image, the ground and the ceiling, the position of the wall corner line in the space is determined;The calibration line of the ground, the ceiling and the wall corner line are displayed in the video perspective image.The method of the embodiment, the space is calibrated in a semi-automatic manner, wherein the ground and the ceiling are manually calibrated, and the wall corner line is automatically calibrated, thereby simplifying the calibration process of the space, improving the calibration efficiency, and automatically calibrating the wall corner line, which can avoid the problem of deviation of the calibration result caused by hand shake and visual error of the user.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a spatial calibration method, apparatus, device, medium, and program. Background Technology

[0002] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR is also a collective term for various technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). By integrating the visual interaction technologies of these three technologies, it brings users a seamless "immersive" experience between the virtual and real worlds.

[0003] In MR applications, room capture is used to calibrate a space, including the location of the floor, walls, and ceiling. Specifically, the user uses rays emitted from the MR device's handle to calibrate the floor, walls, and ceiling. The calibration results can be applied to virtual house viewing, virtual furniture placement, virtual decoration, and even games.

[0004] However, existing calibration methods require manual calibration by the user, which is cumbersome and results in low accuracy due to user hand tremors or visual differences. Summary of the Invention

[0005] This application provides a spatial calibration method, apparatus, device, medium, and program that calibrates space in a semi-automatic manner. The ground and ceiling are calibrated manually, while the corner lines are calibrated automatically. This simplifies the spatial calibration process, improves calibration efficiency, and the automatic calibration of the corner lines avoids calibration result deviations caused by user hand tremors and visual errors.

[0006] In a first aspect, embodiments of this application provide a spatial calibration method, the method comprising:

[0007] Acquire an environmental image of the space, and determine a video perspective image of the space based on the environmental image of the space;

[0008] The ground and ceiling in the space are calibrated based on the position of the virtual rays emitted by the virtual objects in the video perspective image;

[0009] Based on the environmental image and the calibration results of the ground and ceiling, determine the position of the corner line in the space;

[0010] The floor, ceiling, and corner lines are displayed in the video perspective image.

[0011] In some embodiments, determining the position of the corner line in the space based on the environmental image and the calibration results of the ground and ceiling includes:

[0012] A line detection algorithm is used to detect candidate lines that are perpendicular to the ground or the ceiling within the space;

[0013] Based on the calibration results of the ground and ceiling, the corner lines in the space are determined from the candidate straight lines.

[0014] In some embodiments, after displaying the ground, ceiling, and corner lines in the video perspective image, the method further includes:

[0015] Replace the target corner line in the space according to the first user operation; and / or

[0016] A new corner line is added within the space based on the second user's operation.

[0017] In some embodiments, replacing the target corner line in the space according to the first user operation includes:

[0018] Based on the user's calibration operation of the target corner point corresponding to the target corner line, a replacement corner line corresponding to the target corner point is generated and displayed. The starting point of the replacement corner line is the target corner point, which is the corner point formed by the ground and the wall. The replacement corner line is used to replace the target corner line.

[0019] In some embodiments, before generating a replacement corner line corresponding to the target corner line based on the user's calibration result of the target corner line, the method further includes:

[0020] Receive a deletion command for the target corner line;

[0021] The target corner line is deleted according to the deletion command.

[0022] In some embodiments, replacing the target corner line in the space according to the first user operation includes:

[0023] The replacement corner line is generated based on the user's calibration operation for the replacement corner line corresponding to the target corner line.

[0024] In some embodiments, before generating the replacement corner line based on the user's calibration operation on the replacement corner line corresponding to the target corner line, the method further includes:

[0025] Receive a deletion command for the target corner line;

[0026] The target corner line is deleted according to the deletion command.

[0027] In some embodiments, adding a new corner line within the space according to a second user operation includes:

[0028] Based on the user's calibration operation of the first corner point in the space, a new corner line corresponding to the first corner point is generated and displayed. The first corner point is the corner point formed by the ground and the wall.

[0029] In some embodiments, the method operates on an extended reality (XR) device, the XR device having mixed reality (MR) applications and MR services running on it, and the method further includes:

[0030] When a user launches the current application, the MR service obtains the category configuration information of the current application, which is used to indicate whether the current application is an MR application.

[0031] When the category configuration information indicates that the current application is an MR application, the MR service is started.

[0032] If the category configuration information indicates that the current application is not an MR application, then it is determined that the MR service will not be started.

[0033] In some embodiments, the MR service obtains the category configuration information of the current application, including:

[0034] The MR service reads the category configuration information of the current application from the manifest file of the current application.

[0035] In some embodiments, the MR service obtains the category configuration information of the current application, including:

[0036] The current application is launched according to the launch operation;

[0037] After the current application is started, if the current application is an MR application, the MR application sends the category configuration information of the current application to the MR service;

[0038] The MR service receives the category configuration information of the current application.

[0039] On the other hand, embodiments of this application provide a spatial calibration device, the device comprising:

[0040] The acquisition module is used to acquire environmental images of the space and determine video perspective images of the space based on the environmental images of the space.

[0041] The manual calibration module is used to calibrate the ground and ceiling in the space based on the position of the virtual rays emitted by the virtual objects in the video perspective image;

[0042] An automatic calibration module determines the position of the corner lines in the space based on the environmental image and the calibration results of the ground and ceiling.

[0043] The display module is used to display the calibration lines of the ground and ceiling, as well as the corner lines, in the video perspective image.

[0044] On the other hand, embodiments of this application provide an XR device, the XR device including: a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program stored in the memory to perform the method as described in any of the above.

[0045] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in any of the foregoing descriptions.

[0046] On the other hand, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the above-mentioned embodiments.

[0047] This application provides a spatial calibration method, apparatus, device, medium, and program. The method includes: acquiring an environmental image of the space; determining a video perspective image of the space based on the environmental image; calibrating the ground and ceiling in the space based on the positions of virtual rays emitted by virtual objects in the video perspective image; determining the positions of corner lines in the space based on the environmental image, the calibration results of the ground and ceiling, and the corner lines; and displaying the calibration lines of the ground and ceiling, as well as the corner lines, in the video perspective image. This embodiment uses a semi-automatic method to calibrate the space, where the ground and ceiling are calibrated manually, while the corner lines are calibrated automatically. This simplifies the spatial calibration process, improves calibration efficiency, and the automatic corner line calibration avoids calibration result deviations caused by user hand tremors and visual errors. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A schematic diagram of a scene for spatial calibration;

[0050] Figure 2 This is a flowchart of the spatial calibration method provided in Embodiment 1 of this application;

[0051] Figure 3 A flowchart of the spatial calibration method provided in Embodiment 2 of this application;

[0052] Figure 4 A flowchart of the spatial calibration method provided in Embodiment 3 of this application;

[0053] Figure 5 This is a schematic diagram of the spatial calibration device provided in Embodiment 4 of this application;

[0054] Figure 6 This is a schematic diagram of the structure of an XR device provided in Embodiment 5 of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0057] To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts involved in all embodiments of this application will be appropriately explained as follows:

[0058] The spatial calibration method provided in this application can be applied to XR devices, including but not limited to VR devices, AR devices, and MR devices.

[0059] VR: A technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of virtual environment, interactive three-dimensional dynamic visual scenes and simulation of physical behavior, immersing users in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0060] AR: An AR scene refers to a simulated scene in which at least one virtual object is superimposed on a physical scene or its representation. For example, an electronic system may have an opaque display and at least one imaging sensor for capturing images or videos of the physical scene, which are representations of the physical scene. The system combines the images or videos with virtual objects and displays this combination on the opaque display. Individuals use the system to indirectly view the physical scene via images or videos of the physical scene and observe the virtual objects superimposed on the physical scene. When the system uses one or more image sensors to capture images of the physical scene and uses those images to present the AR scene on an opaque display, the displayed images are referred to as video pass-through. Alternatively, the electronic system for displaying the AR scene may have a transparent or semi-transparent display through which an individual can directly view the physical scene. The system may display virtual objects on the transparent or semi-transparent display, allowing an individual to observe the virtual objects superimposed on the physical scene using the system. As another example, the system may include a projection system that projects virtual objects onto the physical scene. Virtual objects can be projected, for example, onto a physical surface or as holograms, allowing individuals to observe virtual objects superimposed on a physical setting using the system. Specifically, a technique involves calculating the camera's pose parameters in the real world (or 3D world, the real world) in real time during image acquisition, and adding virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive experiences.

[0061] MR (Mixed Reality): By presenting virtual scene information within a real-world setting, an interactive feedback loop is established between the real world, the virtual world, and the user to enhance the realism of the user experience. For example, computer-created sensory input (e.g., virtual objects) is integrated with sensory input or its representation from a physical setting within a simulated scene. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0062] Virtual reality devices refer to terminals that achieve virtual reality effects. They are typically available in the form of glasses, head-mounted displays (HMDs), or contact lenses to enable visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these and can be further miniaturized or enlarged according to actual needs.

[0063] Optionally, the virtual reality device (i.e., XR device) described in the embodiments of this application may include, but is not limited to, the following types:

[0064] 1) Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0065] 2) All-in-one virtual reality devices have processors for performing virtual functions, thus having independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal, and have a high degree of freedom of use.

[0066] 3) PC VR (Computer-based Virtual Reality) devices utilize a PC for calculations and data output related to virtual reality functions. External PC VR devices use the data output from the PC to achieve virtual reality effects.

[0067] The prerequisite for using MR applications is room calibration, which includes spatial calibration and furniture calibration within the space. The space consists of a three-dimensional space composed of the floor, walls, and ceiling. Spatial calibration includes the calibration of the floor, walls, and ceiling. Typically, after the spatial calibration is completed, the furniture within the space is calibrated.

[0068] XR devices feature video seethrough (VST) technology. VST technology uses the camera of the head-mounted device (i.e., XR device) to capture a real-time view of the surrounding environment, which is then combined with computer technology and displayed on an opaque display. This gives the user the feeling that they can see the real world directly through the head-mounted device, hence the name "see-through" function, which increases the user's perception of the surrounding environment.

[0069] When users wear the headset indoors, with the VST function enabled, spatial calibration is performed. Users can see the calibration process and results in real time, enhancing user interactivity.

[0070] Currently, spatial calibration is performed using ray calibration. Ray calibration refers to calibration using a virtual ray emitted by a virtual controller. The virtual controller is usually controlled by the user. For example, the user controls the movement of the virtual controller through a physical controller (such as a handle). The virtual ray moves with the movement of the virtual controller. The starting point of the virtual ray is the position of the virtual controller, and the ending point of the virtual ray is the wall, floor, or ceiling to be calibrated.

[0071] Figure 1 A scene diagram for spatial calibration, such as Figure 1 As shown, the user wears a head-mounted device, and the head-mounted device's camera collects environmental data of the space and generates corresponding video perspective images. The user controls the endpoint of the virtual ray in the video perspective image to move to the position that needs to be calibrated by using a controller, thereby forming a calibration frame corresponding to the ground, a calibration frame corresponding to the ceiling, and calibration frames corresponding to each wall. Typically, each calibration frame is a rectangular calibration frame, and adjacent calibration frames are connected to finally form a closed space.

[0072] Taking ground calibration as an example, in one exemplary method, forming a calibration frame requires the following three steps: move the endpoint of the virtual ray to a corner of the ground, click to confirm and form the first calibration point, control the virtual ray to draw a line, click to confirm and form the second calibration point; draw out a plane, click to confirm and form the third calibration point, the third calibration point is perpendicular to the line formed by the first and second calibration points, thus forming a rectangular calibration frame.

[0073] Currently, during spatial calibration, the floor, ceiling, and walls all require manual calibration by the user. Manual calibration means that the calibration process requires interaction with the user, and the calibration can only be completed based on the user's actions. After manually calibrating the walls, the intersection line of two adjacent walls is the corner line, and the position of the corner line is obtained based on the wall calibration operation. However, the manual calibration process is cumbersome, and the calibration results may be inaccurate due to hand tremors or visual differences during manual calibration.

[0074] To address the problems of the prior art, embodiments of this application provide a spatial calibration method. Figure 2 This is a flowchart of a spatial calibration method provided in Embodiment 1 of this application, which is applied in an XR device. Figure 2 As shown, the method provided in this embodiment includes the following steps.

[0075] S101. Obtain the environmental image of the space, and determine the video perspective image of the space based on the environmental image of the space.

[0076] The environmental image of a space is an image of the real environment in which the space is located. The camera of the head-mounted device collects the environmental image of the space in real time, and determines the video perspective image of the space based on the environmental image. The environmental image of the space is a 2D image, and the video perspective image is a 3D image obtained after processing. The head-mounted device displays the video perspective image to the user through a monitor, so that the user can see the outside world.

[0077] S102. Based on the position of the virtual ray emitted by the virtual object in the video perspective image, mark the ground and ceiling in the space.

[0078] Users can manually calibrate the ground and ceiling. Manual calibration means that the calibration process requires interaction with the user, and the calibration can only be completed based on the user's operation.

[0079] For example, a virtual object and a virtual ray emitted by the virtual object are displayed in a video perspective image. The user uses the virtual ray to mark the ground and ceiling in the space. The virtual object includes, but is not limited to, a virtual controller, a virtual hand model, or a virtual hand model + virtual controller (presented as a virtual hand simulating holding a virtual controller). The starting point of the virtual ray is the position of the virtual object, and the virtual ray moves with the virtual object. The movement of the virtual object is operated by the user, and the movement of the virtual object can also be understood as the movement of the virtual ray. The user can control the movement of the virtual object or the virtual ray through one or more of the following methods: the physical controller of the XR device (e.g., a gamepad), voice interaction, gesture interaction, and eye tracking.

[0080] Taking ground calibration as an example, when the endpoint of the virtual ray moves to a corner of the ground, the first calibration point is formed according to the confirmation command. Then, the endpoint of the virtual ray continues to move until it reaches the next corner of the ground. During the movement of the endpoint of the virtual ray, a straight line appears in the video perspective image based on the movement trajectory of the endpoint of the virtual ray. From the user's perspective, it is equivalent to pulling out a straight line. The second calibration point is formed according to the confirmation command. The user continues to move the endpoint of the virtual ray until it reaches the next corner of the ground. Since the first and second calibration points have formed a straight line, a plane appears in the video perspective image as the endpoint of the virtual ray moves. This plane expands or shrinks as the endpoint of the virtual ray moves. From the user's perspective, it is similar to pulling the straight line formed by the first and second calibration points to create a plane. When the endpoint of the virtual ray moves to the third calibration point, the third and fourth calibration points are formed according to the confirmation command for the third calibration point.

[0081] The XR device estimates the 3D coordinates of the virtual ray's endpoint based on the controller's inertial measurement unit (IMU) data and the coordinates of the virtual ray's start and end points in the image. The 3D coordinates of the virtual ray's endpoint are the 3D coordinates of the corresponding calibration point in the world coordinate system, thus enabling the acquisition of the 3D coordinates of each calibration point on the ground and ceiling.

[0082] S103. Based on the environmental image and the calibration results of the ground and ceiling, determine the position of the corner lines in the space.

[0083] The calibration results for the ground and ceiling are the 3D coordinates of each calibration point on the ground and ceiling. The main purpose of calibrating the ground and ceiling is to determine the height of the ground and the height of the ceiling.

[0084] In this embodiment, an automatic calibration method is used when calibrating the corner lines. Automatic calibration, as opposed to manual calibration, means that no user intervention is required during the calibration process; the calibration result is automatically generated based on the environmental image. The position of the corner line can be represented by the positions of its two endpoints, which are also called the corner points corresponding to the corner line. In other words, the position of the corner line is represented by its two corner points; determining the corner line determines the position of its corresponding corner points.

[0085] In one exemplary approach, a line detection algorithm is used to detect candidate lines perpendicular to the ground or ceiling in a space, and the corner lines in the space are determined from the candidate lines based on the calibration results of the ground and ceiling.

[0086] Line detection is a classic low-level vision task. The embodiments of this application can employ any existing line detection algorithm to detect lines in space from spatially corresponding images. Commonly used line detection algorithms are divided into two categories: traditional algorithms and deep learning algorithms.

[0087] Traditional algorithms include, but are not limited to, the Hough transform algorithm and line segment detection (LSD). LSD is a "perceptual clustering" method that relies on carefully designed image features and detection strategies, and its accuracy and algorithm complexity are better than those of Hough line detection.

[0088] Deep learning algorithms include, but are not limited to: Wireframe networks and line-convolutional neural networks (LCNNs).

[0089] XR devices can use one or more environmental images to detect straight lines, detecting multiple straight lines within a space. These detected straight lines may include corner lines, ceiling corner lines, baseboard lines, furniture edges, and door / window edges. Specifically, a corner line is a straight line between two walls, a baseboard line is a straight line between the floor and a wall, and a ceiling corner line is a straight line between a wall and a ceiling.

[0090] The starting point of the corner line is the ground, and the ending point is the ceiling. That is, the corner line is perpendicular to both the ground and the ceiling. Therefore, a line perpendicular to either the ground or the ceiling can be selected as a candidate line from the detected lines in the space. Then, the corner line in the space is determined from the candidate lines based on the ground height and ceiling height. The height of the corner line is equal to the height difference between the ground and the ceiling. First, lines with heights equal to or close to this height difference are selected from the candidate lines. Then, the relative positional relationship between the lines determines whether a line is a corner line.

[0091] There are multiple corner lines in a space. For a relatively square space, there may be four corner lines, and for a space with corners, there may be six or more corner lines.

[0092] It's understandable; this is just an example, and other methods can be used to detect the corner lines in a space.

[0093] S104. Display the ground, ceiling, and corner lines in the video perspective image.

[0094] In this embodiment, the XR device generates and displays a video perspective image of the space in real time based on the environmental images captured by the camera. It also displays the calibration results in real time during the calibration process. After the user calibrates the ceiling and floor, the calibration lines for the floor and ceiling are displayed in the video perspective image. At this point, the user needs to manually calibrate the corner lines. The XR device automatically generates the corner lines based on the environmental images and displays them in the video perspective image. The calibration lines for the floor and ceiling, along with the corner lines, connect to form a closed space, thus completing the calibration of the space.

[0095] In this embodiment, an environmental image of the space is acquired, and a video perspective image of the space is determined based on the environmental image. The ground and ceiling in the space are calibrated according to the positions of virtual rays emitted by virtual objects in the video perspective image. Based on the environmental image and the calibration results of the ground and ceiling, the positions of the corner lines in the space are determined. The calibration lines of the ground and ceiling, as well as the corner lines, are displayed in the video perspective image. The method of this embodiment uses a semi-automatic approach to calibrate the space, wherein the ground and ceiling are calibrated manually, while the corner lines are calibrated automatically. This simplifies the space calibration process, improves calibration efficiency, and the automatic calibration of the corner lines avoids the problem of calibration result deviations caused by user hand tremors and visual errors.

[0096] Based on Embodiment 1, optionally, after automatically generating corner lines, if the automatically generated corner lines are inaccurate, the target corner line in the space can be replaced according to the first user operation, where the target corner line is the corner line to be replaced. Alternatively, if a corner line is missing from the automatically generated corner lines, a new corner line can be added to the space according to the second user operation. The first user operation and the second user operation do not represent a single operation, but can include a series of operations with a sequential order.

[0097] Figure 3 The flowchart of the spatial calibration method provided in Embodiment 2 of this application is as follows: Figure 3 As shown, the method provided in this embodiment includes the following steps.

[0098] S201. Obtain environmental images of the space and determine video perspective images of the space based on the environmental images of the space.

[0099] S202. The ground and ceiling in the space are calibrated according to the position of the virtual ray emitted by the virtual object in the video perspective image.

[0100] S203. Based on the environmental images and the calibration results of the ground and ceiling, determine the location of the corner lines in the space.

[0101] S204. Display the ground, ceiling, and corner lines in the video perspective image.

[0102] S205. Receive the deletion instruction for the target corner line, and delete the target corner line according to the deletion instruction.

[0103] Based on the position of the corner lines displayed in the video perspective image, the user judges whether the position of the corner lines is accurate. If the user finds that the position of a corner line is inaccurate, the user can select the corner line, which becomes the target corner line. Then, the target corner line is deleted, and the XR device generates a deletion command based on the deletion operation.

[0104] Deleting the target corner line includes not only deleting the target corner line displayed in the video perspective image, but also deleting the calibration result of the target corner line, that is, deleting the location information of the target corner line stored in the map of the space.

[0105] For example, the user moves the cursor to the target corner line, clicks on the target corner line to select it, and after the user clicks on the target corner line, a delete option will be displayed, such as whether to delete it. After the user selects the "yes" option, the target corner line will be deleted, and after the target corner line is deleted, it will no longer be displayed in the video perspective image.

[0106] S206. Based on the user's calibration operation of the target corner point, generate the replacement corner line corresponding to the target corner point, display the replacement corner line, and use the replacement corner line to replace the target corner line.

[0107] After deleting the target corner line, a replacement corner line can be generated based on user operation. For example, the user controls the endpoint of the virtual ray to move to the target corner point, which is the starting position of the target corner line as desired by the user. When the user inputs a confirmation command, the 3D coordinates of the target corner point are determined, and a replacement corner line is generated with the position of the target corner point as the starting point. The direction of the replacement corner line is perpendicular to the ground and the ceiling.

[0108] It should be noted that, in the embodiments of this application, the corner point can be the intersection of the ground and the two walls, i.e., the corner point is located on the ground; or it can be the intersection of the ceiling and the two walls, i.e., the corner point is located on the ceiling. The ground and the ceiling are two parallel planes, and a corner line has two endpoints: one endpoint is the corner point on the ground, and the other endpoint is the corner point on the ceiling.

[0109] When generating a corner line, the user can choose to calibrate a corner point on the ground or on the ceiling. If the user chooses a corner point on the ground, a replacement corner line will be generated starting from that point, ending on the ceiling. If the user chooses a corner point on the ceiling, a replacement corner line will be generated starting from that point, ending on the ground.

[0110] Optionally, after the user inputs a confirmation command for the target corner point, the process of generating the replacement corner line is dynamically displayed in the video perspective image. The replacement corner line starts from the target corner point and extends vertically upwards to the ceiling, or extends vertically downwards from the ceiling to the ground.

[0111] In this embodiment, the user deletes the target corner line before calibrating the target corner point. Optionally, in other embodiments of this application, the user may not delete the target corner line before calibrating the target corner point; instead, the target corner line is automatically deleted after generating the replacement corner line corresponding to the target corner point.

[0112] Optionally, the difference between the replaced corner lines and the automatically generated corner lines can be displayed. For example, the replaced corner lines use green lines, while the automatically generated corner lines use yellow lines.

[0113] Before the corner line is replaced, the marked floor, ceiling and corner line form a closed space. Similarly, after the replacement corner line is generated, the replacement corner line is connected to the adjacent marked line in the space to form a closed space.

[0114] S207. Based on the user's calibration operation of the first corner point in the space, generate a new corner line corresponding to the first corner point and display the new corner line.

[0115] If the captured environmental image is inaccurate, or if the detection algorithm fails to automatically recognize some corner lines, the user can add new corner lines. After the user calibrates the first corner point, a new corner line corresponding to the first corner point is automatically generated. The calibration method for the first corner point and the generation process of the new corner line are the same as those in step S206, which describes the calibration method for the target corner point and the generation process for the replacement corner line. These details will not be repeated here.

[0116] In the actual calibration process, steps S205 and S206 may be executed only, while step S207 may not be executed; steps S207 may be executed only, while steps S205 and S206 may not be executed; or steps S205, S206 and S207 may all be executed.

[0117] In this embodiment, after automatically generating the corner lines of the space, the target corner lines in the space can be replaced according to the first user's operation; and / or, new corner lines can be added to the space according to the second user's operation. This achieves the adjustment of the corner lines in the space, making the final calibrated corner lines more accurate.

[0118] Optionally, in other embodiments of this application, step S206 can be replaced by step S206'.

[0119] S206' Generate a replacement corner line based on the user's calibration operation for the replacement corner line corresponding to the target corner line.

[0120] In step S206, the user marks a corner point, and a corner line starting from that corner point can be automatically generated based on the user-marked corner point. However, in this embodiment, the user needs to mark a corner line.

[0121] For example, the user controls the endpoint of the virtual ray to move to the second corner point on the ground, which is the starting position of the replacement corner line that the user expects. The user inputs a confirmation command, and then controls the endpoint of the virtual ray to continue moving to draw a straight line. When the endpoint of the virtual ray moves to the third corner point on the ceiling, the user inputs a confirmation command, and a replacement corner line is generated with the second corner point as the starting point and the third corner point as the ending point.

[0122] Based on Embodiment 1 and Embodiment 2, Embodiment 3 of this application provides a spatial calibration method. This embodiment describes the complete process from the start of MR application to the end of spatial calibration. Figure 4 A flowchart of the spatial calibration method provided in Embodiment 3 of this application is shown below. Figure 4 As shown, the method provided in this embodiment includes the following steps.

[0123] S301, The user opens the application.

[0124] XR devices can run both 3D and 2D applications. 2D applications refer to traditional applications running on electronic devices such as mobile phones, computers, and tablets. 2D applications display 2D images to the user; examples include traditional video playback applications, short video applications, mobile games, and computer games. 3D applications, on the other hand, display 3D images to the user. 3D applications can provide users with 3D virtual scenes, real scenes, or mixed scenes that overlay virtual and real scenes. 3D applications include MR applications, which utilize VST (Virtual Settlement) functionality to spatially calibrate real-world scenes and use the calibration results for games or other functions.

[0125] S302, MR service determines whether the application is an MR application.

[0126] XR devices run MR applications and MR services. MR services acquire and process the basic data provided by the system services, manage the relevant business logic and data of MR applications, such as data isolation, persistent storage, lifecycle management, and provide business support for the Software Development Kit (SDK) layer of the game engine.

[0127] The game engine is a development tool for MR applications. The game engine can be Unity or Unreal Engine (UE). Developers use the game engine to develop MR applications with semi-automatic calibration functions.

[0128] Specifically, upon detecting a user's launch operation on the current application, the MR service obtains the category configuration information of the current application. This category configuration information indicates whether the current application is an MR application. The MR service determines whether the current application is an MR application based on this category configuration information. If the current application is an MR application, the MR service is started, and step S303 is executed. If the current application is not an MR application, the MR service is not started, and step S304 is executed.

[0129] In one implementation, the category configuration information of the current application is located in the manifest file of the current application, and MRservice reads the category configuration information of the current application from the manifest file of the current application.

[0130] The manifest file, also known as the AndroidManifest.xml file, is required for every Android application and is located in the app / manifests directory.

[0131] The game engine writes the application's category configuration information into the AndroidManifest.xml file. When it detects a user's launch operation on the current application, such as a user's click on the application icon, the current application notifies the MR service via runtime. When the MR service learns that the current application has been launched, it reads the application's category configuration information from the AndroidManifest.xml file.

[0132] In another implementation, the current application is started according to the startup operation. After the current application starts, if the current application is an MR application, the MR application sends the category configuration information of the current application to the MR service, and the MR service receives the category configuration information of the current application.

[0133] In the first method, the MapReduce service can obtain the application's category configuration information much earlier, before the application has finished starting. In the second method, the application's category configuration information can only be obtained after the application has started. The earlier the MapReduce service obtains the application's category configuration information, the sooner it can decide whether to start the service. Since starting the MapReduce service also takes time, starting it earlier allows the MapReduce application to run sooner, reducing user waiting time.

[0134] S303, Create or load the calibration space.

[0135] If the current application is a MapReduce (MR) application, calibration can begin after the MR application and MR service are started. If the space has not been calibrated, a calibration space is created, which includes creating a calibration space identifier (ID). The calibration space ID is the map ID of the calibration space. If the space has already been calibrated, the calibration space is loaded, which includes the space's map ID and the calibration results.

[0136] S304 and MR services will not be processed.

[0137] If the current application is not an MR application, the MR service will not be started; only the current application will be started, so the MR service will not perform any processing.

[0138] S305. Select the calibration scenario and calibration method.

[0139] MR applications may offer multiple calibration scenarios for users to choose from. For example, different family members under the same account can choose different calibration scenarios, and the calibration results for the same space may differ under different calibration scenarios. After selecting a calibration scenario, a calibration method can be selected, including semi-automatic calibration and manual calibration. In the following steps, S306-S310 is the semi-automatic calibration process, and S306'-S312' is the manual calibration process.

[0140] S306, Start semi-automatic calibration.

[0141] S307, Calibration Ground.

[0142] S308, Calibration Ceiling.

[0143] S309. Automatically generate corner lines.

[0144] S310, Stop semi-automatic calibration.

[0145] Steps S306-S310 are the semi-automatic calibration process provided in the embodiments of this application. The ground and ceiling are calibrated manually, while the corner lines are calibrated automatically. The specific implementation method is described in the above embodiments and will not be repeated here.

[0146] S306', Start manual calibration.

[0147] S307', Calibration Ground

[0148] S308', Calibration ceiling.

[0149] S309', Calibration wall surface.

[0150] S310', calibrate the corner of the wall.

[0151] S311' Determine whether the calibration result is accurate.

[0152] If the calibration result is accurate, proceed to step S312'. If the calibration result is inaccurate, return to step S306' and recalibrate.

[0153] S312', Stop calibration.

[0154] S306'-S312' are the manual calibration process. During manual calibration, the ground, ceiling, and walls all need to be calibrated manually by the user. After the wall calibration is completed, the position of the corner is also uniquely determined. After the corner calibration is completed, the user can judge whether the calibration result is accurate based on the position of the calibration line displayed in the video perspective image. If the user thinks the calibration result is inaccurate, the calibration can be restarted. If the user thinks the calibration result is accurate, the calibration can be stopped.

[0155] To facilitate better implementation of the spatial calibration method of this application embodiment, this application embodiment also provides a spatial calibration device. Figure 5 This is a schematic diagram of the spatial calibration device provided in Embodiment 4 of this application, as shown below. Figure 5 As shown, the space calibration device 100 may include:

[0156] The acquisition module 11 is used to acquire an environmental image of the space and determine a video perspective image of the space based on the environmental image of the space.

[0157] The manual calibration module 12 is used to calibrate the ground and ceiling in the space according to the position of the virtual ray emitted by the virtual object in the video perspective image;

[0158] The automatic calibration module 13 determines the position of the corner line in the space based on the environmental image and the calibration results of the ground and ceiling;

[0159] Display module 14 is used to display the calibration lines of the ground and ceiling and the corner lines in the video perspective image.

[0160] In some embodiments, the automatic calibration module 13 is specifically used for:

[0161] A line detection algorithm is used to detect candidate lines that are perpendicular to the ground or the ceiling within the space;

[0162] Based on the calibration results of the ground and ceiling, the corner lines in the space are determined from the candidate straight lines.

[0163] In some embodiments, the device 100 further includes an adjustment module for:

[0164] Replace the target corner line in the space according to the first user operation; and / or

[0165] A new corner line is added within the space based on the second user's operation.

[0166] In some embodiments, the adjustment module is specifically used for:

[0167] Based on the user's calibration operation of the target corner point corresponding to the target corner line, a replacement corner line corresponding to the target corner point is generated and displayed. The starting point of the replacement corner line is the target corner point, which is the corner point formed by the ground and the wall. The replacement corner line is used to replace the target corner line.

[0168] In some embodiments, the adjustment module is further configured to:

[0169] Before generating a replacement corner line corresponding to the target corner line, based on the user's calibration result of the target corner line, a deletion instruction for the target corner line is received; the target corner line is deleted according to the deletion instruction.

[0170] In some embodiments, the adjustment module is specifically used for:

[0171] The replacement corner line is generated based on the user's calibration operation for the replacement corner line corresponding to the target corner line.

[0172] In some embodiments, the adjustment module is further configured to: receive a deletion instruction for the target corner line before generating the replacement corner line based on the user's calibration operation for the replacement corner line corresponding to the target corner line; and delete the target corner line according to the deletion instruction.

[0173] In some embodiments, the adjustment module is specifically used for:

[0174] Based on the user's calibration operation of the first corner point in the space, a new corner line corresponding to the first corner point is generated and displayed. The first corner point is the corner point formed by the ground and the wall.

[0175] In some embodiments, the method operates on an extended reality XR device, which has mixed reality (MR) applications and MR services running on it.

[0176] The MR service is used to obtain the category configuration information of the current application after detecting a user's launch operation on the current application. The category configuration information is used to indicate whether the current application is an MR application.

[0177] When the category configuration information indicates that the current application is an MR application, the MR service is started.

[0178] If the category configuration information indicates that the current application is not an MR application, then it is determined that the MR service will not be started.

[0179] In some embodiments, the MR service is specifically used to: read the category configuration information of the current application from the manifest file of the current application.

[0180] In some embodiments, the MR service is specifically used to: receive category configuration information of the current application sent by the MR application, wherein the category configuration information of the current application is sent after the MR application is started.

[0181] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0182] The apparatus 100 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly manifested as execution by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0183] This application also provides an XR device. Figure 6 This is a schematic diagram of the structure of an XR device provided in Embodiment 5 of this application, as shown below. Figure 6 As shown, the XR device 200 may include:

[0184] The system includes a memory 21 and a processor 22. The memory 21 stores computer programs and transfers the program code to the processor 22. In other words, the processor 22 can retrieve and run the computer programs from the memory 21 to implement the methods described in the embodiments of this application.

[0185] For example, the processor 22 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0186] In some embodiments of this application, the processor 22 may include, but is not limited to:

[0187] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0188] In some embodiments of this application, the memory 21 includes, but is not limited to:

[0189] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0190] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to complete the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in an XR device.

[0191] like Figure 6 As shown, the XR device may further include a transceiver 23, which can be connected to the processor 22 or the memory 21.

[0192] The processor 22 can control the transceiver 23 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 23 may include a transmitter and a receiver. The transceiver 23 may further include antennas, and the number of antennas may be one or more.

[0193] Understandable, although Figure 6 As not shown in the diagram, the XR device 200 may also include a camera module, a Wi-Fi module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be described in detail here.

[0194] It should be understood that the various components in the XR device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0195] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0196] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the XR device reads the computer program from the computer-readable storage medium and executes the computer program, causing the XR device to execute the corresponding flow in the user position control method in the virtual scene of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0197] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0198] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0199] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A spatial calibration method, characterized in that, include: Acquire an environmental image of the space, and determine a video perspective image of the space based on the environmental image of the space; The ground and ceiling in the space are calibrated based on the position of the virtual rays emitted by the virtual objects in the video perspective image; Based on the environmental image and the calibration results of the ground and ceiling, the position of the structural corner line in the space is determined, wherein the structural corner line is a straight line in the space that is perpendicular to the ground or the ceiling and detected by a straight line detection algorithm; The video perspective image displays the calibration lines of the ground and ceiling, as well as the corner lines of the structural walls.

2. The method according to claim 1, characterized in that, Determining the position of the structural corner line in the space based on the environmental image and the calibration results of the ground and ceiling includes: A line detection algorithm is used to detect candidate lines that are perpendicular to the ground or the ceiling within the space; Based on the calibration results of the ground and ceiling, the structural corner lines in the space are determined from the candidate straight lines.

3. The method according to claim 1, characterized in that, After displaying the calibration lines of the ground and ceiling, as well as the structural corner lines, in the video perspective image, the method further includes: Replace the target structural corner line in the space according to the first user operation; and / or A new structural corner line is added within the space according to the second user's operation.

4. The method according to claim 3, characterized in that, The replacement of the target structural corner line in the space according to the first user operation includes: Based on the user's calibration operation of the target corner point corresponding to the target structural corner line, a replacement structural corner line corresponding to the target corner point is generated and displayed. The starting point of the replacement structural corner line is the target corner point, which is the corner point formed by the ground and the wall. The replacement structural corner line is used to replace the target structural corner line.

5. The method according to claim 4, characterized in that, Before generating the replacement structural corner line corresponding to the target structural corner line based on the user's calibration result of the target structural corner line, the process further includes: Receive a deletion command for the corner line of the target structure; The target structural corner line is deleted according to the deletion command.

6. The method according to claim 3, characterized in that, The replacement of the target structural corner line in the space according to the first user operation includes: The replacement structural corner line is generated based on the user's calibration operation of the replacement structural corner line corresponding to the target structural corner line.

7. The method according to claim 6, characterized in that, Before generating the replacement structural corner line based on the user's calibration operation of the replacement structural corner line corresponding to the target structural corner line, the process further includes: Receive a deletion command for the corner line of the target structure; The target structural corner line is deleted according to the deletion command.

8. The method according to claim 3, characterized in that, The step of adding a new structural corner line in the space according to the second user's operation includes: Based on the user's calibration operation of the first corner point in the space, a new structural corner line corresponding to the first corner point is generated and displayed. The first corner point is the corner point formed by the ground and the wall.

9. The method according to claim 1, characterized in that, The method operates on an extended reality (XR) device, which has mixed reality (MR) applications and services running on it. The method further includes: When a user launches the current application, the MR service obtains the category configuration information of the current application, which is used to indicate whether the current application is an MR application. When the category configuration information indicates that the current application is an MR application, the MR service is started. If the category configuration information indicates that the current application is not an MR application, then it is determined that the MR service will not be started.

10. The method according to claim 9, characterized in that, The MR service obtains the category configuration information of the current application, including: The MR service reads the category configuration information of the current application from the manifest file of the current application.

11. The method according to claim 9, characterized in that, The MR service obtains the category configuration information of the current application, including: The current application is launched according to the launch operation; After the current application is started, if the current application is an MR application, the MR application sends the category configuration information of the current application to the MR service; The MR service receives the category configuration information of the current application.

12. A space calibration device, characterized in that, The device includes: The acquisition module is used to acquire environmental images of the space and determine video perspective images of the space based on the environmental images of the space. The manual calibration module is used to calibrate the ground and ceiling in the space based on the position of the virtual rays emitted by the virtual objects in the video perspective image; An automatic calibration module determines the position of the structural corner line in the space based on the environmental image and the calibration results of the ground and ceiling. The structural corner line is a straight line in the space that is perpendicular to the ground or the ceiling and detected by a straight line detection algorithm. The display module is used to display the calibration lines of the ground and ceiling, as well as the corner lines of the structural walls, in the video perspective image.

13. An extended reality device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 11.

Images