Method and device for positioning virtual object in mixed reality scene, and storage medium
By calculating the position coordinates of virtual objects in a mixed reality scene and iteratively optimizing their positions in a real space model, the problems of clipping and collision of virtual objects in a real scene are solved, and the reasonable placement of virtual objects in a real space model is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- REALSEE (BEIJING) TECHNOLOGY CO LTD
- Filing Date
- 2023-08-31
- Publication Date
- 2026-07-31
AI Technical Summary
When existing virtual objects are composited into real-world scenes, there are issues such as localized collisions or clipping problems caused by misalignment between the floor plan and the real-world space model.
By calculating the relative position coordinates of a virtual object in the virtual and real space models based on its position coordinates in the floor plan, and iteratively updating its position in the real space model using a gradient descent optimization algorithm until the error converges to a minimum, the target position coordinates are determined.
It effectively solves the problem of virtual objects clipping through real space models, ensuring that virtual objects are placed reasonably and avoiding collisions in real space models.
Smart Images
Figure CN117078748B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of mixed reality technology, and in particular to a method, apparatus and storage medium for locating virtual objects in a mixed reality scene. Background Technology
[0002] Composite virtual objects into real-world scenes / images to achieve a realistic blend of virtual and real worlds is a pressing issue in the field of immersive mixed reality. Existing technologies for composite virtual objects into real-world scenes / images first utilize vector graphics software (Adobe Illustrator, or AI) to automate and stylize the design on a floor plan corresponding to the real-world scene. Then, the positions of the virtual objects on the floor plan are transformed to their positions in the real-world model. However, due to potential misalignment between the floor plan and the real-world model, directly transforming the virtual objects from the floor plan to the real-world model can lead to issues such as localized collisions / clipping problems. Summary of the Invention
[0003] One technical problem to be solved by the embodiments of this disclosure is to provide a method, apparatus and storage medium for locating virtual objects in a mixed reality scene.
[0004] According to one aspect of the present disclosure, a method for locating virtual objects in a mixed reality scene is provided, the method comprising:
[0005] Based on the first position coordinates of the virtual object in the floor plan, the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model are determined.
[0006] At the position corresponding to the second position coordinates, the first depth information of the virtual object in the virtual space model is collected;
[0007] The second depth information of the virtual object in the real space model is collected at the position corresponding to the third position coordinates;
[0008] Calculate the error between the first depth information and the second depth information;
[0009] Based on the third position coordinates and the error, the target position coordinates of the virtual object in the real space model are calculated.
[0010] In one embodiment of this disclosure, calculating the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates, and the error includes:
[0011] The position of the virtual object in the real space model is iteratively updated using the gradient descent optimization algorithm, and the coordinates of the third position are updated using the updated position coordinates.
[0012] Based on the third position coordinates updated in each iteration, the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates is repeated until the error converges to the minimum value.
[0013] The position coordinates of the virtual object in the real space model corresponding to the convergence of the error to the minimum value are determined as the target position coordinates.
[0014] In another embodiment of this disclosure, determining the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model based on the first position coordinates of the virtual object in the floor plan includes:
[0015] The second position coordinate is obtained by left-multiplying the preset first coordinate system transformation matrix by the value of the first position coordinate;
[0016] The third position coordinate is obtained by multiplying the preset second coordinate system transformation matrix on the left by the first position coordinate; or, the third position coordinate is obtained by multiplying the preset third coordinate system transformation matrix on the left by the second position coordinate.
[0017] The second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
[0018] In another embodiment of this disclosure, the first position coordinates are the coordinates of the center point of the virtual object in the floor plan;
[0019] The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model;
[0020] The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.
[0021] In another embodiment of this disclosure, acquiring the first depth information of the virtual object in the virtual space model at the position corresponding to the second position coordinates includes:
[0022] A first depth map is captured by a simulated depth camera at the point corresponding to the second position coordinates. The first depth map is used to represent the depth information of the virtual object in the virtual space model.
[0023] In another embodiment of this disclosure, the step of acquiring the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates includes:
[0024] A second depth map is acquired by a simulated depth camera at the point corresponding to the third position coordinates. The second depth map is used to represent the depth information of the virtual object in the real space model.
[0025] In another embodiment of this disclosure, calculating the error between the first depth information and the second depth information includes:
[0026] The absolute value of the difference between the first depth map and the second depth map is calculated by setting a loss function.
[0027] According to another aspect of the present disclosure, a virtual object positioning device in a mixed reality scene is provided, the device comprising:
[0028] The coordinate transformation module is used to determine the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model based on the first position coordinates of the virtual object in the floor plan.
[0029] The first acquisition module is used to acquire the first depth information of the virtual object in the virtual space model at the position corresponding to the second position coordinates;
[0030] The second acquisition module is used to acquire the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates;
[0031] The calculation module is used to calculate the error between the first depth information and the second depth information;
[0032] The target position determination module is used to calculate the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates, and the error.
[0033] In one embodiment of this disclosure, the target location determination module includes:
[0034] The iterative submodule is used to iteratively update the position of the virtual object in the real space model using a gradient descent optimization algorithm, and update the third position coordinates using the updated position coordinates; based on the third position coordinates updated in each iteration, the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates is repeated until the error converges to the minimum value;
[0035] The target position determination submodule is used to determine the position coordinates of the virtual object in the real space model when the error converges to the minimum value as the target position coordinates.
[0036] In another embodiment of this disclosure, the coordinate transformation module includes:
[0037] The first calculation submodule is used to calculate the value of the first position coordinate by left multiplying the preset first coordinate system transformation matrix by the first position coordinate to obtain the second position coordinate;
[0038] The second calculation submodule is used to calculate the value of the first position coordinate multiplied by the preset second coordinate system transformation matrix to obtain the third position coordinate; or, to calculate the third position coordinate by multiplying the second position coordinate by the preset third coordinate system transformation matrix.
[0039] The second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
[0040] In another embodiment of this disclosure, the first position coordinates are the coordinates of the center point of the virtual object in the floor plan;
[0041] The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model;
[0042] The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.
[0043] In another embodiment of this disclosure, the first acquisition module is specifically used to simulate a depth camera to acquire a first depth map at the point corresponding to the second position coordinates. The first depth map is used to represent the depth information of the virtual object in the virtual space model.
[0044] In another embodiment of this disclosure, the second acquisition module is specifically used to simulate a depth camera to acquire a second depth map at the point corresponding to the third position coordinates. The second depth map is used to represent the depth information of the virtual object in the real space model.
[0045] In another embodiment of this disclosure, the calculation module is specifically used to calculate the absolute value of the difference between the first depth map and the second depth map by setting a loss function.
[0046] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:
[0047] Memory, used to store computer programs;
[0048] The processor is configured to execute a computer program stored in the memory, and when the computer program is executed, to implement the virtual object localization method in the mixed reality scene described above.
[0049] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method for locating virtual objects in a mixed reality scene.
[0050] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the above-described method for locating virtual objects in a mixed reality scene.
[0051] Based on the virtual object positioning method, apparatus, and storage medium in mixed reality scenarios provided in the above embodiments of this disclosure, when virtual objects need to be composited into a real scene, virtual objects can first be designed in a floor plan. Then, based on the first position coordinates of the virtual object in the floor plan, its second position coordinates in the virtual space model and its third position coordinates in the real space model are determined. Next, the first depth information of the virtual object in the virtual space model is collected at the position corresponding to the second position coordinates, and the second depth information of the virtual object in the real space model is collected at the position corresponding to the third position coordinates. Based on the error between the first and second depth information, and the initial third position coordinates of the virtual object in the real space model, the target position coordinates of the virtual object in the real space model are calculated. This technical solution achieves consistency between the relative spatial position (first depth information) of the virtual object in the virtual space model and its initial relative spatial position (second depth information) in the real space model through iterative updates, effectively solving the problem of virtual object clipping caused by inconsistencies between the floor plan and the real space model.
[0052] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0053] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0054] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:
[0055] Figure 1 A flowchart illustrating one embodiment of the virtual object localization method in a mixed reality scene disclosed herein;
[0056] Figure 2A This is a flowchart illustrating how to determine the target position of a virtual object in a real-world spatial model according to the present disclosure.
[0057] Figure 2B This is a schematic diagram illustrating the clipping process in which the position of a virtual object in a floor plan is directly transferred to a real space model.
[0058] Figure 2C This is a schematic diagram illustrating the target depth of a virtual object in a virtual space model, the initial depth of a real space model, and the optimized depth of a virtual object in this disclosure.
[0059] Figure 2D This is a schematic diagram showing the initial position and optimized position of the virtual object in the real space model of this disclosure;
[0060] Figure 3 This is a schematic diagram of the structure of one embodiment of the virtual object positioning device in a mixed reality scene disclosed herein;
[0061] Figure 4 This is a schematic diagram of another embodiment of the virtual object positioning device in a mixed reality scene disclosed herein;
[0062] Figure 5 This is a structural diagram of an electronic device provided as an illustrative embodiment of the present disclosure. Detailed Implementation
[0063] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0064] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0065] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0066] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0067] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0068] This disclosure can be applied to electronic devices such as computer systems / servers, which can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with electronic devices such as computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0069] Electronic devices such as computer systems / servers can be described in the general context of computer-executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0070] Exemplary embodiments
[0071] Figure 1 This is a flowchart illustrating an embodiment of the virtual object localization method in a mixed reality scene disclosed herein; this virtual object localization method in a mixed reality scene can be applied to electronic devices (such as computer systems, servers), such as... Figure 1 As shown, the virtual object localization method in this mixed reality scene includes the following steps:
[0072] In step 101, based on the first position coordinates of the virtual object in the floor plan, the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model are determined.
[0073] In one embodiment, AI technology can be used to design the placement of virtual furniture on a floor plan, thereby obtaining the first position coordinates P_floorplan of each virtual piece of furniture on the floor plan.
[0074] In one embodiment, the virtual space model is determined based on the floor plan, which can be converted into a virtual space model through design instructions and design algorithms, such as computer-aided design (CAD).
[0075] In one embodiment, the first position coordinates can be converted into second position coordinates using equation (1):
[0076] P_vr1=T1*P_floorplan Equation (1)
[0077] In equation (1), P_floorplan represents the first position coordinate of the virtual object in the floor plan; T1 is the known first coordinate system transformation matrix; and P_vr1 represents the second position coordinate of the virtual object in the virtual space model.
[0078] In one embodiment, when calculating the third position coordinates, the first position coordinates can be converted to the third position coordinates using equation (2):
[0079] P_vr2=T2*P_floorplan Equation (2)
[0080] In equation (2), P_floorplan represents the coordinates of the first position of the virtual object in the floor plan; T2 is the known transformation matrix of the second coordinate system; and P_vr2 represents the coordinates of the third position of the virtual object in the virtual space model.
[0081] For the calculation of the third position coordinates, the second position coordinates can also be converted into the third position coordinates using equation (3):
[0082] P_vr2=T3*P_vr1 Equation (3)
[0083] In equation (3), P_vr1 represents the second position coordinate of the virtual object in the floor plan; T3 is the known third coordinate system transformation matrix; and P_vr2 represents the third position coordinate of the virtual object in the virtual space model.
[0084] The second coordinate system transformation matrix T2 is obtained by multiplying the first coordinate system transformation matrix T1 and the third coordinate system transformation matrix T3.
[0085] To implement the embodiments of this disclosure, a first coordinate system transformation matrix, a second coordinate system transformation matrix, and a third coordinate system transformation matrix can be preset to achieve coordinate system transformation.
[0086] In one embodiment, since the space occupied by the virtual object is not a single point, the first position coordinates, the second position coordinates, and the third position coordinates defined in this embodiment are the coordinates of the center point of the box where the virtual object is located in the floor plan, the virtual space model, and the real space model, respectively.
[0087] In one embodiment, in the floor plan, the box containing the virtual object can be a rectangle; in both the virtual space model and the real space model, the box containing the virtual object is a bounding box cuboid.
[0088] In one embodiment, due to the misalignment between the floor plan and the actual space model, the third position coordinates obtained by transforming the first position coordinates may result in a clipping problem in the actual space model. See [link to relevant documentation]. Figure 2B The left image shows the furniture design in the floor plan, and the right image shows the effect of directly converting the furniture positions in the left image to the furniture positions in the real space model. It can be seen that some of the furniture indicated by labels 21 and 22 are clipped into the wall.
[0089] In step 102, the first depth information of the virtual object in the virtual space model is collected at the position corresponding to the second position coordinates.
[0090] In one embodiment, the first depth information is used to represent the relative position of the virtual object in the virtual space model. Specifically, the first depth information can be represented by a depth map, which indicates the target depth map of the virtual object in the space model.
[0091] In one embodiment, a first depth map can be captured by a simulated depth camera at the point corresponding to the second location coordinates. This depth map can be a 360-degree depth panorama.
[0092] In step 103, the second depth information of the virtual object in the real space model is collected at the position corresponding to the third position coordinate.
[0093] In one embodiment, the second depth information is used to represent the relative position of the virtual object in the real space model, and the second depth information can be represented by a depth map.
[0094] In one embodiment, a second depth map can be acquired by a simulated depth camera at the point corresponding to the third position coordinates.
[0095] In step 104, the error between the first depth information and the second depth information is calculated.
[0096] In one embodiment, the error between the first depth information and the second depth information can be obtained by calculating the absolute value of the difference between the first depth map and the second depth map by setting a loss function. The loss function can be the Mean Square Error (MSE) loss function, the Mean Absolute Error (MAE) loss function, or the Huber loss function.
[0097] In step 105, based on the first depth information, the third position coordinates, and the error, the target position coordinates of the virtual object in the real space model are calculated.
[0098] In one embodiment, by optimizing the position of the virtual object in the real space model, the relative spatial position of the virtual object in the real space model can be made consistent with its relative spatial position in the virtual space model. When the relative positions of the virtual object in the real space model and the virtual space model are consistent, the position of the virtual object is the target position coordinate.
[0099] In one embodiment, see Figure 2C This illustrates the target depth of a virtual object in a virtual space model, the initial depth of the real space model, and the optimized depth. Figure 2C The image on the left is the target depth map (that is, the first depth map captured at the location of the virtual object in the virtual space model). Figure 2C The middle image is the initial depth map in the real space model (that is, the second depth map captured at the initial position (third position coordinate) of the virtual object in the real space model), while... Figure 2C The right-hand image shows the optimized depth map collected after the virtual object was moved to the target location coordinates. Through the depth maps on the left and right, it can be seen that through position optimization, the relative positions of the virtual objects in the real space model and the virtual space model have achieved the same effect.
[0100] In one embodiment, see Figure 2D This illustrates the initial position and optimized position of a virtual object within a real-world model. Figure 2D In the left-hand diagram, the television at location 23 has a wall-penetrating issue, and the air conditioner at location 24 also partially has a wall-penetrating issue. After optimizing the relative positions of the television and air conditioner within the room, the following results were obtained: Figure 2D As shown in the image on the right, the issues of mounting televisions and air conditioners have been resolved.
[0101] In one embodiment, the specific implementation method for calculating the target position coordinates of the virtual object in the real space model can be found in [reference needed]. Figure 2A The embodiments shown are not described in detail here.
[0102] In steps 101-105 above, when it is necessary to composite virtual objects into a real scene, the virtual object can first be designed in the floor plan. Then, based on the first position coordinates of the virtual object in the floor plan, its second position coordinates in the virtual space model and its third position coordinates in the real space model are determined. Next, the first depth information of the virtual object in the virtual space model is collected at the position corresponding to the second position coordinates, and the second depth information of the virtual object in the real space model is collected at the position corresponding to the third position coordinates. Based on the error between the first and second depth information, and the initial third position coordinates of the virtual object in the real space model, the target position coordinates of the virtual object in the real space model are calculated. This disclosed technical solution achieves consistency between the relative spatial position (first depth information) of the virtual object in the virtual space model and its initial relative spatial position (second depth information) in the real space model through iterative updates, effectively solving the problem of virtual object clipping caused by the inconsistency between the floor plan and the real space model.
[0103] To better illustrate the scheme for locating virtual objects in mixed reality scenarios disclosed herein, another embodiment is described below.
[0104] Figure 2A This is a flowchart illustrating how to determine the target position of a virtual object in a real-world spatial model, as disclosed in this disclosure. Figure 2B This is a schematic diagram illustrating the clipping mechanism in this disclosure, which directly transfers the position of a virtual object in a floor plan to a real-world spatial model. Figure 2C This is a schematic diagram illustrating the target depth of a virtual object in a virtual space model, the initial depth of a real space model, and the optimized depth of a virtual object in this disclosure. Figure 2D This document presents schematic diagrams of the initial and optimized positions of virtual objects in a real-world spatial model. This embodiment uses the example of how to locate virtual objects in a mixed reality scene as an example for illustrative purposes. Figure 2A As shown, it includes the following steps:
[0105] In step 201, the position of the virtual object in the real space model is iteratively updated using the gradient descent optimization algorithm, and the coordinates of the third position are updated using the updated position coordinates.
[0106] In one embodiment, before iteratively updating the position of the virtual object in the real space model, it is first necessary to calculate the error |Depth| between the depth map of the virtual object in the virtual space model and the depth map of the virtual object in the real space model by setting a loss function. vr -Depth floorplan |(where, Depth)vr Depth is a depth map rendered in a real-world spatial model at the location of a simulated depth camera. floorplan To simulate the depth map rendered by a depth camera in a virtual spatial model of the floor plan.
[0107] In one embodiment, after calculating the error between the depth map of the virtual object in the virtual space model and the depth map in the real space model, the position of the simulated depth camera in the real space model (i.e., the position of the virtual object in the real space model) can be optimized by using the gradient descent optimization algorithm, and the coordinates of the third position are updated using the coordinates of the position after each iteration.
[0108] In step 202, based on the third position coordinates updated in each iteration, the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates is repeated until the error converges to the minimum value.
[0109] In one embodiment, after updating the third position coordinates using the position coordinates of the position updated in each iteration through the gradient descent optimization algorithm, the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates can be executed again. After calculating the second depth information, the error between the first depth information and the second depth information is calculated until the error converges to the minimum value. Then, the last updated third position coordinates can be determined as the position coordinates of the virtual object in the real space model, which is also the target position coordinates.
[0110] In step 203, the position coordinates of the virtual object in the real space model corresponding to the minimum value of the error are determined as the target position coordinates.
[0111] In one embodiment, when the error converges to the minimum value, it can be determined that the relative position of the virtual object in the real space model is consistent with its relative position in the virtual space model. Since the virtual object will not clip through in the virtual space model, after optimizing the position, the virtual object also avoids the clipping problem in the real space model.
[0112] Through steps 201 to 203 above, the gradient descent optimization algorithm can automatically reposition the virtual object to its optimal position in the real space model, avoiding clipping / collision phenomena, while ensuring that the virtual furniture is still in the reasonable placement position designed by the AI in the floor plan, thus optimizing the synthesis effect of the virtual object in the real scene.
[0113] Corresponding to the aforementioned embodiments of the virtual object positioning method in mixed reality scenarios, this disclosure also provides embodiments of the virtual object positioning device in mixed reality scenarios.
[0114] Figure 3 This is a schematic diagram of the structure of one embodiment of the virtual object positioning device in a mixed reality scene disclosed herein. This device is applied to electronic devices (such as computer systems, servers, etc.). Figure 3 As shown, the device includes:
[0115] The coordinate transformation module 31 is used to determine the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model based on the first position coordinates of the virtual object in the floor plan.
[0116] The first acquisition module 32 is used to acquire the first depth information of the virtual object in the virtual space model at the position corresponding to the second position coordinates;
[0117] The second acquisition module 33 is used to acquire the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates;
[0118] Calculation module 34 is used to calculate the error between the first depth information and the second depth information;
[0119] The target position determination module 35 is used to calculate the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates and the error.
[0120] Figure 4 This is a schematic diagram of another embodiment of the virtual object positioning device in a mixed reality scene disclosed herein, as shown below. Figure 4 As shown, in Figure 3 Based on the illustrated embodiment, in one embodiment, the target location determination module 35 includes:
[0121] The iterative submodule 351 is used to iteratively update the position of the virtual object in the real space model through the gradient descent optimization algorithm, and update the third position coordinates using the updated position coordinates until the error converges to the minimum value.
[0122] The target position determination submodule 352 is used to determine the position coordinates of the virtual object in the real space model when the error converges to the minimum value as the target position coordinates.
[0123] In one embodiment, the coordinate transformation module 31 includes:
[0124] The first calculation submodule 311 is used to calculate the value of the first position coordinates by left multiplying the preset first coordinate system transformation matrix by the first position coordinates to obtain the second position coordinates;
[0125] The second calculation submodule 312 is used to calculate the value of the first position coordinate multiplied by the preset second coordinate system transformation matrix to obtain the third position coordinate; or, to calculate the third position coordinate by multiplying the second position coordinate by the preset third coordinate system transformation matrix.
[0126] The second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
[0127] In one embodiment, the first position coordinates are the coordinates of the center point of the virtual object in the floor plan;
[0128] The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model;
[0129] The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.
[0130] In one embodiment, the first acquisition module 32 is specifically used to simulate a depth camera to acquire a first depth map at the point corresponding to the second position coordinates. The first depth map is used to represent the depth information of the virtual object in the virtual space model.
[0131] In one embodiment, the second acquisition module 33 is specifically used to simulate a depth camera to acquire a second depth map at the point corresponding to the third position coordinates. The second depth map is used to represent the depth information of the virtual object in the real space model.
[0132] In one embodiment, the calculation module 34 is specifically used to calculate the absolute value of the difference between the first depth map and the second depth map by setting a loss function.
[0133] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0134] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0135] Below, for reference Figure 5This describes an electronic device according to embodiments of the present disclosure, wherein apparatus for implementing methods according to embodiments of the present disclosure may be integrated. Figure 5 This is a structural diagram of an electronic device provided in an illustrative embodiment of the present disclosure, such as... Figure 5 As shown, the electronic device includes one or more processors 51, one or more memory 52s of computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 52 is executed, the virtual object localization method in the mixed reality scene described above can be implemented.
[0136] Specifically, in practical applications, the electronic device may also include components such as an input device 53 and an output device 54, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art will understand that... Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or certain components, or different component arrangements. Wherein:
[0137] The processor 51 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities. It performs various functions and processes data by running or executing software programs and / or modules stored in memory 52 and calling data stored in memory 52, thereby providing overall monitoring of the electronic device.
[0138] The memory 52 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 51 may execute the program instructions to implement the virtual object localization method in a mixed reality scene of the various embodiments of this disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.
[0139] The input device 53 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0140] The output device 54 can output various information to the outside, including determined distance information, direction information, etc. The output device 54 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0141] Electronic devices may also include a power supply for powering various components, which can be logically connected to the processor 51 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply may also include one or more DC or AC power sources, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and any other components.
[0142] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0143] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the virtual object localization method in a mixed reality scene according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.
[0144] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0145] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the virtual object localization method in a mixed reality scene according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.
[0146] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0147] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0149] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.
[0150] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0151] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.
Claims
1. A method for locating virtual objects in a mixed reality scene, characterized in that, The method includes: Based on the first position coordinates of the virtual object in the floor plan, the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model are determined. The virtual space model is obtained by transforming the floor plan. At the position corresponding to the second position coordinates, the first depth information of the virtual object in the virtual space model is collected; The second depth information of the virtual object in the real space model is collected at the position corresponding to the third position coordinates; Calculate the error between the first depth information and the second depth information; Based on the first depth information, the third position coordinates, and the error, the target position coordinates of the virtual object in the real space model are calculated.
2. The method of claim 1, wherein, The step of calculating the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates, and the error includes: The position of the virtual object in the real space model is iteratively updated using the gradient descent optimization algorithm, and the coordinates of the third position are updated using the updated position coordinates. Based on the third position coordinates updated in each iteration, the operation step of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates is repeated until the error converges to the minimum value. The position coordinates of the virtual object in the real space model corresponding to the convergence of the error to the minimum value are determined as the target position coordinates.
3. The method of claim 1, wherein, The process of determining the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model based on the first position coordinates of the virtual object in the floor plan includes: The second position coordinate is obtained by left-multiplying the preset first coordinate system transformation matrix by the value of the first position coordinate; The third position coordinate is obtained by multiplying the preset second coordinate system transformation matrix on the left by the first position coordinate; or, the third position coordinate is obtained by multiplying the preset third coordinate system transformation matrix on the left by the second position coordinate. The second coordinate system transformation matrix is obtained by multiplying the first coordinate system transformation matrix and the third coordinate system transformation matrix.
4. The method of claim 1, wherein, The first position coordinates are the coordinates of the center point of the virtual object in the floor plan; The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model; The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.
5. The method of claim 1, wherein, The step of acquiring the first depth information of the virtual object in the virtual space model at the position corresponding to the second position coordinate includes: A first depth map is captured by a simulated depth camera at the point corresponding to the second position coordinates. The first depth map is used to represent the depth information of the virtual object in the virtual space model.
6. The method of claim 5, wherein, The step of acquiring the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate includes: A second depth map is acquired by a simulated depth camera at the point corresponding to the third position coordinates. The second depth map is used to represent the depth information of the virtual object in the real space model.
7. The method of claim 6, wherein, The calculation of the error between the first depth information and the second depth information includes: The absolute value of the difference between the first depth map and the second depth map is calculated by setting a loss function.
8. A device for positioning a virtual object in a mixed reality scene, characterized in that The device includes: The coordinate transformation module is used to determine the second position coordinates of the virtual object in the virtual space model and the third position coordinates in the real space model based on the first position coordinates of the virtual object in the floor plan. The virtual space model is obtained by transformation based on the floor plan. The first acquisition module is used to acquire the first depth information of the virtual object in the virtual space model at the position corresponding to the second position coordinates; The second acquisition module is used to acquire the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates; The calculation module is used to calculate the error between the first depth information and the second depth information; The target position determination module is used to calculate the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates, and the error.
9. An electronic device, comprising: include: Memory, used to store information in computer products; A processor for executing a computer product stored in the memory, wherein, when the computer product is executed, it implements the method described in any one of claims 1-7.
10. A computer readable storage medium having stored thereon computer instructions, wherein, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-7.