Virtual object positioning method and apparatus in mixed reality scenario, and storage medium

Abstract

The present disclosure provides a virtual object positioning method and apparatus in a mixed reality scenario, and a storage medium. The method comprises: determining a second position coordinate of a virtual object in a virtual space model and a third position coordinate of the virtual object in a real space model on the basis of a first position coordinate of the virtual object in a floorplan; collecting first depth information of the virtual object at a position corresponding to the second position coordinate; collecting second depth information of the virtual object at a position corresponding to the third position coordinate; acquiring an error between the first depth information and the second depth information; and then determining a target position coordinate of the virtual object in the real space model on the basis of the first depth information, the third position coordinate and the error. Thus, the virtual-real fusion effect can be improved.

Description

Method, device and storage medium for positioning virtual objects in mixed reality scenes

[0001] This disclosure claims priority to Chinese patent application number CN202311121113.2, filed with the State Intellectual Property Office of China on August 31, 2023, entitled “Method, device and storage medium for positioning virtual objects in mixed reality scenes,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to mixed reality technology, and more particularly to a method, device, and storage medium for positioning virtual objects in a mixed reality scene. Background Art

[0003] Virtual object synthesis technology plays an important role in the field of immersive and realistic mixed reality. It aims to achieve a realistic fusion of virtual and real environments, making it impossible to distinguish virtual objects from real environments. It has been widely used in games, education, medical care, interior design and other fields. In the field of house decoration design, in order to synthesize virtual objects into real scenes / images, related technologies use vector drawing software (Adobe Illustrator, AI) technology to perform automated and stylized design in the floor plan corresponding to the real scene, and directly transform the position of the designed virtual object in the floor plan to the position in the real space model corresponding to the real scene. Since the floor plan and the real space model may be misaligned, directly transforming the virtual object into the real space model may cause local collisions / piercing of the virtual object.

[0004] Summary of the Invention

[0005] Embodiments of the present disclosure provide a method, device, and storage medium for positioning virtual objects in a mixed reality scene.

[0006] According to one aspect of an embodiment of the present disclosure, a method for positioning a virtual object in a mixed reality scene is provided, the method comprising:

[0007] Determining, based on the first position coordinates of the virtual object in the floor plan, second position coordinates of the virtual object in the virtual space model and third position coordinates of the virtual object in the real space model;

[0008] collecting first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;

[0009] collecting second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;

[0010] Obtaining an error between the first depth information and the second depth information;

[0011] The target position coordinates of the virtual object in the real space model are determined based on the first depth information, the third position coordinates and the error.

[0012] In any method embodiment of the present disclosure, determining the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates, and the error includes:

[0013] Iteratively updating the third position coordinate of the virtual object in the real space model by a gradient descent optimization algorithm until the error converges to a minimum value;

[0014] The third position coordinates of the virtual object corresponding to the time when the error converges to the minimum value in the real space model are determined as the target position coordinates.

[0015] In any method embodiment of the present disclosure, iteratively updating the third position coordinate of the virtual object in the real space model by a gradient descent optimization algorithm until the error converges to a minimum value includes:

[0016] Iterate and perform the following operations until the error converges to a minimum:

[0017] updating the third position coordinate of the virtual object in the real space model by a gradient descent optimization algorithm to reduce the error;

[0018] Based on the updated third position coordinates, the operation of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates and the operation of obtaining the error between the first depth information and the second depth information are performed.

[0019] In any method embodiment of the present disclosure, determining the second position coordinates of the virtual object in the virtual space model and the third position coordinates of the virtual object in the real space model based on the first position coordinates of the virtual object in the floor plan includes:

[0020] Multiplying the first position coordinate by a preset first coordinate system transformation matrix to obtain the second position coordinate;

[0021] Multiplying the first position coordinate by a preset second coordinate system transformation matrix to obtain the third position coordinate; or multiplying the second position coordinate by a preset third coordinate system transformation matrix to obtain the third position coordinate;

[0022] The second coordinate system transformation matrix is ​​obtained by multiplying the first coordinate system transformation matrix by the third coordinate system transformation matrix.

[0023] In any method embodiment of the present disclosure, the first position coordinates are the coordinates of the center point of the virtual object in the floor plan;

[0024] The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model;

[0025] The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.

[0026] In any method embodiment of the present disclosure, collecting first depth information of the virtual object in the virtual space model at the position corresponding to the second position coordinate includes:

[0027] A depth camera is simulated at a point corresponding to the second position coordinate to collect a first depth map, where the first depth map is used to represent depth information of the virtual object in the virtual space model.

[0028] In any method embodiment of the present disclosure, collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinate includes:

[0029] A second depth map is collected by simulating a depth camera at a point corresponding to the third position coordinate, where the second depth map is used to represent depth information of the virtual object in the real space model.

[0030] In any method embodiment of the present disclosure, obtaining an error between the first depth information and the second depth information includes:

[0031] The absolute value of the difference between the first depth map and the second depth map is obtained through a preset loss function.

[0032] According to another aspect of an embodiment of the present disclosure, a device for positioning a virtual object in a mixed reality scene is provided, the device comprising:

[0033] a coordinate conversion module, configured to determine, based on the first position coordinates of the virtual object in the floor plan, the second position coordinates of the virtual object in the virtual space model and the third position coordinates of the virtual object in the real space model;

[0034] a first acquisition module, configured to acquire first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;

[0035] a second acquisition module, configured to acquire second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;

[0036] a calculation module, configured to obtain an error between the first depth information and the second depth information;

[0037] A target position determination module is configured to obtain the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates, and the error.

[0038] In any device embodiment of the present disclosure, the target location determination module includes:

[0039] an iterative submodule, configured to iteratively update the third position coordinate of the virtual object in the real space model by using a gradient descent optimization algorithm until the error converges to a minimum value;

[0040] The target position determination submodule is configured to determine the third position coordinates of the virtual object in the real space model corresponding to when the error converges to a minimum value as the target position coordinates.

[0041] In any device embodiment of the present disclosure, the iterative submodule is specifically used to iteratively perform the following operations until the error converges to a minimum value: updating the third position coordinates of the virtual object in the real space model through a gradient descent optimization algorithm to reduce the error; based on the updated third position coordinates, instructing the second acquisition module to collect the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates, and instructing the calculation module to obtain the error between the first depth information and the second depth information.

[0042] In any device embodiment of the present disclosure, the coordinate conversion module includes:

[0043] A first calculation submodule, configured to obtain the second position coordinates by left-multiplying the first position coordinates by a preset first coordinate system transformation matrix;

[0044] A second calculation submodule is configured to obtain the third position coordinates by multiplying the first position coordinates by a preset second coordinate system transformation matrix; or to obtain the third position coordinates by multiplying the second position coordinates by a preset third coordinate system transformation matrix;

[0045] The second coordinate system transformation matrix is ​​obtained by multiplying the first coordinate system transformation matrix by the third coordinate system transformation matrix.

[0046] In any device embodiment of the present disclosure, the first position coordinates are the coordinates of the center point of the virtual object in the floor plan;

[0047] The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model;

[0048] The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.

[0049] In any device embodiment of the present disclosure, the first acquisition module is specifically used to simulate a depth camera to acquire a first depth map at a point corresponding to the second position coordinate, and the first depth map is used to represent the depth information of the virtual object in the virtual space model.

[0050] In any device embodiment of the present disclosure, the second acquisition module is specifically used to simulate a depth camera to acquire a second depth map at the point corresponding to the third position coordinate, and the second depth map is used to represent the depth information of the virtual object in the real space model.

[0051] In any device embodiment of the present disclosure, the calculation module is specifically configured to obtain an absolute value of a difference between the first depth map and the second depth map by using a preset loss function.

[0052] According to another aspect of the present disclosure, an electronic device is provided, the electronic device including:

[0053] Memory for storing computer programs;

[0054] The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method for positioning a virtual object in a mixed reality scene of any embodiment of the present disclosure is implemented.

[0055] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for positioning a virtual object in a mixed reality scene of any embodiment of the present disclosure is implemented.

[0056] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the method for positioning a virtual object in a mixed reality scene of any embodiment of the present disclosure.

[0057] According to the embodiments of the present disclosure, when synthesizing a virtual object into a real scene, the virtual object can be designed in a floor plan. Based on the first position coordinates of the virtual object in the floor plan, its second position coordinates in the virtual space model and its third position coordinates in the real space model are determined. First depth information of the virtual object in the virtual space model is collected at a position corresponding to the second position coordinates, and second depth information of the virtual object in the real space model is collected at a position corresponding to the third position coordinates. Based on the first depth information, the third position coordinates of the virtual object in the real space model, and the error between the first depth information and the second depth information, the target position coordinates of the virtual object in the real space model are determined. Thus, for the real scene, the relative spatial position of the virtual object in the real space model is optimized based on the relative spatial position of the virtual object in the virtual space model (i.e., the first depth information) and the initial relative spatial position in the real space model (the second depth information). This ensures that the relative spatial position of the virtual object in the real space model is consistent with its relative spatial position in the virtual space model, thereby avoiding collision / piercing issues of the virtual object due to misalignment between the floor plan and the real space model, and helping to improve the virtual-reality fusion effect.

[0058] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0060] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0061] FIG1 is a flow chart of an embodiment of a method for locating virtual objects in a mixed reality scene according to the present disclosure;

[0062] FIG2 is a diagram illustrating an example of a mold insertion in an embodiment of the present disclosure;

[0063] FIG3 is an example diagram of the target depth of a virtual object in a virtual space model, the initial depth in a real space model, and the optimized depth in an embodiment of the present disclosure;

[0064] FIG4 is an example diagram of the initial position and optimized position effects of a virtual object in a real space model according to the present disclosure;

[0065] FIG5 is a flow chart of determining the target position coordinates of a virtual object in a real space model according to an embodiment of the present disclosure;

[0066] FIG6 is a schematic structural diagram of an embodiment of a virtual object positioning device in a mixed reality scene according to the present disclosure;

[0067] FIG7 is a structural diagram of another embodiment of a virtual object positioning device in a mixed reality scene disclosed herein;

[0068] FIG8 is a schematic structural diagram of an application embodiment of the electronic device disclosed herein. DETAILED DESCRIPTION

[0069] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0070] It should be understood that for the convenience of description, the sizes of various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0071] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended to limit the present disclosure, its application, or uses.

[0072] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0073] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0074] The disclosed embodiments can be applied to electronic devices such as computer systems / servers, which can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with electronic devices such as computer systems / servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0075] Computer systems / servers and other electronic devices may be described in the general context of computer system-executable instructions (such as program modules) executed by the computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computer system storage media, including storage devices.

[0076] FIG1 is a flow chart of an embodiment of a method for locating virtual objects in a mixed reality scene disclosed herein. The method for locating virtual objects in a mixed reality scene disclosed herein can be applied to electronic devices (such as computer systems and servers). As shown in FIG1 , the embodiment includes the following steps:

[0077] In step 101 , based on the first position coordinates of the virtual object in the floor plan, the second position coordinates of the virtual object in the virtual space model and the third position coordinates of the virtual object in the real space model are determined.

[0078] In the embodiments of the present disclosure, virtual objects can be any virtual items such as virtual furniture, virtual home appliances, virtual accessories, virtual home decoration materials (such as walls, etc.) that can appear in the real space or change the layout of the real space. The embodiments of the present disclosure do not limit this.

[0079] The virtual object in the embodiments of the present disclosure can be one or more virtual objects. When there are multiple virtual objects, the embodiments of the present disclosure can be executed separately for each virtual object. Unless otherwise specified, the embodiments of the present disclosure are described using a single virtual object as an example. The implementation of multiple virtual objects can be similarly implemented.

[0080] Optionally, in some embodiments of the present disclosure, taking virtual furniture as an example, AI technology can be used to design the placement of virtual furniture in the floor plan to obtain the position coordinates of the virtual furniture in the floor plan, which are referred to as the first position coordinates for ease of distinction.

[0081] In the disclosed embodiments, the floor plan, virtual space model, and real space model represent the same real-world scene. The floor plan is a planar spatial layout diagram of a house, representing the real-world scene. This diagram describes the function, location, and size of each independent space in the house.

[0082] In the embodiments of the present disclosure, the virtual space model is a space model generated based on the above-mentioned floor plan. Alternatively, in some embodiments of the present disclosure, the floor plan can be converted into a virtual space model using design instructions and design algorithms in methods such as Computer Aided Design (CAD). The embodiments of the present disclosure do not limit the implementation method of generating a space model based on a floor plan.

[0083] In the embodiments of the present disclosure, the real space model is a three-dimensional space model established for the above-mentioned real scene. Optionally, in some embodiments of the present disclosure, a camera can be used to scan or photograph the real scene, and a computer algorithm can be used to reconstruct the real scene in three dimensions based on the collected image data to obtain a three-dimensional space model. Alternatively, in other embodiments, a point cloud acquisition device, such as a lidar, can be used to collect point cloud data for the real scene, and the three-dimensional real scene can be reconstructed based on the collected point cloud data to obtain a three-dimensional space model. The embodiments of the present disclosure do not limit the implementation method of obtaining the real space model.

[0084] Optionally, in some embodiments of the present disclosure, the second position coordinates of the virtual object in the virtual space model may be determined based on the first position coordinates of the virtual object in the floor plan by the following formula (1): P_vr1 = T1 * P_floorplan Formula (1)

[0085] In formula (1), P_floorplan represents the first position coordinate of the virtual object in the floor plan; T1 is a predetermined first coordinate system transformation matrix, which is used to represent the transformation relationship from the floor plan to the virtual space model (for example, translation, rotation, etc.); P_vr1 represents the second position coordinate of the virtual object in the virtual space model.

[0086] Optionally, in some embodiments of the present disclosure, the third position coordinates of the virtual object in the real space model may be determined based on the first position coordinates of the virtual object in the floor plan by the following formula (2): P_vr2 = T2*P_floorplan Formula (2)

[0087] In formula (2), P_floorplan represents the first position coordinate of the virtual object in the floor plan; T2 is a predetermined second coordinate system transformation matrix, which is used to represent the transformation relationship from the floor plan to the real space model (for example, translation, rotation, etc.); P_vr2 represents the third position coordinate of the virtual object in the real space model.

[0088] Alternatively, in some other embodiments, the third position coordinates of the virtual object in the virtual space model may be obtained based on the second position coordinates of the virtual object in the virtual space model by using the following formula (3): P_vr2 = T3*P_vr1 Formula (3)

[0089] In formula (3), P_vr1 represents the second position coordinate of the virtual object in the virtual space model; T3 is a predetermined third coordinate system transformation matrix, which is used to represent the transformation relationship from the virtual space model to the real space model (for example, translation, rotation, etc.); P_vr2 represents the third position coordinate of the virtual object in the real space model.

[0090] Optionally, in some embodiments of the present disclosure, the second coordinate system transformation matrix T2 may be obtained by multiplying the first coordinate system transformation matrix T1 by the third coordinate system transformation matrix T3.

[0091] Optionally, in some embodiments of the present disclosure, the transformation matrix between the floor plan and the virtual space model can be pre-acquired as the above-mentioned first coordinate system transformation matrix, the transformation matrix between the floor plan and the real space model can be pre-acquired as the above-mentioned second coordinate system transformation matrix, and the transformation matrix between the virtual space model and the real space model can be pre-acquired as the above-mentioned third coordinate system transformation matrix.

[0092] In a specific implementation, since the space occupied by the virtual object is not a point, the first position coordinate, the second position coordinate, and the third position coordinate in this embodiment can be the coordinates of the center point of the box where the virtual object is located in the floor plan, the virtual space model, and the real space model, respectively.

[0093] Optionally, in some embodiments of the present disclosure, in the floor plan, the box where the virtual object is located may be its circumscribed rectangular frame; in the virtual space model and the real space model, the box where the virtual object is located may be its bounding box cuboid.

[0094] In an optional example, step 101 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the coordinate conversion module 31 executed by the processor.

[0095] In step 102, depth information of the virtual object in the virtual space model is collected at a position corresponding to the second position coordinate to obtain first depth information.

[0096] Optionally, in some embodiments of the present disclosure, the first depth information is used to represent the relative position of the virtual object in the virtual space model. The first depth information can be specifically represented as a depth map, which indicates the target space position of the virtual object in the virtual space model, which is also the real space position.

[0097] Optionally, in some embodiments of the present disclosure, a first depth map may be captured by simulating a depth camera at a point corresponding to the second position coordinates. In some implementation examples, the first depth map may be a 360-degree depth panorama, i.e., a 360-degree depth panorama representing the target spatial position of the virtual object in the virtual space model.

[0098] In an optional example, step 102 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first acquisition module 32 executed by the processor.

[0099] In step 103, depth information of the virtual object in the real space model is collected at a position corresponding to the third position coordinate to obtain second depth information.

[0100] Optionally, in some embodiments of the present disclosure, the second depth information is used to represent the relative spatial position of the virtual object in the real space model, and the second depth information can be specifically represented as a depth map.

[0101] Optionally, in some embodiments of the present disclosure, a depth camera may be simulated at a point corresponding to the third position coordinate to capture the second depth map.

[0102] In an optional example, step 103 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the second acquisition module 33 executed by the processor.

[0103] In step 104 , an error between the first depth information and the second depth information is obtained.

[0104] Optionally, in some embodiments of the present disclosure, the absolute value of the difference between the first depth map and the second depth map can be obtained by using a preset loss function as the error between the first depth information and the second depth information. The loss function can be a mean square error (MSE) loss function, a mean absolute error (MAE) loss function, a Huber loss function, etc. The embodiments of the present disclosure do not limit the specific loss function used.

[0105] In an optional example, step 104 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the computing module 34 executed by the processor.

[0106] In step 105 , the target position coordinates of the virtual object in the real space model are determined based on the first depth information, the third position coordinates and the error.

[0107] Optionally, in some embodiments of the present disclosure, the third position of the virtual object in the real space model can be optimized based on the first depth information and the error between the first depth information and the second depth information, so that the relative spatial position of the virtual object in the real space model is consistent with its target spatial position in the virtual space model. When the relative spatial position of the virtual object in the real space model is consistent with its target spatial position in the virtual space model, the third position coordinates of the virtual object in the real space model are the target position coordinates.

[0108] In an optional example, step 105 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the target position determination module 35 executed by the processor.

[0109] According to this embodiment, when synthesizing a virtual object into a real scene, the virtual object can be designed in the floor plan. Based on the first position coordinates of the virtual object in the floor plan, its second position coordinates in the virtual space model and third position coordinates in the real space model are determined. First depth information of the virtual object in the virtual space model is collected at the position corresponding to the second position coordinates, and second depth information of the virtual object in the real space model is collected at the position corresponding to the third position coordinates. Based on the first depth information, the third position coordinates of the virtual object in the real space model, and the error between the first depth information and the second depth information, the target position coordinates of the virtual object in the real space model are determined. In this way, for the real scene, the relative spatial position of the virtual object in the real space model is optimized based on the relative spatial position of the virtual object in the virtual space model (i.e., the first depth information) and the initial relative spatial position in the real space model (the second depth information). This ensures that the relative spatial position of the virtual object in the real space model is consistent with its relative spatial position in the virtual space model, thereby avoiding collision / piercing of the virtual object due to misalignment between the floor plan and the real space model, and helping to improve the virtual-reality fusion effect.

[0110] Due to the misalignment between the floor plan and the real space model, directly converting the first position coordinates of the virtual object in the floor plan into the third position coordinates in the real space model may cause collision / mold penetration problems of the virtual object in the real space model. See Figure 2, the left side of which shows the furniture design in the floor plan, and the right side of which shows the rendering of the furniture position in the real space model determined based on the furniture position in the left side of the figure. It can be seen that part of the furniture indicated by numbers 21 and 22 penetrates the wall.

[0111] See Figure 3, which is an example diagram of the target depth of the virtual object in the virtual space model, the initial depth and the optimized depth in the real space model. Among them, the left figure in Figure 3 is the target depth map, that is, the first depth map collected at the position of the virtual object in the virtual space model (that is, the second position coordinate); the middle figure in Figure 3 is the initial depth map in the real space model, that is, the second depth map collected at the initial position of the virtual object in the real space model (that is, the third position coordinate before optimization); the right figure in Figure 3 is the optimized depth map, that is, the second depth map collected after the virtual object is moved to the target position coordinate (that is, the third position coordinate finally optimized). Through the left depth map and the right depth map in Figure 3, it can be seen that by optimizing the third position coordinate of the virtual object in the real space model, the relative spatial position of the virtual object in the real space model and its target spatial position in the virtual space model achieve the same effect.

[0112] As an application example of an embodiment of the present disclosure, FIG4 shows the initial position (i.e., the third position coordinate before optimization) and optimized position (i.e., the third position coordinate after optimization) of a virtual object in a real space model in an embodiment of the present disclosure. In FIG4 , at the position indicated by number 23 in the left figure, the television, as a virtual object, appears to penetrate the wall, and at the position indicated by number 24, a portion of the air conditioner, as a virtual object, also appears to penetrate the wall. After optimizing the position of the television in the indoor space (as a real space model) and the position of the air conditioner in the indoor space, the right figure in FIG4 is obtained, which shows that the penetration problem of the television and air conditioner has been solved.

[0113] Optionally, in some embodiments of the present disclosure, in step 105, the third position coordinates of the virtual object in the real space model can be iteratively updated through a gradient descent optimization algorithm until the error between the first depth information and the second depth information converges to a minimum value, and then the third position coordinates of the virtual object in the real space model corresponding to when the error converges to the minimum value are determined as the target position coordinates.

[0114] FIG5 is a flow chart of determining the target position coordinates of a virtual object in a real space model according to an embodiment of the present disclosure. As shown in FIG5 , the process includes the following steps:

[0115] In step 201 , the third position coordinates of the virtual object in the real space model are updated by a gradient descent optimization algorithm to reduce the error between the first depth information and the second depth information.

[0116] Optionally, in some embodiments of the present disclosure, before updating the third position coordinates of the virtual object in the real space model, the error between the first depth map of the virtual object in the virtual space model and the second depth map in the real space model can be calculated using a preset loss function. vr -Depth floorplan |(, where Depth vr Depth represents the second depth map rendered at the simulated depth camera position (i.e., the third position coordinate) in the real space model. floorplan The first depth map is rendered at the simulated depth camera position (ie, the second position coordinate) in the virtual space model.

[0117] After calculating the error between the first depth map of the virtual object in the virtual space model and the second depth map in the real space model, the position of the simulated depth camera in the real space model (that is, the third position coordinates of the virtual object in the real space model) can be updated through the gradient descent optimization algorithm to reduce the above error, and then the coordinates of the updated position are used as the third position coordinates of the virtual object in the real space model.

[0118] In step 202, based on the updated third position coordinates, an operation of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates (i.e., step 103) and an operation of obtaining the error between the first depth information and the second depth information (i.e., step 104) are performed.

[0119] In an optional example, steps 201 - 202 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the iterative submodule 351 executed by the processor.

[0120] Based on this embodiment, through the gradient descent optimization algorithm, based on the third position coordinates after each update, the second depth information of the virtual object in the real space model is collected again at the position corresponding to the third position coordinates, and the error between the first depth information and the second depth information is obtained. The above steps 201-202, that is, steps 201, 103-104, are iteratively executed until the error between the first depth information and the second depth information converges to a minimum value. At this time, the third position coordinates obtained by the last update are the accurate position coordinates of the virtual object in the real space model, that is, the target position coordinates.

[0121] In step 203 , the third position coordinates of the virtual object in the real space model corresponding to when the error converges to the minimum value are determined as the target position coordinates.

[0122] In an optional example, step 203 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the target position determination submodule 352 executed by the processor.

[0123] When the error between the first depth information and the second depth information converges to a minimum value, it can be determined that the relative spatial position of the virtual object in the real space model is consistent or basically consistent with the target spatial position in the virtual space model. Since the virtual object will not appear through the model in the virtual space model, after position optimization, the problem of virtual objects appearing through the model in the real space model is also avoided.

[0124] Based on this embodiment, while ensuring that the virtual object remains in the placement position designed by AI in the floor plan, the gradient descent optimization algorithm can be used to automatically reposition the virtual object to the optimal position in the real space model, thereby avoiding the problem of virtual object penetration / collision in the real space model and optimizing the synthesis effect of the virtual object in the real scene.

[0125] The method for locating virtual objects in a mixed reality scene provided by any embodiment of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to a terminal device and a server. Alternatively, the method for locating virtual objects in a mixed reality scene provided by any embodiment of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute the method for locating virtual objects in a mixed reality scene provided by any embodiment of the present disclosure. This will not be further described below.

[0126] In addition, the present disclosure also provides corresponding embodiments of a virtual object positioning device in a mixed reality scene, which can be used to implement the above-mentioned embodiments of the virtual object positioning method in each mixed reality scene of the present disclosure.

[0127] FIG6 is a schematic diagram of the structure of an embodiment of a virtual object positioning device in a mixed reality scene disclosed herein, which can be applied to electronic devices (such as computer systems and servers). As shown in FIG6 , the embodiment includes:

[0128] A coordinate conversion module 31 is configured to determine a second position coordinate of the virtual object in the virtual space model and a third position coordinate of the virtual object in the real space model based on the first position coordinate of the virtual object in the floor plan;

[0129] A first acquisition module 32 is configured to acquire first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinate;

[0130] A second acquisition module 33 is configured to acquire second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate;

[0131] a calculation module 34, configured to obtain an error between the first depth information and the second depth information;

[0132] The target position determination module 35 is configured to determine the target position coordinates of the virtual object in the real space model based on the first depth information, the third position coordinates and the error.

[0133] Figure 7 is a schematic diagram of another embodiment of a virtual object positioning device in a mixed reality scenario of the present disclosure. As shown in Figure 4 , based on the embodiment shown in Figure 6 , in some embodiments, the target position determination module 35 may include: an iteration submodule 351 for iteratively updating the third position coordinates of the virtual object in the real-space model using a gradient descent optimization algorithm until the error converges to a minimum value; and a target position determination submodule 352 for determining the third position coordinates of the virtual object in the real-space model corresponding to the time when the error converges to the minimum value as the target position coordinates.

[0134] Optionally, in some embodiments of the present disclosure, the iterative submodule 351 is specifically used to iteratively perform the following operations until the error converges to a minimum value: updating the third position coordinates of the virtual object in the real space model through a gradient descent optimization algorithm to reduce the error; based on the updated third position coordinates, instructing the second acquisition module 33 to collect the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates, and instructing the calculation module 34 to obtain the error between the first depth information and the second depth information.

[0135] Optionally, in some embodiments of the present disclosure, the coordinate conversion module 31 may include: a first calculation submodule 311, used to left-multiply the first position coordinate with the preset first coordinate system transformation matrix to obtain the second position coordinate; a second calculation submodule 312, used to left-multiply the first position coordinate with the preset second coordinate system transformation matrix to obtain the third position coordinate; or, left-multiply the second position coordinate with the preset third coordinate system transformation matrix to obtain the third position coordinate; wherein the second coordinate system transformation matrix is ​​obtained based on the multiplication of the first coordinate system transformation matrix and the third coordinate system transformation matrix.

[0136] Optionally, in some embodiments of the present disclosure, the first position coordinates are the coordinates of the center point of the virtual object in the floor plan;

[0137] The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model;

[0138] The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.

[0139] Optionally, in some embodiments of the present disclosure, the first acquisition module 32 is specifically configured to simulate a depth camera at a point corresponding to the second position coordinates to acquire a first depth map, where the first depth map is used to represent depth information of the virtual object in the virtual space model.

[0140] Optionally, in some embodiments of the present disclosure, the second acquisition module 33 is specifically configured to simulate a depth camera at a point corresponding to the third position coordinate to acquire a second depth map, where the second depth map is used to represent depth information of the virtual object in the real space model.

[0141] Optionally, in some embodiments of the present disclosure, the calculation module 34 is specifically configured to obtain an absolute value of a difference between the first depth map and the second depth map through a preset loss function.

[0142] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0143] For the above-mentioned device embodiment, since it corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device and method of the embodiment of the present disclosure correspond to each other in specific implementation and beneficial technical effects, and the relevant contents can be referenced to each other and will not be repeated here.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in the same unit or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the disclosed solution. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0145] Below, with reference to FIG8 , an electronic device according to an embodiment of the present disclosure is described, in which a device for implementing a method according to an embodiment of the present disclosure may be integrated. FIG8 is a schematic structural diagram of an application embodiment of an electronic device according to the present disclosure. As shown in FIG8 , the electronic device includes one or more processors 81, one or more computer-readable storage media memory 82, and a computer program stored in the memory and executable on the processor. When executing the program in the memory 82, the aforementioned method for locating virtual objects in a mixed reality scene can be implemented.

[0146] Specifically, in actual applications, the electronic device may further include components such as an input device 83 and an output device 84, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art will appreciate that the structure of the electronic device shown in FIG8 does not limit the electronic device, and may include more or fewer components than shown, or certain components, or different component arrangements.

[0147] The processor 81 can be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 82, and calling data stored in the memory 82, thereby monitoring the electronic device as a whole.

[0148] The memory 82 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 81 may execute the program instructions to implement the virtual object positioning method in the mixed reality scene of each embodiment of the present disclosure above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0149] The input device 83 may be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0150] The output device 84 can output various information to the outside, including the determined distance information, direction information, etc. The output device 84 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0151] The electronic device may also include a power supply for supplying power to various components, which may be logically connected to the processor 81 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply may also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0152] Of course, for simplicity, FIG8 only shows some of the components of the electronic device related to the present disclosure, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may further include any other appropriate components depending on the specific application.

[0153] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for positioning virtual objects in a mixed reality scene according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0154] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0155] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method for positioning virtual objects in a mixed reality scene according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0156] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0157] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0158] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For system embodiments, since they are essentially identical to the method embodiments, their description is relatively simple. For relevant parts, refer to the descriptions of the method embodiments.

[0159] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0160] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0161] The description of the present disclosure is provided for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the disclosed form. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure and design various embodiments with various modifications suitable for specific applications.

Claims

1. A method for positioning a virtual object in a mixed reality scene, characterized in that: The method comprises: Determining, based on the first position coordinates of the virtual object in the floor plan, the second position coordinates of the virtual object in the virtual space model and the third position coordinates of the virtual object in the real space model; Collecting first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinates; Collecting second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate; Obtaining an error between the first depth information and the second depth information; Based on the first depth information, the third position coordinates and the error, the target position coordinates of the virtual object in the real space model are determined.

2. The method according to claim 1, characterized in that The determining, based on the first depth information, the third position coordinates and the error, the target position coordinates of the virtual object in the real space model comprises: Iteratively updating the third position coordinate of the virtual object in the real space model by a gradient descent optimization algorithm until the error converges to a minimum value; The third position coordinates of the virtual object corresponding to the error convergence to the minimum value in the real space model are determined as the target position coordinates.

3. The method according to claim 2, characterized in that The iterative updating of the third position coordinate of the virtual object in the real space model by the gradient descent optimization algorithm until the error converges to a minimum value includes: The following operations are iterated until the error converges to a minimum value: Updating the third position coordinate of the virtual object in the real space model by a gradient descent optimization algorithm to reduce the error; Based on the updated third position coordinates, the operation of collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates and the operation of obtaining the error between the first depth information and the second depth information are performed.

4. The method according to claim 1, characterized in that: The determining, based on the first position coordinates of the virtual object in the floor plan, the second position coordinates of the virtual object in the virtual space model and the third position coordinates of the virtual object in the real space model comprises: Multiply the first position coordinate by the preset first coordinate system transformation matrix to obtain the second position coordinate; The first position coordinate is multiplied by a preset second coordinate system transformation matrix to obtain the third position coordinate; or the second position coordinate is multiplied by a preset third coordinate system transformation matrix to obtain the third position coordinate; The second coordinate system transformation matrix is ​​obtained by multiplying the first coordinate system transformation matrix by the third coordinate system transformation matrix.

5. The method according to claim 1, characterized in that The first position coordinates are the coordinates of the center point of the virtual object in the floor plan; The second position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the virtual space model; The third position coordinates are the coordinates of the center point of the cuboid bounding box of the virtual object in the real space model.

6. The method according to any one of claims 1 to 5, characterized in that: The collecting first depth information of the virtual object in the virtual space model at the position corresponding to the second position coordinates includes: A depth camera is simulated at a point corresponding to the second position coordinates to collect a first depth map, where the first depth map is used to represent depth information of the virtual object in the virtual space model.

7. The method according to claim 6, characterized in that The collecting the second depth information of the virtual object in the real space model at the position corresponding to the third position coordinates includes: A second depth map is collected by simulating a depth camera at a point corresponding to the third position coordinates, where the second depth map is used to represent depth information of the virtual object in the real space model.

8. The method according to claim 7, characterized in that The acquiring an error between the first depth information and the second depth information includes: The absolute value of the difference between the first depth map and the second depth map is obtained through a preset loss function.

9. A virtual object positioning device in a mixed reality scene, characterized in that: The device comprises: A coordinate conversion module, for determining a second position coordinate of the virtual object in the virtual space model and a third position coordinate of the virtual object in the real space model based on a first position coordinate of the virtual object in the floor plan; A first acquisition module, configured to acquire first depth information of the virtual object in the virtual space model at a position corresponding to the second position coordinates; A second acquisition module, configured to acquire second depth information of the virtual object in the real space model at a position corresponding to the third position coordinate; A calculation module, configured to obtain an error between the first depth information and the second depth information; A target position determination module is used to obtain a target position based on the first depth information, the third position coordinates and the error. The target position coordinates of the virtual object in the real space model are obtained.

10. An electronic device, characterized in that: include: Memory, used to store computer products; A processor is used to execute the computer product stored in the memory, and when the computer product is executed, the method described in any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images