Scene reconstruction method and device, augmented reality equipment and readable storage medium

Abstract

The invention discloses a scene reconstruction method and device, augmented reality equipment and a readable storage medium, and relates to the technical field of augmented reality, and the method comprises the steps: obtaining a three-dimensional reconstruction map of a target environment; searching a region which is marked as unsuccessfully reconstructed in the three-dimensional reconstruction image, wherein the unsuccessfully reconstructed region comprises a plurality of target voxels; determining the uncertainty degree of each position point in the image space of the three-dimensional reconstruction image based on the uncertainty numerical values corresponding to the plurality of target voxels; determining a target position in a physical space based on the uncertainty degree of each position point in the image space of the three-dimensional reconstruction image; and if it is detected that the augmented reality device is located at the target position, taking the environment of the target position as a new target environment, obtaining a new three-dimensional reconstruction map of the new target environment, and updating the three-dimensional reconstruction map of the target environment through the new three-dimensional reconstruction map so as to perform at least partial reconstruction on the area marked as unsuccessfully reconstructed. And the three-dimensional reconstruction efficiency is improved.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 63 / 578,714, filed with the U.S. Patent Office on August 25, 2023, and entitled “GUIDED INDOOR SCENE SCANNING METHOD FOR XR DEVICES,” the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of extended reality technology, and more specifically, to a scene reconstruction method, apparatus, extended reality device, and readable storage medium. Background Art

[0004] At present, in the field of extended reality (XR) technology, it is important to reconstruct the scene in three dimensions. Although users can scan the scene and then reconstruct the scene in three dimensions to obtain the corresponding three-dimensional reconstruction image of the scene, the efficiency of the current three-dimensional reconstruction is low. Summary of the invention

[0005] The present application proposes a scene reconstruction method, apparatus, extended reality device and readable storage medium.

[0006] In a first aspect, an embodiment of the present application provides a scene reconstruction method, which is applied to an extended reality device, the method comprising: obtaining a three-dimensional reconstruction map of a target environment; searching the three-dimensional reconstruction map for an area marked as unsuccessfully reconstructed, the unsuccessfully reconstructed area comprising a plurality of target voxels; determining the degree of uncertainty of each position point in the image space of the three-dimensional reconstruction map based on uncertainty values ​​corresponding to the plurality of target voxels, the uncertainty value being used to characterize the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstruction map, the standard information being matched with reference information corresponding to the target voxel in the physical space; determining the target position in the physical space based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstruction map; if it is detected that the extended reality device is at the target position, taking the environment of the target position as a new target environment, obtaining a new three-dimensional reconstruction map of the new target environment, and updating the three-dimensional reconstruction map of the target environment through the new three-dimensional reconstruction map to at least partially reconstruct the area marked as unsuccessfully reconstructed.

[0007] In a second aspect, the embodiment of the present application also provides a scene reconstruction device, which is applied to an extended reality device, and the device includes: a three-dimensional reconstruction map acquisition unit, an unsuccessful reconstruction area acquisition unit, an uncertainty degree determination unit, a target position determination unit, and an update unit. Among them, the three-dimensional reconstruction map acquisition unit is used to obtain a three-dimensional reconstruction map of the target environment; the uncertainty degree determination unit is used to determine the uncertainty degree of each position point in the image space of the three-dimensional reconstruction map based on the uncertainty values ​​corresponding to multiple target voxels, and the uncertainty value is used to characterize the difficulty of the corresponding target voxel in obtaining standard information in the three-dimensional reconstruction map, and the standard information matches the reference information corresponding to the target voxel in the physical space; the target position determination unit is used to determine the target position in the physical space based on the uncertainty degree of each position point in the image space of the three-dimensional reconstruction map; the update unit is used to, if it is detected that the extended reality device is at the target position, take the environment of the target position as a new target environment, obtain a new three-dimensional reconstruction map of the new target environment, and update the three-dimensional reconstruction map of the target environment through the new three-dimensional reconstruction map to at least partially reconstruct the area marked as unsuccessful reconstruction.

[0008] In a third aspect, an embodiment of the present application further provides an extended reality device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method described in the first aspect above.

[0010] The scene reconstruction method, device, extended reality device and readable storage medium provided by the present application first obtain a three-dimensional reconstruction map of the target environment; then search for an area marked as unsuccessfully reconstructed in the three-dimensional reconstruction map, wherein the unsuccessfully reconstructed area includes multiple target voxels; based on the uncertainty values ​​corresponding to the multiple target voxels, determine the degree of uncertainty of each position point in the image space of the three-dimensional reconstruction map; then determine the target position in the physical space based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstruction map; if it is detected that the extended reality device is at the target position, take the environment of the target position as a new target environment, obtain a new three-dimensional reconstruction map of the new target environment, and update the three-dimensional reconstruction map of the target environment through the new three-dimensional reconstruction map, so as to at least partially reconstruct the area marked as unsuccessfully reconstructed. If the user randomly moves or scans in the scene, thereby completing the three-dimensional reconstruction of the scene and obtaining the three-dimensional reconstruction map corresponding to the scene, the efficiency of the three-dimensional reconstruction is low. In the present application, what is obtained is not a three-dimensional reconstruction of the target environment at a random position in the physical space, but the target position in the physical space can be determined by the uncertainty degree of each position point in the image space of the three-dimensional reconstruction. If the extended reality device is subsequently detected to be at the target position, the environment of the target position is taken as a new target environment, a new three-dimensional reconstruction of the new target environment is obtained, and the three-dimensional reconstruction of the target environment is updated by the new three-dimensional reconstruction, so as to reconstruct at least part of the area marked as unsuccessfully reconstructed, wherein the degree of uncertainty is determined based on the uncertainty values ​​corresponding to multiple target voxels, and the uncertainty values ​​are used to characterize the difficulty of the corresponding target voxel to obtain standard information in the three-dimensional reconstruction, and the standard information matches the reference information corresponding to the target voxel in the physical space. In other words, the target position is not randomly selected by the user, but is determined based on the degree of uncertainty, so that the repeated acquisition of the three-dimensional reconstruction of certain positions caused by the random selection of the position can be avoided to a certain extent, and the targeted reconstruction of the unreconstructed area is achieved at least partly, thereby improving the efficiency of the three-dimensional reconstruction.

[0011] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 An application scenario diagram of the scene reconstruction method provided by an embodiment of the present application is shown;

[0014] Figure 2 A method flow chart of a scene reconstruction method provided by an embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram of a three-dimensional reconstruction image provided by an embodiment of the present application is shown;

[0016] Figure 4 A method flow chart of a scene reconstruction method provided by another embodiment of the present application is shown;

[0017] Figure 5 A schematic diagram showing the degree of uncertainty of each position point provided in an embodiment of the present application is shown;

[0018] Figure 6 A schematic diagram showing the degree of uncertainty of each position point provided by another embodiment of the present application is shown;

[0019] Figure 7 A method flow chart of a scene reconstruction method provided by another embodiment of the present application is shown;

[0020] Figure 8 A schematic diagram of an uncertainty diagram provided by an embodiment of the present application is shown;

[0021] Fig. 9 A schematic diagram showing prompt information provided by an embodiment of the present application;

[0022] Fig.10 A method flow chart of a scene reconstruction method provided by yet another embodiment of the present application is shown;

[0023] Fig.11 A structural block diagram of a scene reconstruction device provided in an embodiment of the present application is shown;

[0024] Fig.12 A structural block diagram of an extended reality device provided in an embodiment of the present application is shown;

[0025] Fig.13 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0027] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0028] At present, it is important to perform three-dimensional reconstruction of scenes in the field of extended reality (XR) technology. Although users can scan a scene and then perform three-dimensional reconstruction of the scene to obtain a three-dimensional reconstruction image corresponding to the scene, the efficiency of three-dimensional reconstruction is currently low. How to improve the efficiency of three-dimensional reconstruction is an urgent problem to be solved.

[0029] Among them, extended reality can include augmented reality (AR), virtual reality (VR) or mixed reality (MR). Extended reality can provide users with an immersive experience, allowing users to interact with virtual objects in the real world. Three-dimensional reconstruction of the scene is an important part of extended reality.

[0030] Currently, users can reconstruct a scene in three dimensions by randomly moving and scanning in the scene, thereby obtaining a three-dimensional reconstruction image corresponding to the scene.

[0031] However, the inventors found in their research that the 3D reconstruction of a scene by random movement and random scanning by the user is less efficient because the 3D reconstruction image obtained each time may be less different from the 3D reconstruction image obtained previously, resulting in a lower efficiency of 3D reconstruction. In addition, there may be problems of incomplete or inaccurate 3D reconstruction, resulting in a lower accuracy of the 3D reconstruction image.

[0032] Therefore, in order to overcome or partially overcome the above-mentioned defects, the present application provides a scene reconstruction method, apparatus, extended reality device and readable storage medium.

[0033] See also Figure 1 , Figure 1 An application scenario diagram of the scene reconstruction method provided in an embodiment of the present application is shown, namely, a scene reconstruction scene 100. In the scene reconstruction scene 100, there is a user 110, and the user 110 is at a position 120 in a physical space, and the user 110 wears an extended reality device 130. Thus, the scene reconstruction method can be executed by the extended reality device 130, the target position 140 in the physical space can be determined, and prompt information can be output. If it is subsequently detected that the extended reality device 130 is at the target position 140, a three-dimensional reconstruction map of the environment at the target position can be obtained by the extended reality device 130 to achieve three-dimensional reconstruction. Among them, a detailed introduction to the scene reconstruction method can be found in the subsequent method embodiments.

[0034] It should be noted that, since the user 110 wears the extended reality device 130 , the movement of the user 110 will drive the extended reality device 130 to move, and the position of the user can be approximately regarded as the position of the extended reality device 130 .

[0035] See also Figure 2 , Figure 2 The method flow chart of the scene reconstruction method provided by the embodiment of the present application is shown. The scene reconstruction method can be applied to an extended reality device, for example, the processor of the extended reality device can be used as the execution subject. Specifically, the method includes steps S110 to S150.

[0036] Step S110: Obtain a three-dimensional reconstruction image of the target environment.

[0037] The extended reality device may be a head-mounted device (HMD), and the extended reality device may be configured with an image acquisition module, so that environmental information of the surrounding environment of the extended reality device can be obtained through the image acquisition module. Exemplarily, the image acquisition module may include a camera, so that the image of the surrounding environment can be acquired through the camera. Specifically, for example, the camera may be a red, green, and blue (RGB) camera, and the acquired image of the surrounding environment is a red, green, and blue image. In another exemplary embodiment, the image acquisition module may also include a depth camera or a depth sensor, so that the depth data of the surrounding environment can be acquired through the depth camera or the depth sensor.

[0038] It is understandable that the extended reality device generally includes a display module, through which information can be displayed to achieve interaction with the user. Thus, the user can wear the extended reality device, view the displayed content through the display module of the extended reality device, and collect environmental information of the surrounding environment through the image acquisition module of the extended reality device.

[0039] The target environment may be the surrounding environment of the location of the extended reality device, for example, the surrounding environment may be a room, or may be a local scene corresponding to a range around a certain location in the room.

[0040] It is understandable that, when a user wears the extended reality device, the user's location can be approximately regarded as the location of the extended reality device. Thus, the extended reality device can be used to collect environmental information of the target environment, and then a three-dimensional reconstruction map can be constructed based on the environmental information.

[0041] For some embodiments, the environmental information may include color images and depth data. For example, the red, green, and blue images of the surrounding environment collected by the red, green, and blue cameras are used as color images; the depth data of the surrounding environment is collected by the depth camera or depth sensor. Then, a three-dimensional reconstruction map can be constructed based on the environmental information.

[0042] The three-dimensional reconstruction image can represent the target environment through the image space. For example, see Figure 3 , Figure 3 A schematic diagram of a three-dimensional reconstruction image provided by an embodiment of the present application is shown. Figure 3 A three-dimensional reconstruction image 300 is shown in FIG. 3 , and the three-dimensional reconstruction image 300 is a three-dimensional reconstruction image corresponding to the target environment.

[0043] Among them, the three-dimensional reconstruction map can be constructed by combining the reconstruction model with the environmental information. The reconstruction model can be used to convert the input quantity into a three-dimensional grid representation through the neural scene representation, and then the three-dimensional reconstruction map can be constructed based on the three-dimensional grid representation. For a detailed introduction, please refer to the subsequent embodiments.

[0044] Step S120: searching the three-dimensional reconstructed image for an area marked as unsuccessfully reconstructed, wherein the unsuccessfully reconstructed area includes a plurality of target voxels.

[0045] It is understandable that the three-dimensional reconstruction of the scene may require multiple acquisitions of the three-dimensional reconstruction map of the surrounding environment, where the scene is the scene corresponding to the location of the extended reality device, for example, if the extended reality device is in a room, then the scene is a room; for another example, if the extended reality device is at a road intersection, then the scene is a road intersection. The surrounding environment of the location of the extended reality device can be the scene corresponding to the location, or it can be a local scene in the scene corresponding to the location. For example, if the extended reality device is at a certain position in the room, the target environment can be the room, or it can be a local scene corresponding to the range around a certain position in the room. The three-dimensional reconstruction of the scene is essentially to obtain the three-dimensional reconstruction map corresponding to the environment in the scene multiple times, and update the three-dimensional reconstruction map obtained for the first time by the three-dimensional reconstruction map corresponding to the environment in the scene obtained each time subsequently, so that the three-dimensional reconstruction map is finally obtained by the scene reconstruction method provided by the present application as the result of the three-dimensional reconstruction of the scene.

[0046] Therefore, the three-dimensional reconstruction image of the target environment obtained in the above steps may not be able to fully represent the scene in the image space. For example, the obtained three-dimensional reconstruction image may correspond to an area marked as unsuccessfully reconstructed, so that the area marked as unsuccessfully reconstructed cannot be represented in the image space by the three-dimensional reconstruction image. The area corresponding to the area in the physical space corresponds to the area marked as unsuccessfully reconstructed.

[0047] For some implementations, the unsuccessfully reconstructed areas may be marked in the 3D reconstruction image by leaving the areas blank; or the unsuccessfully reconstructed areas may be marked by dashed boxes, which is not specifically limited in the embodiments of the present application.

[0048] For example, please refer to Figure 3 , Figure 3 In the unsuccessfully reconstructed region 310 in the three-dimensional reconstruction image 300 shown in FIG. Figure 3 In the example shown, the 3D reconstruction image 300 marks the unsuccessfully reconstructed area by leaving the area blank. That is, it can be seen from the unsuccessfully reconstructed area 310 in the 3D reconstruction image 300 that the 3D reconstruction image 300 cannot represent that the area 310 marked as unsuccessfully reconstructed corresponds to an area in the physical space.

[0049] It is understandable that the unsuccessfully reconstructed area is a partial area in the three-dimensional reconstruction image, and the three-dimensional reconstruction image represents the target environment through the image space, so the unsuccessfully reconstructed area is also in the image space. Therefore, the unsuccessfully reconstructed area can be represented in the image space by voxels (Volume Pixel). Among them, voxels can be regarded as the unit of data segmentation in the three-dimensional image space, similar to the unit pixel in the two-dimensional space. In the embodiment provided in the present application, the unsuccessfully reconstructed area may include multiple target voxels.

[0050] In some implementations, when the reconstruction model generates a three-dimensional reconstruction image, the unsuccessfully reconstructed area may be marked.

[0051] Step S130: Based on the uncertainty values ​​corresponding to multiple target voxels, determine the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image, the uncertainty value is used to characterize the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstructed image, and the standard information matches the reference information corresponding to the target voxel in the physical space.

[0052] In order to subsequently determine the target position, the uncertainty degree of each position point in the image space of the three-dimensional reconstruction image may be first determined. For the plurality of target voxels included in the unsuccessfully reconstructed region shown above, each target voxel may correspond to an uncertainty value. Thus, the uncertainty degree of each position point in the image space of the three-dimensional reconstruction image may be determined based on the uncertainty values ​​corresponding to the plurality of target voxels.

[0053] Among them, the uncertainty value is used to characterize the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstructed image, and the standard information matches the reference information corresponding to the target voxel in the physical space. It can be understood that the three-dimensional reconstructed image corresponds to the physical space, so that the target voxel has corresponding standard information in the three-dimensional reconstructed image, wherein the standard information matches the reference information corresponding to the target voxel in the physical space, and the difficulty of obtaining the corresponding standard information in the three-dimensional reconstructed image is not the same for different target voxels. Therefore, the difficulty of obtaining the corresponding standard information can be characterized by the uncertainty value corresponding to the target voxel. Exemplarily, the reference information may include physical reference position information in the physical space, so that the standard information is the standard position information corresponding to the physical reference position information in the three-dimensional reconstructed image space.

[0054] In some embodiments, the target voxels matching each position point in the image space of the three-dimensional reconstructed image can be determined, and then the uncertainty degree of the position point can be determined based on the uncertainty value corresponding to the target voxels matching each position point in the image space. Optionally, the uncertainty value corresponding to each target voxel can also be obtained through the aforementioned reconstruction model, and a detailed description can be referred to in the subsequent embodiments.

[0055] Therefore, the target position in the physical space is subsequently determined based on the uncertainty degree of each position point, so that the target position is determined by taking into account the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstructed image, rather than a randomly selected target position, making the selection of the target position more targeted.

[0056] Step S140: determining the target position in the physical space based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image.

[0057] In the image space of the three-dimensional reconstructed image, multiple position points can be determined, and then the target voxels matched by each position point are further determined, so as to obtain the degree of uncertainty of each position point. Exemplarily, the position point can be randomly selected in the image space of the three-dimensional reconstructed image, or can be obtained by uniform sampling in the image space of the three-dimensional reconstructed image. For detailed description, please refer to the introduction of the subsequent embodiments.

[0058] From the foregoing introduction, it can be seen that the three-dimensional reconstructed image includes areas that have not been successfully reconstructed. Therefore, the target position in the physical space can be further determined based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image, so that a new three-dimensional reconstructed image can be collected at the target position subsequently.

[0059] It can be understood that the position point is a position in the image space, and the target position is a position in the physical space, which can be regarded as a space in reality. Therefore, the designated position point can be first determined based on the uncertainty degree of each position point in the image space of the three-dimensional reconstructed image, and then the position corresponding to the designated position point in the physical space can be obtained through position mapping as the target position. For example, the position point corresponding to the highest degree of uncertainty among the degrees of uncertainty of each position point in the image space of the three-dimensional reconstructed image can be used as the designated position point. For a detailed description, please refer to the subsequent embodiments.

[0060] Optionally, after the target position is determined, prompt information may be output, wherein the prompt information may be used to guide the user to move the extended reality device to the target position.

[0061] Step S150: If it is detected that the extended reality device is at the target position, the environment of the target position is used as a new target environment, a new three-dimensional reconstruction map of the new target environment is obtained, and the three-dimensional reconstruction map of the target environment is updated by the new three-dimensional reconstruction map to at least partially reconstruct the area marked as unsuccessfully reconstructed.

[0062] It can be understood that, when a user wears the extended reality device, the user's movement to the target position can be approximately regarded as the extended reality device also moving to the target position.

[0063] Furthermore, the extended reality device can detect the location where it is located, so that if it is detected that the extended reality device is at the target location, the environment of the target location can be used as a new target environment, and a new three-dimensional reconstruction image of the new target environment can be obtained. The method of obtaining the three-dimensional reconstruction image of the new target environment is similar to the method of obtaining the three-dimensional reconstruction image of the target environment in the above steps, and will not be repeated here.

[0064] The extended reality device may be equipped with a positioning module, so that the position of the extended reality device in the physical space can be determined according to the positioning module. Then, it is determined whether the position is the same as or close to the target position. If the position is the same as or close to the target position, it can be determined that the extended reality device is detected to be at the target position.

[0065] The new 3D reconstruction image is acquired by the extended display device at the target location, so that the 3D reconstruction image of the target environment can be updated by the new 3D reconstruction image to at least partially reconstruct the area marked as unsuccessfully reconstructed. That is, at least partially reconstruct the unsuccessfully reconstructed area in a targeted manner, thereby improving the efficiency of 3D reconstruction.

[0066] Exemplarily, updating the three-dimensional reconstruction map of the target environment by using a new three-dimensional reconstruction map may be performed by replacing the unsuccessfully reconstructed areas in the three-dimensional reconstruction map of the target environment with areas in the new three-dimensional reconstruction map corresponding to the unsuccessfully reconstructed areas in the three-dimensional reconstruction map of the target environment, while retaining areas outside the unsuccessfully reconstructed areas in the three-dimensional reconstruction map of the target environment, thereby obtaining an updated three-dimensional reconstruction map.

[0067] It can be understood that if all the areas marked as unsuccessfully reconstructed are reconstructed in the new 3D reconstruction map, the 3D reconstruction map of the target environment is updated through the new 3D reconstruction map, and all the areas marked as unsuccessfully reconstructed in the 3D reconstruction map can be reconstructed. If some of the areas marked as unsuccessfully reconstructed are reconstructed in the new 3D reconstruction map, the 3D reconstruction map of the target environment is updated through the new 3D reconstruction map, and some of the areas marked as unsuccessfully reconstructed in the 3D reconstruction map can be reconstructed.

[0068] For some implementations, the updated three-dimensional reconstruction image can be regarded as completing the three-dimensional reconstruction of the scene where the extended reality device is located. The updated three-dimensional reconstruction image is the three-dimensional reconstruction image obtained by performing three-dimensional reconstruction of the scene.

[0069] For other implementations, it is possible to detect whether there are unsuccessfully reconstructed areas in the updated 3D reconstruction image. When it is detected that there are no unsuccessfully reconstructed areas in the updated 3D reconstruction image, the updated 3D reconstruction image is used as the 3D reconstruction image obtained by 3D reconstruction of the scene. For a detailed description, please refer to the subsequent embodiments.

[0070] The scene reconstruction method provided by the present application first obtains a three-dimensional reconstruction map of the target environment; then searches for an area marked as unsuccessfully reconstructed in the three-dimensional reconstruction map, wherein the unsuccessfully reconstructed area includes a plurality of target voxels; based on the uncertainty values ​​corresponding to the plurality of target voxels, determines the degree of uncertainty of each position point in the image space of the three-dimensional reconstruction map; then determines the target position in the physical space based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstruction map; if it is detected that the extended reality device is at the target position, the environment at the target position is used as a new target environment, a new three-dimensional reconstruction map of the new target environment is obtained, and the three-dimensional reconstruction map of the target environment is updated by the new three-dimensional reconstruction map, so as to at least partially reconstruct the area marked as unsuccessfully reconstructed. If the user randomly moves or scans in the scene, thereby completing the three-dimensional reconstruction of the scene and obtaining the three-dimensional reconstruction map corresponding to the scene, the efficiency of the three-dimensional reconstruction is low. In the present application, what is obtained is not a three-dimensional reconstruction of the target environment at a random position in the physical space, but the target position in the physical space can be determined by the uncertainty degree of each position point in the image space of the three-dimensional reconstruction. If the extended reality device is subsequently detected to be at the target position, the environment of the target position is taken as a new target environment, a new three-dimensional reconstruction of the new target environment is obtained, and the three-dimensional reconstruction of the target environment is updated by the new three-dimensional reconstruction, so as to reconstruct at least part of the area marked as unsuccessfully reconstructed, wherein the degree of uncertainty is determined based on the uncertainty values ​​corresponding to multiple target voxels, and the uncertainty values ​​are used to characterize the difficulty of the corresponding target voxel to obtain standard information in the three-dimensional reconstruction, and the standard information matches the reference information corresponding to the target voxel in the physical space. In other words, the target position is not randomly selected by the user, but is determined based on the degree of uncertainty, so that the repeated acquisition of the three-dimensional reconstruction of certain positions caused by the random selection of the position can be avoided to a certain extent, and the targeted reconstruction of the unreconstructed area is achieved at least partly, thereby improving the efficiency of the three-dimensional reconstruction.

[0071] See also Figure 4 , Figure 4 The method flow chart of the scene reconstruction method provided by the embodiment of the present application is shown. The scene reconstruction method can be applied to an extended reality device, for example, the processor of the extended reality device can be used as the execution subject. Specifically, the method includes steps S210 to S280.

[0072] Step S210: Obtain a three-dimensional reconstruction image of the target environment.

[0073] Step S220: searching the three-dimensional reconstructed image for an area marked as unsuccessfully reconstructed, wherein the unsuccessfully reconstructed area includes a plurality of target voxels.

[0074] In some embodiments, when the reconstruction model generates a three-dimensional reconstruction image, the area that has not been successfully reconstructed can be marked. Exemplarily, the reconstruction model can obtain the uncertainty value of each position point in the image space corresponding to the three-dimensional reconstruction image, and if there is at least one position point whose uncertainty value is greater than a specified threshold, the area matching the position point corresponding to the uncertainty value greater than the specified threshold can be marked as an unreconstructed area.

[0075] Among them, step S210 and step S220 have been introduced in detail in the above embodiments and will not be repeated here.

[0076] Step S230: determining at least one candidate plane and at least one candidate reference point in each candidate plane in the image space of the three-dimensional reconstructed image, and each candidate reference point serves as a position point in the image space of the three-dimensional reconstructed image.

[0077] From the foregoing introduction, it can be known that multiple position points can be determined in the image space of the three-dimensional reconstructed image. In the embodiment provided in the present application, at least one alternative plane and at least one alternative reference point in each of the alternative planes can be determined in the image space of the three-dimensional reconstructed image, so that each alternative reference point is used as a position point in the image space of the three-dimensional reconstructed image.

[0078] For some implementations, the image space of the three-dimensional reconstructed image may be segmented to obtain a plurality of two-dimensional planes, and then the position points may be determined in the obtained two-dimensional planes.

[0079] Specifically, step S230 may include step S231 and step S232.

[0080] Step S231 divides the three-dimensional reconstructed image in the image space to obtain a plurality of mutually parallel two-dimensional planes as candidate planes.

[0081] Step S232: uniformly sampling each candidate plane to obtain a plurality of sampling points as candidate reference points in the candidate plane, wherein the sampling points include position information of the sampling points in the image space of the three-dimensional reconstructed image.

[0082] The three-dimensional reconstruction image can be segmented in the image space to obtain multiple mutually parallel two-dimensional planes as candidate planes. For example, see 5, Figure 5 A schematic diagram of determining the degree of uncertainty of each position point provided in an embodiment of the present application is shown. Figure 53D reconstruction image space 500 is shown in FIG. The image space 500 is divided to obtain a two-dimensional plane 501 and a two-dimensional plane 502, wherein the two-dimensional plane 501 and the two-dimensional plane 502 are parallel to each other. Then the two-dimensional plane 501 and the two-dimensional plane 502 are candidate planes.

[0083] Furthermore, each candidate plane may be sampled to obtain multiple sampling points. For example, the candidate planes may be uniformly sampled. Figure 5 ,right Figure 5 Uniform sampling of the two-dimensional plane 501 shown in the figure can obtain sampling points 5011 and sampling points 5012; and uniform sampling of the two-dimensional plane 502 can obtain sampling points 5021 and sampling points 5022. Each sampling point includes the position information of the sampling point in the image space of the three-dimensional reconstructed image. Each sampling point obtained by sampling can be used as an alternative reference point of the two-dimensional plane corresponding to the sampling point. For example, Figure 5 The sampling points 5011 and 5012 shown in FIG. 5 are candidate reference points of the two-dimensional plane 501 ; the sampling points 5021 and 5022 are candidate reference points of the two-dimensional plane 502 .

[0084] Each candidate reference point obtained above can be used as a position point in the image space of the 3D reconstruction image. Figure 5 , that is Figure 5 The sampling points 5011, 5012, 5021 and 5022 shown in the figure are all used as position points in the image space of the three-dimensional reconstructed image, that is, the sampling point 5011 is used as the position point 5011, the sampling point 5012 is used as the position point 5012, the sampling point 5021 is used as the position point 5021, and the sampling point 5022 is used as the position point 5022.

[0085] In some embodiments, the two-dimensional plane obtained above can be called a viewing potential field (VPF).

[0086] Step S240: Determine a designated spatial range corresponding to each of the position points, wherein a distance between the position point and any point within the designated spatial range is less than or equal to a detection distance of the extended reality device.

[0087] In order to determine the target voxel that matches the position point, the specified spatial range corresponding to each position point can be first determined. From the above introduction, it can be known that the extended reality device can collect environmental information of the surrounding environment through the image acquisition module, and the extended reality device has a corresponding detection distance for collecting environmental information of the surrounding environment through the image acquisition module, for example, the detection distance can be 1.5 meters, 3 meters, etc. Taking the detection distance of 3 meters as an example, the image acquisition module can be used to collect environmental information within the surrounding 3 meters.

[0088] Therefore, the designated spatial range may be determined based on the detection distance of the extended reality device, so that the distance between the location point and any point within the designated spatial range is less than or equal to the detection distance of the extended reality device.

[0089] For example, each position point can be used as the center of a circle, the detection distance can be used as the radius, and the spherical space formed by the center of the circle and the radius can be used as the designated spatial range corresponding to the position point. It can be seen that in the designated spatial range, the distance between any point and the position point corresponding to the designated spatial range is less than or equal to the radius of the spherical space, and is also less than or equal to the detection distance.

[0090] For example, please see Figure 5 , Figure 5 The sampling point 5011 is used as the position point, and the specified spatial range of the position point is 591. It should be noted that Figure 5 The other sampling points in are used as location points and also correspond to the specified spatial range. Figure 5 The specified spatial range corresponding to other sampling points is not drawn.

[0091] As another exemplary embodiment, each position point may be taken as the center of a circle, and a length slightly smaller than the detection distance may be taken as the radius, and a spherical space formed by the center of the circle and the radius may be taken as the designated spatial range corresponding to the position point.

[0092] In another exemplary embodiment, each position point may be used as a center point, the detection distance may be used as a diagonal line of a rectangle, and the rectangular space formed by the center point and the diagonal line may be used as a designated spatial range corresponding to the position point.

[0093] Step S250: searching for a target voxel whose uncertain value is greater than a specified value within a specified spatial range corresponding to each of the position points as a target voxel matching the position point.

[0094] Furthermore, after obtaining the designated spatial range corresponding to each position point, the target voxel that meets the designated condition can be searched in the corresponding designated spatial range based on the uncertainty value of the target voxel.

[0095] For some implementations, a target voxel with an uncertain value greater than a specified value may be searched within a specified spatial range corresponding to each position point as the target voxel matching the position point. Figure 5 ,exist Figure 5 In the designated spatial range 591 shown in FIG, there are target voxel 5911, target voxel 5912, and target voxel 5913. The uncertainty value corresponding to target voxel 5911 is 1; the uncertainty value corresponding to target voxel 5912 is 2; and the uncertainty value corresponding to target voxel 5913 is 3. If the designated value is 1, then it can be determined that target voxel 5912 and target voxel 5913 are target voxels that match sampling point 5011, that is, target voxels that match position point 5011.

[0096] It is understandable that if the extended reality device is at a position point later, a three-dimensional reconstruction map corresponding to the surrounding environment of the position point can be constructed, thereby realizing the collection of voxels matching the position, so as to reduce the scope of the unsuccessfully reconstructed area in the three-dimensional reconstruction map to a certain extent, or make some target voxels in the unsuccessfully reconstructed area more correctly described. However, there may be obstacles in the target environment, so the three-dimensional reconstruction map can also include obstacle information, so that the target voxels matching each position point can be further determined in combination with the obstacle information, thereby improving the accuracy of determining the matching target voxels, thereby improving the accuracy of the target position determined subsequently, and further improving the efficiency of three-dimensional reconstruction.

[0097] In some implementations, step S250 may further include step S251 and step S252.

[0098] Step S251: searching for a target voxel whose uncertain value is greater than a specified value within a specified spatial range corresponding to each position point as a voxel to be determined for matching the position point.

[0099] Step S252: Based on the obstacle position information and obstacle volume information, determine the voxels to be determined that are not blocked by the obstacle in the line of sight between the position point and each of the voxels to be determined that match the position point, as the target voxels for matching the position point.

[0100] First, a target voxel with an uncertain value greater than a specified value can be searched within the specified spatial range corresponding to each position point as the voxel to be determined for matching the position point. Detailed description can be found in the above description, which will not be repeated here.

[0101] The obstacle information in the three-dimensional reconstructed image may include obstacle position information and obstacle volume information of the obstacle. That is, the spatial position of the obstacle in the image space of the three-dimensional reconstructed image can be determined through the obstacle information. Further, it can be determined whether the line of sight between each position point and the to-be-determined voxel matching the position point is blocked by the obstacle.

[0102] In some embodiments, each position point and the voxel to be determined that matches the position point can be connected by a straight line in the image space, that is, the line of sight is represented by the straight line. If the straight line passes through an obstacle, it represents that the position point corresponding to the straight line and the voxel to be determined are blocked by the obstacle in the line of sight; if the straight line does not pass through an obstacle, it represents that the position point corresponding to the straight line and the voxel to be determined are not blocked by the obstacle in the line of sight. The voxel to be determined that is not blocked by the obstacle in the line of sight can be used as the target voxel matched by the position point.

[0103] Whether the straight line passes through an obstacle can be determined comprehensively based on the obstacle volume information and the obstacle position information in the obstacle information.

[0104] Please continue reading Figure 5 ,by Figure 5 The target voxel 5912 and the target voxel 5913 shown in the figure are used as the voxels to be determined to match the position point 5011, that is, the target voxel 5912 is used as the voxel 5912 to be matched, and the target voxel 5913 is used as the voxel 5913 to be matched. Figure 5 In the image space 500 of the three-dimensional reconstruction shown, there is an obstacle 580, which is located on the straight line connecting the position point 5011 and the voxel to be determined 5912, so it can be determined that the line of sight between the position point 5011 and the voxel to be determined 5912 is blocked by the obstacle, while the line of sight between the position point 5011 and the voxel to be determined 5913 is not blocked by the obstacle. Therefore, it can be determined that the target voxel 5913 is the target voxel matching the position point 5011.

[0105] Step S260: Determine the degree of uncertainty of each position point based on the uncertainty value corresponding to the target voxel matched by each position point in the image space.

[0106] Furthermore, the uncertainty degree of each position point can be determined based on the uncertainty value corresponding to the target voxel matched by each position point in the image space. For some embodiments, the sum of the uncertainty values ​​corresponding to the target voxel matched by each position point in the image space can be used as the uncertainty degree of the position point.

[0107] Please continue reading Figure 5 ,by Figure 5Continuing with the example of the position point 5011 shown, the target voxels matched by the position point 5011 include the target voxel 5913. Therefore, if the uncertainty value corresponding to the target voxel 5913 is 3, it can be determined that the uncertainty degree of the position point 5011 is 3.

[0108] To more clearly explain how to determine the degree of uncertainty at each location, continue with the following Figure 6 Take the following as an example. Specifically, Figure 6 A schematic diagram of determining the degree of uncertainty of each position point provided in an embodiment of the present application is shown. Figure 6 A cross-sectional view 600 of the image space of the three-dimensional reconstruction is shown in FIG. Figure 5 , but is another example). By dividing the cross-sectional view 600 of the image space, a two-dimensional plane 601 can be obtained, wherein the two-dimensional plane 601 is perpendicular to the cross section, so that Figure 6 The two-dimensional plane 601 is shown in the form of lines. The two-dimensional plane 601 is the viewing potential field. Figure 6 The two-dimensional plane 601 shown in the figure is uniformly sampled to obtain sampling points 6011 and sampling points 6012. The sampling points 601 and 6012 can be used as position points in the image space of the three-dimensional reconstructed image, that is, the sampling point 6011 is used as the position point 6011, and the sampling point 6012 is used as the position point 6012.

[0109] For further information, please refer to Figure 6 , the specified spatial range 691 corresponding to the position point 6011, and the specified spatial range 692 corresponding to the position point 6012.

[0110] Furthermore, within the specified spatial range corresponding to each position point, a target voxel with an uncertain value greater than a specified value is searched as a voxel to be determined for matching the position point. Figure 6 In the specified spatial range 691, there are to-be-determined voxels 6911, to-be-determined voxels 6912, to-be-determined voxels 6913 and to-be-determined voxels 6914 that match the position point 6011; in the specified spatial range 692, there are to-be-determined voxels 6921, to-be-determined voxels 6922, to-be-determined voxels 6923 and to-be-determined voxels 6924 that match the position point 6022.

[0111] In the cross-sectional view 600 of the image space, there is an obstacle 680. Through the method of determining the target voxel from the voxels to be determined as described above, it can be determined that there are target voxels 6911, target voxel 6912 and target voxel 6913 matching the position point 6012 within the specified spatial range 691; and there are target voxels 6921, target voxel 6922 and target voxel 6923 matching the position point 6022 within the specified spatial range 692.

[0112] Please continue reading Figure 6 , Figure 6 Each target voxel in can also be numerically identified with an uncertain value corresponding to the voxel to be determined. Specifically, Figure 6 The uncertainty value corresponding to the target voxel 6911 is 1, the uncertainty value corresponding to the target voxel 6912 is 1, and the uncertainty value corresponding to the target voxel 6913 is 2; the uncertainty value corresponding to the target voxel 6921 is 2, the uncertainty value corresponding to the target voxel 6922 is 2, and the uncertainty value corresponding to the target voxel 6923 is 3.

[0113] Take the sum of the uncertainty values ​​corresponding to the target voxels that match each position point in the image space as an example to illustrate the uncertainty degree of the position point. Figure 6 , the degree of uncertainty corresponding to position point 6011 is 1+1+2=4; the degree of uncertainty corresponding to position point 6012 is 2+2+3=7.

[0114] Step S270: Determine the target position in the physical space based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image.

[0115] Step S280: If it is detected that the extended reality device is at the target position, the environment of the target position is used as a new target environment, a new three-dimensional reconstruction map of the new target environment is obtained, and the three-dimensional reconstruction map of the target environment is updated by the new three-dimensional reconstruction map to at least partially reconstruct the area marked as unsuccessfully reconstructed.

[0116] Among them, step S270 and step S280 have been introduced in detail in the above embodiments and will not be repeated here.

[0117] In the scene reconstruction method provided by the present application, the target voxels matching each position point can be further determined in combination with the obstacle information to improve the accuracy of determining the matching target voxels. In addition, the sum of the uncertainty values ​​corresponding to the target voxels matching each position point in the image space is used as the uncertainty degree of the position point, and then the target position is subsequently determined based on the uncertainty degree of each position point, further improving the efficiency of three-dimensional reconstruction.

[0118] See also Figure 7 , Figure 7 The method flow chart of the scene reconstruction method provided by the embodiment of the present application is shown. The scene reconstruction method can be applied to an extended reality device, for example, the processor of the extended reality device can be used as the execution subject. Specifically, the method includes steps S310 to S3110.

[0119] Step S310: Collecting environmental information of the target environment through the extended reality device.

[0120] Step S320: input the environmental information as an input to the reconstruction model to obtain first three-dimensional grid information output by the reconstruction model, wherein the first three-dimensional grid information includes each grid obtained by dividing the target environment and the position information of each grid.

[0121] Step S330: Determine a three-dimensional reconstruction image of the target environment based on the grids obtained by dividing the target environment and the position information of each grid.

[0122] As can be seen from the above introduction, the extended reality device can be configured with some modules, so that the environmental information of the target environment can be collected through the configured modules. Among them, the target environment can be the surrounding environment of the location of the extended reality device. For example, the surrounding environment can be a room, or it can be a local scene corresponding to the range around a certain location in the room.

[0123] Exemplarily, the extended reality device may be configured with an image acquisition module. Thus, the environmental information may include information acquired by the image acquisition module configured based on the extended reality device. Specifically, for example, the image acquisition module includes a red, green, and blue camera and a depth sensor, so that the red, green, and blue images of the surrounding environment may be acquired by the red, green, and blue camera; and the depth data of the surrounding environment may be acquired by the depth sensor. That is, for an extended reality device configured with an image acquisition module including a red, green, and blue camera and a depth sensor, the acquired environmental information of the target environment may include red, green, and blue images and depth data.

[0124] Furthermore, the collected environmental information can be input as an input into the reconstruction model to obtain the first three-dimensional grid information output by the reconstruction model. The first three-dimensional grid information includes each grid obtained by dividing the target environment and the position information of each grid. It can be understood that the reconstruction model can be used to convert the input into a three-dimensional grid representation through a neural scene representation. In three-dimensional reconstruction, the three-dimensional grid representation method helps to obtain a three-dimensional reconstruction image.

[0125] It should be noted that in the first three-dimensional grid information obtained by reconstructing the model, each grid obtained by dividing the target environment actually refers to the processing of the environmental information collected for the target environment by reconstructing the model, such as the grid division processing in the image space, and does not refer to the division of the target environment in the physical space. The position information of each grid in the first three-dimensional grid information is actually used to characterize the relative position relationship of each grid relative to the extended reality device. For example, the relative position relationship can be characterized by the world coordinate system, or by the device coordinate system, where the device coordinate system is a coordinate system established based on the extended reality device.

[0126] For some embodiments, the first three-dimensional grid information output by the reconstruction model can be regarded as a specific data format representing the three-dimensional reconstruction image, so that after obtaining the first three-dimensional grid information, the three-dimensional reconstruction image of the target environment can be determined based on the grids obtained by dividing the target environment and the position information of each grid. The schematic diagram of the three-dimensional reconstruction image can be referred to in Figure 3 .

[0127] It is understandable that in order to reconstruct a scene through an extended reality device, it may be necessary to place the extended reality device in multiple different positions in the scene, and obtain the three-dimensional reconstruction map corresponding to the surrounding environment multiple times, so as to obtain the three-dimensional reconstruction map corresponding to the scene. Therefore, after obtaining the environmental information, the historical environmental information can also be read, and then the environmental information and the historical environmental information are input into the reconstruction model together as input quantities. Among them, the historical environmental information is the information of the historical environment collected by the extended reality device, and the historical environment is the environment in which the extended reality device is located before the three-dimensional reconstruction map of the target environment is obtained this time. Thereby, the accuracy of the three-dimensional reconstruction map can be improved to a certain extent.

[0128] Step S340: searching the three-dimensional reconstructed image for an area marked as unsuccessfully reconstructed, wherein the unsuccessfully reconstructed area includes a plurality of target voxels.

[0129] Among them, step S340 has been described in detail in the above embodiment and will not be repeated here.

[0130] Step S350: Based on the grids obtained by dividing the unsuccessfully reconstructed area and the position information of each grid, determine the target voxel in the unsuccessfully reconstructed area that has a mapping relationship with each grid.

[0131] Step S360: taking the uncertainty value corresponding to each grid as the uncertainty value corresponding to the target voxel having a mapping relationship with the grid.

[0132] It can be seen from the introduction of the above embodiments that in order to determine the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image, it is necessary to first obtain the uncertainty value corresponding to the target voxel in the unsuccessfully reconstructed area.

[0133] For some embodiments, the reconstruction model can also be used to output second three-dimensional grid information. That is, the environmental information is input as an input to the reconstruction model, and the reconstruction model can also output second three-dimensional grid information. The second three-dimensional grid information includes each grid obtained by dividing the unsuccessfully reconstructed area, the position information of each grid, and the uncertainty value corresponding to each grid.

[0134] It should be noted that, similar to the first three-dimensional grid information, each grid obtained by dividing the unsuccessfully reconstructed area actually refers to the processing of the environmental information collected for the target environment through the reconstruction model, such as the grid division processing in the image space for the area that has not been successfully reconstructed in the target environment, rather than the division of the area that has not been successfully reconstructed in the target environment in the physical space. The position information of each grid in the second three-dimensional grid information is actually used to characterize the relative position relationship of each grid relative to the extended reality device. For example, the relative position relationship can be characterized by the world coordinate system, or by the device coordinate system, wherein the device coordinate system is a coordinate system established based on the extended reality device. In addition, the second three-dimensional grid information may also include uncertainty values ​​corresponding to each grid obtained by dividing the unsuccessfully reconstructed area. The uncertainty value corresponding to each grid is positively correlated with the difficulty of correctly describing the corresponding grid in the three-dimensional reconstruction image.

[0135] Among them, the reconstruction model can be trained to identify prominent features of the environment in the scene, which may include red, green and blue information of the image corresponding to the environment in the scene, and depth information corresponding to the environment in the scene, and generate first three-dimensional grid information and second three-dimensional grid information based on the prominent features.

[0136] Furthermore, since the unsuccessfully reconstructed area includes multiple target voxels, it is necessary to first determine the target voxels by dividing the unsuccessfully reconstructed area into various grids. Specifically, based on the grids obtained by dividing the unsuccessfully reconstructed area and the position information of each grid, the target voxels in the unsuccessfully reconstructed area that have a mapping relationship with each grid can be determined, and the target voxels can include a position relationship, specifically, the relative position relationship between the target voxel and the extended reality device.

[0137] For some embodiments, whether there is a mapping relationship between the grid and the target voxel can be determined by the position information of the grid and the position information of the target voxel. Exemplarily, it can be determined that the grid and the target voxel with the same or similar position information have a mapping relationship. For example, the position information of grid A is A1, the position information of grid B is B1, the position information of target voxel C is A1, and the position information of target voxel D is D1, where D1 is close to B1. Then it can be determined that grid A has a mapping relationship with target voxel C, and it can be determined that grid B has a mapping relationship with target voxel D.

[0138] It should be noted that, since the grid can correspond to the spatial range in the image space, the position information corresponding to the grid can include the spatial range; and the voxel can be approximately regarded as a dot in the image space, so the position information corresponding to the voxel is relatively small, for example, it can be approximately a coordinate point in the image space. Therefore, the same position information can be that the position information corresponding to the voxel is within the spatial range corresponding to the position information of the grid; and the similar position information can be that the position information corresponding to the voxel is at a position closer to the spatial range corresponding to the position information of the grid, for example, within the extended spatial range and outside the spatial range corresponding to the position information of the grid, wherein the extended spatial range can be a spatial range that includes the spatial range corresponding to the position information of the grid and is a specified multiple of the spatial range corresponding to the position information of the grid, and the specified multiple can be 1.1, 1.2 or 1.5, etc.

[0139] Furthermore, the uncertainty value corresponding to each grid can be used as the uncertainty value corresponding to the target voxel having a mapping relationship with the grid. The uncertainty value corresponding to each grid is generated by the reconstruction model based on the input environmental information.

[0140] Continuing with the example above, in which mesh A has a mapping relationship with target voxel C, and mesh B has a mapping relationship with target voxel D, the following description is provided. If the uncertainty value corresponding to mesh A is 1, and the uncertainty value corresponding to mesh B is 2, then the uncertainty value 1 corresponding to mesh A can be used as the uncertainty value corresponding to target voxel C having a mapping relationship with mesh A, and the uncertainty value 2 corresponding to mesh B can be used as the uncertainty value corresponding to target voxel D having a mapping relationship with mesh B. That is, the uncertainty value corresponding to target voxel C is 1, and the uncertainty value corresponding to target voxel D is 2.

[0141] Similarly, the second three-dimensional mesh information output by the reconstructed model can be regarded as a specific data format.

[0142] Optionally, for some embodiments, an uncertainty map may be generated based on each target voxel and the uncertainty value corresponding to each target voxel. The uncertainty map may be a thermal map corresponding to the three-dimensional reconstruction map, and the uncertainty map characterizes the uncertainty value of each target voxel in the unsuccessfully reconstructed area by the high and low thermal values. In the three-dimensional reconstruction map, the corresponding thermal value may be set as a default value for the area other than the unsuccessfully reconstructed area.

[0143] For example, see Figure 8 , Figure 8 FIG. 800 is a schematic diagram of an uncertainty diagram provided by an embodiment of the present application. The uncertainty diagram 800 can be viewed as Figure 3 . It can be seen that in the uncertainty map 800, the thermal value corresponding to the region 810 is relatively high, thereby indicating that the uncertainty value of each target voxel in the region 810 is relatively high, while in the uncertainty map 800, the regions other than the region 810 have relatively low thermal values. Therefore, the uncertainty value corresponding to each target voxel can be observed conveniently and quickly through the uncertainty map.

[0144] For some embodiments, after obtaining the second three-dimensional grid, the reconstruction model can directly generate the above uncertainty map based on the second three-dimensional grid and output the uncertainty map. Thus, uncertainty values ​​corresponding to multiple target voxels can be obtained in combination with the uncertainty map, and then the degree of uncertainty of each position point in the image space of the three-dimensional reconstruction map can be determined based on the uncertainty values ​​corresponding to the multiple target voxels. The quality of the three-dimensional reconstruction map obtained by scene reconstruction can be improved, that is, the three-dimensional reconstruction map has a higher accuracy. It should be noted that the accuracy of the three-dimensional reconstruction map is used to characterize the accuracy of the three-dimensional reconstruction map in representing the scene in the real physical space through the image space.

[0145] Step S370: Based on the uncertainty values ​​corresponding to multiple target voxels, determine the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image, the uncertainty value is used to characterize the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstructed image, and the standard information matches the reference information corresponding to the target voxel in the physical space.

[0146] Among them, step S370 has been introduced in detail in the above embodiment and will not be repeated here.

[0147] Optionally, for some implementations, the entropy corresponding to each target voxel may also be determined through the aforementioned reconstruction model, so that the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image may be subsequently determined based on the entropies corresponding to the multiple target voxels.

[0148] It should be noted that if the entropy corresponding to each target voxel needs to be determined through the reconstruction model, the reconstruction model needs to be adjusted beforehand, and a training set related to entropy needs to be constructed to train the reconstruction model.

[0149] Step S380: taking the position point corresponding to the highest degree of uncertainty among the degrees of uncertainty of the position points in the image space of the three-dimensional reconstructed image as the designated position point.

[0150] Step S390: Determine the position mapped by the designated location point in the physical space as the target position in the physical space, and output prompt information, wherein the prompt information is used to guide the user to move the extended reality device to the target position.

[0151] For some implementations, the position point corresponding to the highest degree of uncertainty among the degrees of uncertainty of the position points in the image space of the three-dimensional reconstructed image may be used as the designated position point.

[0152] It can be understood that the determined designated position point is a position point in the image space, so the position mapped by the designated position point in the physical space can be further determined as the target position in the physical space.

[0153] Exemplarily, each position in the physical space and each position in the image space may have a mapping function, and the mapping function may be used to convert each position in the physical space and each position in the image space. Thus, the position mapped by the designated position point in the physical space may be determined by the mapping function as the target position in the physical space, wherein the mapping function may be generated by the reconstruction model.

[0154] Therefore, the subsequent three-dimensional reconstruction map of the environment of the target location is a three-dimensional reconstruction map generated for the location point with a higher degree of uncertainty. Obtaining the environmental information of the environment corresponding to the location point with a higher degree of uncertainty, and then obtaining the corresponding three-dimensional reconstruction map, can reduce the number of times the environmental information of the environment is collected, thereby improving the efficiency of three-dimensional reconstruction.

[0155] After obtaining the target position, a prompt message can be output. As can be seen from the foregoing introduction, the extended reality device may also include a display module, so that the prompt message can be displayed through the reality module. For example, the prompt message may include a mark of the target position, and may also include a guide line to guide the user to move to the target position, thereby instructing the user to move to the target position along the guide line. Optionally, the prompt message may also display text information, for example, the text information may include "Please move to the target position". For another example, the extended reality device may also include a playback module, so that on the basis of displaying the prompt message through the reality module, voice information can also be played through the playback module. For example, the playback module can be a speaker, so that the voice message "Please move to the target position" can be played through the speaker.

[0156] Optionally, for some implementations, a planning trajectory can also be generated based on the current target position and the historical target positions before the current target position, so that the target position can be displayed to the user by displaying the planning trajectory to the user, which is interesting and improves the user experience to a certain extent.

[0157] Optionally, the prompt information may also include a guide trajectory, which may be displayed to the user through a display module of the extended reality device. The guide trajectory is a planned route of a running path from the current position of the extended reality device to the target position. It is understandable that when generating the guide trajectory, the obstacle information in the three-dimensional reconstruction image may also be combined so that the planned route of the running path from the current position of the extended reality device to the target position avoids obstacles as much as possible. Then, guiding the user to the target position through the guide trajectory not only helps to move to the expected target position efficiently, but also ensures that the user avoids obstacles and avoids collisions. Moreover, the accuracy of the subsequently acquired three-dimensional reconstruction image can be improved.

[0158] The end point of the guide trajectory is the target position. For some implementations, the position corresponding to the guide trajectory in the image space of the three-dimensional reconstructed image can be determined, and then the guide trajectory can be superimposed on the three-dimensional reconstructed image for display.

[0159] See also Fig. 9 , Fig. 9 A schematic diagram showing prompt information provided by an embodiment of the present application is shown. Specifically, Fig. 9 A three-dimensional reconstruction image 900 is shown, in which an augmented reality device 910 is currently at a position 920 and a target position 930 , so that a guide track 990 may be displayed to instruct the user to move to the target position.

[0160] Optional, in Fig. 9In the three-dimensional reconstruction diagram shown in , a corresponding user 950 can also be generated according to the current position 920 of the extended reality device 910, and the user 950 can also be regarded as being at the position 920.

[0161] It is understandable that the image acquisition module configured by the extended reality device collects the environmental information of the target environment. When the extended reality device faces different directions, the environmental information obtained may be different. Therefore, the prompt information may also include the target direction. The target direction is used to indicate the direction of the user at the target location. Please continue to refer to Fig. 9 , Fig. 9 A target direction 980 may also be displayed to indicate the user's direction at the target location.

[0162] For example, the target orientation may be the direction of the voxel with the highest uncertainty value relative to the target position among the target voxels matched by the position point corresponding to the target position. Thus, the quality of the 3D reconstruction image obtained by scene reconstruction can be improved by the target orientation.

[0163] In the embodiments provided by the present application, prompt information is output to improve the interactive experience between the extended reality device and the user during the scene reconstruction process, thereby improving the user's experience. Moreover, even if the user does not have the ability to scan the surrounding environment through the extended device to reconstruct the scene, the embodiments of the present application can also guide the user through prompt information, so that any user can improve the efficiency of scene reconstruction according to the prompt information.

[0164] The user is guided to the target position by outputting prompt information, and the target position can also be regarded as the next best view (NBV).

[0165] Step S3100: If it is detected that the extended reality device is at the target position, the environment of the target position is taken as a new target environment, a new three-dimensional reconstruction map of the new target environment is obtained, and the three-dimensional reconstruction map of the target environment is updated by the new three-dimensional reconstruction map to at least partially reconstruct the area marked as unsuccessfully reconstructed.

[0166] Among them, step S3100 has been described in detail in the above embodiment and will not be repeated here.

[0167] Step S3101: whether there is an area marked as unsuccessfully reconstructed in the updated 3D reconstructed image.

[0168] If it is detected that there is an area marked as unsuccessfully reconstructed in the updated 3D reconstructed image, the execution may jump to step S3110 ; if it is detected that there is no area marked as unsuccessfully reconstructed in the updated 3D reconstructed image, the execution may jump to step S3120 .

[0169] In some embodiments, the updated 3D reconstruction image can be processed by the aforementioned reconstruction model to obtain uncertainty values ​​of each position point in the image space corresponding to the updated 3D reconstruction image. Then, based on the uncertainty values ​​of each position point, it can be determined whether there is an area marked as unsuccessfully reconstructed in the updated 3D reconstruction image.

[0170] For example, the uncertainty value of each position point can be compared with a specified threshold value. If the uncertainty value of each position point is less than or equal to the specified threshold value, it can be determined that there is no area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction map; and if there is at least one position point with an uncertainty value greater than the specified threshold value, it can be determined that there is an area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction map, and the area marked as unsuccessfully reconstructed is the area that matches the position point corresponding to the uncertainty value greater than the specified threshold value. Optionally, it can also be determined that there is an area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction map only when it is detected that there are a specified number of position points with uncertainty values ​​greater than the specified threshold value.

[0171] For some other implementations, the uncertainty values ​​of each position point in the image space corresponding to the updated three-dimensional reconstruction image can also be compared with the uncertainty values ​​of each position point in the image space corresponding to the three-dimensional reconstruction image before the current update, so as to determine whether there is an area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction image. Specifically, for example, the uncertainty values ​​of each position point in the image space corresponding to the updated three-dimensional reconstruction image can be summed to obtain a first value, and the uncertainty values ​​of each position point in the image space corresponding to the three-dimensional reconstruction image before the current update can be summed to obtain a second value. Then, when the first value is the same or close to the second value, it can be determined that there is no area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction image; and when the first value is significantly different from the second value, it can be determined that there is an area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction image.

[0172] Step S3110: when it is detected that there is an area marked as unsuccessfully reconstructed in the updated 3D reconstructed image, return to executing the search for the area marked as unsuccessfully reconstructed in the updated 3D reconstructed image and subsequent steps until it is detected that there is no area marked as unsuccessfully reconstructed in the updated 3D reconstructed image.

[0173] That is to say, in the case of detecting that there is an area marked as unsuccessfully reconstructed in the updated 3D reconstruction map, the process of searching for the area marked as unsuccessfully reconstructed in the updated 3D reconstruction map and subsequent steps can be returned, that is, the target position can be determined again through the steps in the aforementioned embodiments, and then a new 3D reconstruction map of the new target environment is obtained, and the 3D reconstruction map of the target environment is updated through the new 3D reconstruction map, so as to realize at least partial reconstruction of the area marked as unsuccessfully reconstructed again. The detailed method description is similar to the steps in the aforementioned embodiments and will not be repeated here.

[0174] Step S3120: when it is detected that there is no area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction image, the updated three-dimensional reconstruction image is used as the three-dimensional reconstruction image corresponding to the scene where the extended reality device is located.

[0175] When it is detected that there is no area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction image, it can be determined that there is no need to perform the subsequent steps of searching for the area marked as unsuccessfully reconstructed in the updated three-dimensional reconstruction image again, and the updated three-dimensional reconstruction image can be directly used as the three-dimensional reconstruction image corresponding to the scene in which the extended reality device is located.

[0176] In the scene reconstruction method provided by the present application, uncertainty values ​​corresponding to multiple target voxels in the three-dimensional reconstruction map and the unsuccessful reconstruction area are obtained based on the reconstruction model. So that the target position in the physical space can be guided by the uncertainty values ​​corresponding to each target voxel in the subsequent determination, thereby improving the efficiency of three-dimensional reconstruction. In addition, in the embodiment provided by the present application, the position point corresponding to the highest degree of uncertainty among the uncertainty degrees of each position point in the image space of the three-dimensional reconstruction map is used as the designated position point; then the position mapped by the designated position point in the physical space is determined as the target position in the physical space, and the prompt information is output. Thus, the three-dimensional reconstruction map of the environment of the target position obtained subsequently is a three-dimensional reconstruction map generated for the position point with a higher degree of uncertainty, which can further improve the efficiency of three-dimensional reconstruction. Furthermore, in the embodiment of the present application, the prompt information can also include a guide trajectory and a target orientation, and then the user is guided to the target position by the guide trajectory, which not only helps to move to the expected target position efficiently, but also ensures that the user avoids obstacles, avoids collisions, and improves the quality of the three-dimensional reconstruction map obtained by scene reconstruction.

[0177] See also Fig.10 , Fig.10The method flow chart of the scene reconstruction method provided by the embodiment of the present application is shown. The scene reconstruction method can be applied to an extended reality device, for example, the processor of the extended reality device can be used as the execution subject. Specifically, the method includes steps S410 to S460.

[0178] Step S410: The user executes a specified application through the extended reality device.

[0179] In some embodiments, an application may be run in the extended reality device, for example, a specified application may be run, so that the user is guided to perform scene reconstruction through the specified application. Exemplarily, the specified application may be a scan guide (ScanGuide). Thus, the user may wear the extended reality device to execute the specified application through the extended reality device.

[0180] Step S420: Collecting environmental information of the target environment through the extended reality device.

[0181] The extended reality device may be configured with an image acquisition module, so that environmental information of the target environment may be acquired through the image acquisition module. The environmental information may include red, green, and blue images and depth data, and a detailed description may refer to the aforementioned embodiment.

[0182] Step S430: input the environmental information as input to the reconstruction model.

[0183] Step S440: Obtain a three-dimensional reconstruction image and an uncertainty image.

[0184] The specified application may be configured with an algorithm module, so that the three-dimensional reconstruction image and the uncertainty map can be obtained through the algorithm module. Exemplarily, the algorithm module may be a Mapping module, which includes a reconstruction model, so that the three-dimensional reconstruction image and the uncertainty map can be obtained based on the reconstruction model.

[0185] Optionally, in some implementations, obtaining the 3D reconstruction image and the uncertainty image through the reconstruction model can be performed simultaneously, rather than in a sequential manner. The reconstruction model can output the first 3D mesh information and the second 3D mesh information, and the detailed description can refer to the above embodiment.

[0186] Step S450: Determine the target location.

[0187] Step S460: Generate a planning trajectory and a guiding trajectory.

[0188] A planner may also be configured in the specified application, so that the target position can be determined and the planning trajectory and guide trajectory can be generated by the planner. The planning trajectory may include the current target position and the historical target position before the current target position; the guide trajectory is the planned route of the running path from the current position of the extended reality device to the target position. Therefore, prompt information can be output based on the planning trajectory and the guide trajectory to instruct the user to move to the target position based on the guide trajectory. Please refer to the aforementioned embodiment for a detailed description, which will not be repeated here.

[0189] The embodiments of the present application improve the efficiency of scene reconstruction through a scene reconstruction method, and can promote the development of more advanced extended reality devices to a certain extent.

[0190] See also Fig.11 , Fig.11 A structural block diagram of a scene reconstruction device provided in an embodiment of the present application is shown, which is applied to an extended reality device. The scene reconstruction device 1100 includes: a three-dimensional reconstruction image acquisition unit 1110, an unsuccessful reconstruction area acquisition unit 1120, an uncertainty degree determination unit 1130, a target position determination unit 1140 and a return execution unit 1150.

[0191] The 3D reconstruction image acquisition unit 1110 is used to acquire a 3D reconstruction image of the target environment.

[0192] Optionally, the three-dimensional reconstruction image acquisition unit 1110 can also be used to collect environmental information of the target environment through the extended reality device; input the environmental information as an input into the reconstruction model to obtain the first three-dimensional grid information output by the reconstruction model, wherein the first three-dimensional grid information includes each grid obtained by dividing the target environment and the position information of each grid; based on the each grid obtained by dividing the target environment and the position information of each grid, determine the three-dimensional reconstruction image of the target environment.

[0193] Optionally, the three-dimensional reconstruction image acquisition unit 1110 can also use the grids obtained by dividing the unsuccessfully reconstructed area and the position information of each grid to determine the target voxel in the area to be reconstructed that has a mapping relationship with each grid; and use the uncertainty value corresponding to each grid as the uncertainty value corresponding to the target voxel that has a mapping relationship with the grid.

[0194] The unsuccessfully reconstructed region acquiring unit 1120 is configured to search the three-dimensional reconstructed image for a region marked as unsuccessfully reconstructed, wherein the unsuccessfully reconstructed region includes a plurality of target voxels.

[0195] The uncertainty degree determination unit 1130 is used to determine the uncertainty degree of each position point in the image space of the three-dimensional reconstructed image based on the uncertainty values ​​corresponding to multiple target voxels, and the uncertainty values ​​are used to characterize the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstructed image, and the standard information matches the reference information corresponding to the target voxel in the physical space.

[0196] Optionally, the uncertainty degree determination unit 1130 can also be used to determine at least one alternative plane and at least one alternative reference point in each of the alternative planes in the image space of the three-dimensional reconstructed image, each alternative reference point serving as a position point in the image space of the three-dimensional reconstructed image; determine a specified spatial range corresponding to each of the position points, wherein the distance between the position point and any point within the specified spatial range is less than or equal to the detection distance of the extended reality device; search for a target voxel with an uncertainty value greater than a specified value within the specified spatial range corresponding to each of the position points as a target voxel matching the position point; and determine the uncertainty degree of the position point based on the uncertainty value corresponding to the target voxel matching each position point in the image space.

[0197] Optionally, the uncertainty degree determination unit 1130 may also be configured to take the sum of the uncertainty values ​​corresponding to the target voxels matching each position point in the image space as the uncertainty degree of the position point.

[0198] Optionally, the uncertainty degree determination unit 1130 may also be used to search for a target voxel whose uncertainty value is greater than a specified value within a specified spatial range corresponding to each of the position points as a voxel to be determined that matches the position point; based on the obstacle position information and the obstacle volume information, determine a voxel to be determined that is not blocked by an obstacle in the line of sight between the position point and each of the voxels to be determined that matches the position point as a target voxel to be determined that matches the position point.

[0199] Optionally, the uncertainty degree determination unit 1130 can also be used to divide the three-dimensional reconstructed image in the image space to obtain multiple mutually parallel two-dimensional planes as alternative planes; uniformly sample each of the alternative planes to obtain multiple sampling points as alternative reference points in the alternative plane, wherein the sampling point includes the position information of the sampling point in the image space of the three-dimensional reconstructed image.

[0200] The target position determination unit 1140 is used to determine the target position in the physical space based on the uncertainty degree of each position point in the image space of the three-dimensional reconstructed image.

[0201] Optionally, the target position determination unit 1140 can also be used to take the position point corresponding to the highest degree of uncertainty among the position points in the image space of the three-dimensional reconstructed image as the designated position point; determine the position mapped by the designated position point in the physical space as the target position in the physical space, and output prompt information, wherein the prompt information is used to guide the user to move the extended reality device to the target position.

[0202] Return to execution unit 1150, which is used to, if it is detected that the extended reality device is at the target position, take the environment of the target position as a new target environment, obtain a new three-dimensional reconstruction map of the new target environment, and update the three-dimensional reconstruction map of the target environment by means of the new three-dimensional reconstruction map, so as to at least partially reconstruct the area marked as unsuccessfully reconstructed.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0204] In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] See also Fig.12 , Fig.12 The structural block diagram of an extended reality device 1200 provided in an embodiment of the present application is shown. The extended reality device 1200 in the present application may include one or more of the following components: a processor 1211, a memory 1212, and one or more application programs, wherein the processor 1211 is electrically connected to the memory 1212, and the one or more programs are configured to execute the method described in each embodiment of the aforementioned scene reconstruction method.

[0206] Among them, the processor 1211 may include one or more processing cores. The processor 1211 uses various interfaces and lines to connect the various parts of the entire extended reality device 1200, and executes various functions and processes data of the extended reality device 1200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1212, and calling data stored in the memory 1212. Optionally, the processor 1211 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 1211 can integrate one or more combinations of central processing unit (CPU), graphics processing unit (GPU), modem, etc. Among them, the CPU mainly processes the operating system, user interface, computer program, etc.; the GPU is responsible for rendering and drawing display content; the modem is used to handle wireless communication. It is understandable that the above-mentioned modem may not be integrated into the processor 1211, but may be implemented by a communication chip alone. Specifically, the method described in the above-mentioned embodiment may be executed by one or more processors 1211.

[0207] For some embodiments, the memory 1212 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1212 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1212 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the extended reality device 1200 during use, etc.

[0208] See also Fig.13 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0209] The computer readable storage medium 1300 may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1300 includes a non-transitory computer readable storage medium. The computer readable storage medium 1300 has storage space for program code 1310 that executes any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1310 can be compressed, for example, in an appropriate form.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A scene reconstruction method, characterized in that: Applied to an extended reality device, the method comprises: Obtain a three-dimensional reconstruction of the target environment; Searching for an area marked as unsuccessfully reconstructed in the three-dimensional reconstruction image, wherein the unsuccessfully reconstructed area includes a plurality of target voxels; determining the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image based on uncertainty values ​​corresponding to a plurality of target voxels, the uncertainty values ​​being used to characterize the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstructed image, the standard information being matched with reference information corresponding to the target voxel in the physical space; Determining the target position in the physical space based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image; If it is detected that the extended reality device is at the target position, the environment of the target position is used as a new target environment, a new three-dimensional reconstruction map of the new target environment is obtained, and the three-dimensional reconstruction map of the target environment is updated by the new three-dimensional reconstruction map to at least partially reconstruct the area marked as unsuccessfully reconstructed.

2. The method according to claim 1, characterized in that The step of determining the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image based on the uncertainty values ​​corresponding to the plurality of target voxels includes: Determine at least one candidate plane and at least one candidate reference point in each candidate plane in the image space of the three-dimensional reconstructed image, each candidate reference point serving as a position point in the image space of the three-dimensional reconstructed image; Determine a designated spatial range corresponding to each of the position points, wherein a distance between the position point and any point within the designated spatial range is less than or equal to a detection distance of the extended reality device; Searching for a target voxel whose uncertain value is greater than a specified value within a specified spatial range corresponding to each of the position points as a target voxel matching the position point; Based on the uncertainty value corresponding to the target voxel matched by each position point in the image space, the degree of uncertainty of the position point is determined.

3. The method according to claim 2, characterized in that The step of determining the uncertainty degree of each position point based on the uncertainty value corresponding to the target voxel matched by each position point in the image space comprises: The sum of the uncertainty values ​​corresponding to the target voxels matching each position point in the image space is taken as the uncertainty degree of the position point.

4. The method according to claim 2, characterized in that: The three-dimensional reconstructed image includes obstacle information, and the obstacle information includes obstacle position information and obstacle volume information of the obstacle. The step of searching for a target voxel whose uncertain value is greater than a specified value within a specified spatial range corresponding to each position point as a target voxel matching the position point includes: Searching for a target voxel whose uncertain value is greater than a specified value within a specified spatial range corresponding to each position point as a voxel to be determined for matching the position point; Based on the obstacle position information and obstacle volume information, a voxel to be determined that is not blocked by the obstacle in the line of sight between the position point and each of the voxels to be determined that matches the position point is determined as a target voxel to be matched with the position point.

5. The method according to claim 2, characterized in that: The step of determining at least one candidate plane and at least one candidate reference point in each candidate plane in the image space of the three-dimensional reconstructed image comprises: The three-dimensional reconstruction image is segmented in the image space to obtain a plurality of mutually parallel two-dimensional planes as candidate planes; Each candidate plane is uniformly sampled to obtain a plurality of sampling points as candidate reference points in the candidate plane, wherein the sampling points include position information of the sampling points in the image space of the three-dimensional reconstructed image.

6. The method according to claim 1, characterized in that The step of determining the target position in the physical space based on the uncertainty degree of each position point in the image space of the three-dimensional reconstructed image comprises: taking the position point corresponding to the highest uncertainty degree among the uncertainty degrees of the position points in the image space of the three-dimensional reconstructed image as the designated position point; The position mapped by the designated location point in the physical space is determined as the target position in the physical space, and prompt information is output, wherein the prompt information is used to guide the user to move the extended reality device to the target position.

7. The method according to claim 1, characterized in that The step of obtaining a three-dimensional reconstruction image of the target environment includes: Collecting environmental information of the target environment through the extended reality device; Inputting the environmental information as an input into a reconstruction model to obtain first three-dimensional grid information output by the reconstruction model, wherein the first three-dimensional grid information includes each grid obtained by dividing the target environment and position information of each grid; Based on the grids obtained by dividing the target environment and the position information of each grid, a three-dimensional reconstruction map of the target environment is determined.

8. The method according to claim 7, characterized in that The reconstruction model is further used to output second three-dimensional grid information, wherein the second three-dimensional grid information includes each grid obtained by dividing the unsuccessfully reconstructed area, position information of each grid, and uncertainty value corresponding to each grid. Before determining the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image based on the uncertainty values ​​corresponding to the multiple target voxels, the method further includes: Based on the grids obtained by dividing the unsuccessfully reconstructed area and the position information of each grid, determining the target voxel in the unsuccessfully reconstructed area that has a mapping relationship with each grid; The uncertainty value corresponding to each grid is used as the uncertainty value corresponding to the target voxel having a mapping relationship with the grid.

9. The method according to claim 1, characterized in that: If it is detected that the extended reality device is at the target position, the environment at the target position is used as a new target environment, a new three-dimensional reconstruction map of the new target environment is obtained, and the three-dimensional reconstruction map of the target environment is updated by the new three-dimensional reconstruction map to at least partially reconstruct the area marked as unsuccessfully reconstructed, the method further includes: When it is detected that there is an area marked as unsuccessfully reconstructed in the updated 3D reconstructed image, the process returns to searching for the area marked as unsuccessfully reconstructed in the updated 3D reconstructed image and subsequent steps until it is detected that there is no area marked as unsuccessfully reconstructed in the updated 3D reconstructed image.

10. A scene reconstruction device, characterized in that: Applied to an extended reality device, the device comprises: A three-dimensional reconstruction image acquisition unit, used to acquire a three-dimensional reconstruction image of the target environment; An unsuccessfully reconstructed region acquisition unit, used to search the three-dimensional reconstructed image for a region marked as unsuccessfully reconstructed, wherein the unsuccessfully reconstructed region includes a plurality of target voxels; an uncertainty degree determination unit, for determining the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image based on uncertainty values ​​corresponding to a plurality of target voxels, wherein the uncertainty values ​​are used to characterize the difficulty of obtaining standard information of the corresponding target voxel in the three-dimensional reconstructed image, and the standard information matches the reference information corresponding to the target voxel in the physical space; A target position determination unit, configured to determine the target position in the physical space based on the degree of uncertainty of each position point in the image space of the three-dimensional reconstructed image; An updating unit is used to, if it is detected that the extended reality device is at the target position, take the environment of the target position as a new target environment, obtain a new three-dimensional reconstruction map of the new target environment, and update the three-dimensional reconstruction map of the target environment by means of the new three-dimensional reconstruction map, so as to at least partially reconstruct the area marked as unsuccessfully reconstructed.

11. An extended reality device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 9.

Images