Method, apparatus, device, and medium for managing three-dimensional models

Abstract

Methods, apparatus, devices, and media for managing 3D models are provided. In one method, a panoramic image and a 3D model of a real scene to be presented in a virtual roaming application are acquired. The 3D model and a bounding box model of the 3D model are loaded at the viewpoint of the panoramic image. The bounding box model has an initial size larger than the size of the 3D model. The size of the bounding box model is adjusted based on the size of the 3D model such that the distance between the outer contour of the adjusted bounding box model and the outer contour of the 3D model satisfies a predetermined threshold condition. Using exemplary implementations of this disclosure, containing a bounding box model outside the 3D model can reduce undesirable visual impressions of holes.

Description

Technical Field

[0001] Exemplary implementations of this disclosure generally relate to virtual roaming, and more particularly to methods, apparatus, devices, and computer-readable storage media for managing three-dimensional models in virtual roaming applications. Background Technology

[0002] With the development of digital technology, various virtual tour applications have been developed. These applications can load 3D models and allow users to roam within virtual scenes, providing an immersive visual experience. However, due to limitations in data acquisition equipment, voids may appear in the generated 3D models. Therefore, finding a more convenient and effective way to handle these voids and present the 3D models in virtual tours with near-realistic effects has become a pressing issue. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for managing three-dimensional models is provided. In this method, a panoramic image of a real-world scene to be presented in a virtual roaming application and a three-dimensional model are acquired. The three-dimensional model and a bounding box model of the three-dimensional model are loaded at the viewpoint of the panoramic image. The bounding box model has an initial size larger than the size of the three-dimensional model. The size of the bounding box model is adjusted based on the size of the three-dimensional model such that the distance between the outer contour of the adjusted bounding box model and the outer contour of the three-dimensional model satisfies a predetermined threshold condition.

[0004] In a second aspect of this disclosure, an apparatus for managing three-dimensional models is provided. The apparatus includes: an acquisition module configured to acquire a panoramic image of a real-world scene to be presented in a virtual roaming application and a three-dimensional model; a loading module configured to load the three-dimensional model and a bounding box model of the three-dimensional model at a viewpoint of the panoramic image, the bounding box model having an initial size larger than the size of the three-dimensional model; and an adjustment module configured to adjust the size of the bounding box model based on the size of the three-dimensional model, such that the distance between the outer contour of the adjusted bounding box model and the outer contour of the three-dimensional model satisfies a predetermined threshold condition.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform a method according to a first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] In the following detailed description, in conjunction with the accompanying drawings, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1A A block diagram of an example environment in which the implementation of this disclosure can be implemented is shown;

[0010] Figure 1B A block diagram illustrating the content presented in a virtual roaming application according to some technical solutions is shown;

[0011] Figure 2 A block diagram illustrating the process for managing 3D models according to some technical solutions is shown;

[0012] Figure 3 A block diagram is shown illustrating a process for managing a 3D model according to some implementations of this disclosure;

[0013] Figure 4 A block diagram is shown illustrating how the shape of a bounding box model is determined based on the type of panoramic image, according to some implementations of this disclosure.

[0014] Figure 5 A block diagram is shown illustrating the determination of the shape of a bounding box model based on the outer contour of a 3D model according to some implementations of this disclosure;

[0015] Figure 6 A block diagram is shown for adjusting the bounding box model according to some implementations of this disclosure;

[0016] Figure 7A and 7B Block diagrams are shown for presenting a bounding box model after a hole according to some implementations of this disclosure;

[0017] Figure 8 A block diagram is shown illustrating the process of using a bounding box model to compensate for holes in a 3D model according to some implementations of this disclosure;

[0018] Figure 9A flowchart is shown of a method for managing a 3D model according to some implementations of this disclosure;

[0019] Figure 10 A block diagram of an apparatus for managing a three-dimensional model according to some implementations of this disclosure is shown;

[0020] Figure 11 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0021] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0026] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] Example Environment

[0029] Various virtual tour applications have been developed, which can load 3D models and allow users to roam within virtual scenes. During the creation of the 3D model, sensor devices can be used to collect data from real objects in the real environment and then build the 3D model. However, during the data collection process, objects in the real environment, such as mirrors, glass, televisions, and carpets, may not reflect the beams emitted by the sensor devices as expected, potentially resulting in voids at corresponding locations in the 3D model.

[0030] See Figure 1A and Figure 1B To describe more details about the application environment, the Figure 1A A block diagram 100A is shown illustrating an example environment in which the implementation of this disclosure can be achieved. For example... Figure 1A As shown, a sensor device can be used to acquire 3D data of real objects in a real scene 110. The real scene 110 may include multiple objects, such as ceilings, walls, floors, and mirrors 120 on the walls, etc. When the sensor device emits a beam towards each object, the objects such as ceilings, walls, and floors can reflect the beam from the sensor device. The sensor device can then determine its distance to the objects such as ceilings, walls, and floors based on the received echoes. However, because mirror 120 performs specular reflection, this results in the sensor device not receiving an echo sufficient to determine the distance to mirror 120. In this case, holes may appear in the area where mirror 120 is located in the generated 3D model.

[0031] Figure 1BA block diagram 100B illustrates content presented in a virtual tour application according to some technical solutions. A 3D model 150 with a hole 140 can be loaded into the virtual tour application. The location of the hole 140 does not have a surface structure, allowing the model behind the 3D model 150 to be seen through the hole 140. Typically, a sky bounding box can be set, textures can be mapped onto the surface of the sky bounding box, and the sky bounding box can be loaded outside the 3D model 150. However, due to the large distance between the 3D model 150 and the sky bounding box, the rate of change of the texture on the surface of the sky bounding box seen through the hole 140 during roaming in the virtual scene 130 is significantly inconsistent with the rate of change of the texture on the walls surrounding the hole 140.

[0032] See Figure 2 To describe more details about the rate of change, the Figure 2 A block diagram illustrating the process for managing 3D models according to some technical solutions is shown. Figure 2 A top-down view shows a 3D model 150 in a virtual scene, which includes a hole 140, with the edges of the hole 140 shown as points A and B, respectively. Typically, to set an initial viewpoint in a virtual roaming application, a bounding box model 210 can be loaded, and the viewpoint from which the texture (i.e., the panoramic image) of the bounding box model 210 is captured can be set as the initial viewpoint. The bounding box model 210 has a large size to accommodate the entire 3D model 150 that will be loaded into the virtual roaming application. At this time, the bounding box model 210 is outside the 3D model 150.

[0033] Textures can be assigned to the bounding box model 210; for example, a panoramic image taken at a location in a real scene (such as the center or another location) can be mapped onto the bounding box model 210. In this way, a bounding box model 210 containing the panoramic image will be presented outside the 3D model 150. Figure 2 As shown, assuming a user (i.e., the virtual camera representing the user's viewpoint) moves from point P1 to point P2 in the virtual scene, if the user is located between point P1 and point PM (the center point of P1 and P2), a panoramic image 220 taken at point P1 is presented in the virtual roaming scene. If the user is located between point PM and point P2, a panoramic image 230 taken at point P2 is presented. Because the 3D model 150 includes holes 140, this allows the user to see the bounding box model 210 through the holes 140.

[0034] Assuming the user moves to point PM, the color of point A at the edge of hole 140 is the color Ca of the intersection of the ray from point P1 to point A and the bounding box model 210. If the position of a nearby point A within hole 140 is projected onto position C of the bounding box model, the color seen from the nearby point A is the color Cc of the intersection of the ray from point P1 to point C and the bounding box. Since the color of point A on 3D model 150 is different from the color of point C on bounding box model 210 (Ca ≠ Cc), this results in a discontinuity between the color seen through hole 140 and the color on 3D model 150 when moving to point PM, and the texture within hole 140 is significantly different from the texture around hole 140 on 3D model 150.

[0035] Typically, to ensure that the bounding box model 210 can accommodate 3D models 150 of different sizes, the size of the bounding box model 210 is usually set to be much larger than the size of the 3D model 150. This results in a difference between the distance between the user and the 3D model 150 in the area surrounding the hole 140 and the distance between the user and the bounding box model 210 seen through the hole 140 when the user moves in the virtual scene. If the distance is close, the texture moves slowly; if the distance is far, the texture moves quickly, that is, the projection speed of the texture moves faster. This makes the user clearly feel the presence of the hole 140 and reduces the visual experience of the virtual scene. At this point, how to handle the holes in the 3D model in a more convenient and effective way, and thus present the 3D model in the virtual roaming model with a near-realistic effect, becomes an urgent problem to be solved.

[0036] Overview of the process of managing 3D models

[0037] To address the shortcomings of the aforementioned technical solutions, a technical solution for managing 3D models is proposed based on an exemplary implementation of this disclosure. See also... Figure 3 Describe an overview of managing a 3D model. Figure 3 A block diagram 300 illustrates a process for managing a 3D model according to some implementations of this disclosure. For example... Figure 3 As shown, a panoramic image 310 and a 3D model 150 of the real scene to be presented in the virtual roaming application can be acquired. The initial viewpoint of the virtual roaming application can be set based on the viewpoint 320 of the panoramic image 310 (that is, the shooting point of the panoramic image 310).

[0038] A 3D model 150 and its bounding box model 210 can be loaded at viewpoint 320 of the panoramic image. Initially, the bounding box model 210 has an initial size, which can be larger than (e.g., significantly larger than) the size of the 3D model 150. In other words, the bounding box model 210 can initially enclose the 3D model 150. Furthermore, the size of the bounding box model 210 can be adjusted based on the size of the 3D model 150, such that the distance between the outer contour of the adjusted bounding box model 330 and the outer contour of the 3D model 150 becomes smaller and less than or equal to a predetermined threshold 340.

[0039] Using an exemplary implementation of this disclosure, the size of the bounding box model 210 is slightly larger than the size of the 3D model 150, and the size difference between the two is limited by a predetermined threshold 340. In this way, the sizes of the bounding box model 210 and the 3D model 150 are on similar orders of magnitude, and compared to the large bounding box model suitable for all 3D models in conventional solutions, the adjusted bounding box model 330 wraps around and is closer to the 3D model 150. This makes the movement speed of the texture inside the hole 140 (i.e., the texture in the bounding box model 330 seen through the hole 140) and outside the hole 140 (i.e., the texture in the 3D model 150 seen) similar from the user's viewpoint. This reduces the unpleasant visual experience of the hole 140 during user movement.

[0040] Using the exemplary implementation of this disclosure, it is unnecessary to check each facet in the 3D model for voids. Instead, the size of the bounding box model can be directly set based on the size of the 3D model 150, ensuring that the bounding box model tightly encloses the 3D model 150. In this way, regardless of whether voids exist in the 3D model 150, the bounding box model 210 outside the 3D model 150 can reduce the unpleasant visual appearance of voids when needed (i.e., when voids exist in the 3D model 150). Even if there are no voids in the 3D model 150, deploying the bounding box model 210 adjacent to the 3D model 150 does not increase the computational overhead of the virtual roaming application. Therefore, the bounding box model 210 provides a unified void handling technology for 3D models of different sizes and shapes, thus reducing the overhead of processing each 3D model individually.

[0041] Detailed process of managing 3D models

[0042] See already Figure 3The overview of managing the 3D model 150 is described below, with further details on generating and adjusting the bounding box model based on the 3D model. According to an exemplary implementation of this disclosure, a panoramic image 310 and a 3D model 150 of a real scene to be presented in a virtual roaming application can be obtained. It will be understood that the panoramic image 310 may be a pre-captured panoramic image used as an initial viewpoint for the virtual roaming application, and the 3D model 150 may be a 3D model obtained based on various technologies currently known and / or to be developed in the future (e.g., reconstruction based on point cloud data obtained from 3D scanning).

[0043] Furthermore, a bounding box model 210 can be generated, and the acquired 3D model 150 and its bounding box model 210 can be loaded at viewpoint 320 of the panoramic image 310. First, the method for determining the shape of the bounding box model 210 is described. The shape of the bounding box model 210 can be determined based on various methods. Figure 4 A block diagram 400 illustrates how the shape of the bounding box model 210 is determined based on the type of the panoramic image 310, according to some implementations of this disclosure. For example... Figure 4 As shown, the shape of the bounding box model 210 can be determined based on the type of the panoramic image 310.

[0044] According to an exemplary implementation of this disclosure, if the panoramic image 310 is of type cube 410, that is, the panoramic image 310 includes six images (front view, rear view, left view, right view, top view, and bottom view) in six directions from the viewpoint, a cube-type bounding box model 412 can be generated. It will be understood that the cube-type bounding box model 412 is more suitable for presenting the six images in the original panoramic image 310 in six directions, thus making fuller use of the surrounding environment information in the originally acquired panoramic image 310 and avoiding the risk of information loss during panoramic image type conversion.

[0045] According to an exemplary implementation of this disclosure, if the panoramic image 310 is of type 420 (e.g., an image with an aspect ratio of 2:1 or a circular image), then the panoramic image 310 is more suitable for being mapped to a bounding box model 422 of a sphere shape. In this case, a bounding box model 422 of the sphere type can be generated. It will be understood that since the mapping relationship between such panoramic images and the spherical coordinates of a sphere is relatively simple, generating a bounding box model 422 of the sphere type can reduce the complexity of subsequent texture mapping.

[0046] See already Figure 4The process of determining the shape of the bounding box model 210 based on the panoramic image 310 is described. Alternatively and / or additionally, the shape of the bounding box model 210 can be determined based on the 3D model 150. For example, the shape of the bounding box model 210 can be determined based on the outer contour of the 3D model 150. Assuming that the 3D model 150 is a rectangular architectural model, a cube-type bounding box model can be generated. It will be understood that the three sides of the rectangle in this bounding box model 210 can have the same or different side lengths. For example, a cube with equal side lengths can be generated based on the longest side of the architectural model. Alternatively and / or additionally, rectangles with different side lengths can be generated based on the proportions of the three side lengths of the architectural model.

[0047] See Figure 5 Describe situations where the 3D model involves irregular shapes. Figure 5 A block diagram 500 illustrates determining the shape of a bounding box model based on the outer contour of a 3D model according to some implementations of this disclosure. Figure 5 A top view of the 3D model 510 is shown, as follows: Figure 5 As shown, a corresponding bounding box model 520 can be generated based on the outer contour shape of the projection of the 3D model 510 onto the ground plane. Specifically, the outer contour shape of the projection of the bounding box model 520 onto the ground plane can be the same as the outer contour shape of the projection of the 3D model 510. For example, the outer contour of the 3D model 510 can be enlarged to determine the bounding box 520.

[0048] Using the exemplary implementation of this disclosure, the shape of the generated bounding box model 520 is similar to that of the 3D model 510. Therefore, the surfaces of the bounding box model 520 and the 3D model 510 have the same trend of change. This allows the surface of the bounding box model 520 to better match the surface of the 3D model 510, increasing the fit between them. In this way, situations where some surfaces of the bounding box model fit the 3D model while others do not can be avoided. Thus, by enhancing the fit between the bounding box model and the 3D model, the unpleasant visual experience of emptiness during virtual tours can be further reduced, thereby enhancing the user experience of virtual tour applications.

[0049] The process of determining the shape of the bounding box model and generating a bounding box model with initial dimensions has been described above. It will be understood that the initial dimensions can be set to be much larger than the dimensions of the 3D model 150. For example, this initial dimension can have the same or different values ​​for different 3D models 150. For instance, a uniformly large initial dimension can be set for virtual tour applications. For small indoor navigation applications, the initial dimension can be set to a cube bounding box model with, for example, a side length of 500 meters (or other data), so that the bounding box model can accommodate the various 3D models to be loaded. In this way, each 3D model can be processed in a uniform manner, thus avoiding the manpower and computational resource overhead associated with processing each 3D model individually.

[0050] With the bounding box model 210 already generated, the 3D model 150 and its bounding box model 210 can be loaded at viewpoint 320 of the panoramic image 310. Viewpoint 320 can be the initial viewpoint for a virtual roaming application. For example, an image taken at the center of a real scene can be used as the initial panoramic image. In this case, the initial viewpoint corresponds to the center of the real scene, and both the loaded 3D model 150 and bounding box model 210 are located at this initial viewpoint; that is, both the 3D model 150 and bounding box model 210 are located at the center of the virtual scene.

[0051] According to an exemplary implementation of this disclosure, the size of the bounding box model 210 can be adjusted based on the size of the 3D model 150 so that the distance between the outer contour of the bounding box model 210 and the outer contour of the 3D model 150 satisfies a predetermined threshold condition (e.g., less than or equal to a predetermined threshold). Figure 6 A block diagram 600 is shown for adjusting the bounding box model 210 according to some implementations of this disclosure. For example... Figure 6 As shown, the adjusted size 630 of the bounding box model can be determined based on a predetermined threshold 340, the size 610 of the 3D model 150, and the initial size 620 of the bounding box model 210. Furthermore, the bounding box model 210 can be scaled to the adjusted size 630 to obtain the adjusted bounding box model 330.

[0052] Here, the predetermined threshold 340 can be determined in various ways. For example, the predetermined threshold 340 can be set to a fixed default value (e.g., 0.5 meters or 1 meter, etc.). Alternatively and / or additionally, the predetermined threshold 340 can be determined based on the size 610 of the 3D model 140. Specifically, the predetermined threshold 340 can be set to a predetermined proportion of the size 610 of the 3D model 150 (e.g., 5%, 10%, or other proportions).

[0053] Assuming the 3D model 150 has dimensions 610 of 10m × 10m × 3m (rectangular), the predetermined threshold 340 can be determined as 10 × 5% = 0.5m based on the 10m side length of the 3D model 150 in the horizontal direction. In this case, the bounding box model 330 can have dimensions of (10 + 0.5 × 2)m × (10 + 0.5 × 2)m × (2 + 0.5 × 2)m = 11m × 11m × 4m. Alternatively and / or additionally, the length, width, and height of the 3D model 150 can be processed separately, and predetermined thresholds can be determined for each of the three dimensions. It will be understood that since the bounding box model 330 needs to accommodate the 3D model 150, the adjusted size 630 of the bounding box model should be larger than the size 610 of the 3D model 150.

[0054] Using the exemplary implementation of this disclosure, the relationship between the size 340 of the 3D model 150 and the size of the adjusted bounding box model 330 can be predefined. This ensures that the bounding box model 330 always appears at a predetermined threshold outside the hole 140. In this way, on the one hand, the unpleasant visual experience of the hole 140 can be reduced, and on the other hand, flickering problems caused by the bounding box model 330 being too close to the 3D model 150 can be avoided.

[0055] According to an exemplary implementation of this disclosure, the size of the bounding box model can be reduced, for example, by reducing the size of the bounding box model from the initial size to the adjusted size 630 along a direction from the initial size to the adjusted size 630 obtained based on a predetermined threshold 340. Using the exemplary implementation of this disclosure, the adjusted size of the bounding box model 330 can be determined with only simple mathematical calculations.

[0056] According to one exemplary implementation of this disclosure, the size of the bounding box model can be adjusted only once during the use of a virtual roaming application. Alternatively and / or additionally, the size of the bounding box model can be adjusted multiple times based on the user's roaming position. It will be understood that when the bounding box model is too close to the 3D model, this may cause a flickering problem in the corresponding facets within the bounding box model 210 due to the excessively close distance between the 3D model 150 and the facets. Here, "close" refers to the distance between the virtual camera and the facets in the virtual roaming application. In this case, the size of the bounding box model can be adjusted by adjusting a predetermined threshold based on the distance between the virtual camera and the facets in the outer contour of the 3D model 150.

[0057] According to an exemplary implementation of this disclosure, assuming the distance between the virtual camera and the surface is large, a predetermined threshold can be appropriately increased (that is, the bounding box model can be set to correspond to a larger scaling factor); assuming the distance between the virtual camera and the surface is small, the predetermined threshold can be appropriately decreased (that is, the bounding box model can be set to correspond to a smaller scaling factor). Using the exemplary implementation of this disclosure, a balance can be struck between preventing flicker and reducing the unpleasant visual experience caused by holes, thereby improving the user's roaming experience.

[0058] According to one exemplary implementation of this disclosure, the content presented in the virtual scene 130 is determined based on the position of the virtual camera in the virtual roaming application. In other words, the 3D model 140 and the bounding box model 330 can be presented based on the position of the virtual camera in the virtual roaming application.

[0059] According to an exemplary implementation of this disclosure, the bounding box model 330 includes two parts: a three-dimensional structure of the model and a corresponding texture. In the initial stage, the panoramic image 310 can be mapped onto the bounding box model 330 as a texture. The mapping relationship between the texture coordinates and the three-dimensional coordinates of the bounding box model can be determined based on various mapping methods currently known and / or to be developed in the future, which will not be elaborated further in this disclosure. According to an exemplary implementation of this disclosure, the panoramic image 310 can be mapped onto the surface of the bounding box model 330. Furthermore, a bounding box model 330 with a texture can be loaded, and the bounding box model 330 with its surface covered by the panoramic image 310 can be presented in a virtual roaming application. For example, at the surface of the bounding box model 330 behind the hole 140, a portion of the panoramic image 310, i.e., the content of the mirrored portion of the panoramic image 310, can be presented.

[0060] Figure 7A A block diagram 700A is shown illustrating the presentation of a bounding box model after a hole, according to some implementations of this disclosure. Using an exemplary implementation of this disclosure, a corresponding portion of the panoramic image 310 is presented at the surface of the bounding box model 330. In this way, the corresponding portion of the panoramic image can be used to compensate for the cutout effect of the hole 140 in the 3D model 150. It will be understood that although a portion of the panoramic image 310 can be displayed after the hole 140 and the cutout effect can be mitigated, the content displayed at this time may not always match the user's current position.

[0061] Suppose that the initial panoramic image 310 used as the texture was captured on the left side of the mirror, but the current user is located on the right side. In this case, the texture of the bounding box model will not match the user's current position, causing the user to see an incorrect content in the mirror area. It will be understood that as the user moves within the virtual scene, the surrounding scenery seen by the user will continuously change. Specifically, the content of the mirror area seen by the user will be completely different when the user is located on the left or right side of the mirror. Therefore, based on the position of the virtual camera at the user's location, a panoramic image matching the position can be selected, and the texture of the bounding box model can be updated using the selected panoramic image.

[0062] Figure 7B A block diagram 700B is shown illustrating some implementations of this disclosure of presenting a bounding box model after a hole. For example... Figure 7B As shown, a 3D model 150 can be rendered in the virtual scene 130, and this 3D model 150 has a hole 140. Further, a portion of a bounding box model 330 (i.e., the portion seen through the hole 140) can be rendered after the 3D model 150. A panoramic image 710, whose shooting location is closest to the user's current location, can be selected. Further, the selected panoramic image 710 can be mapped as a texture onto the bounding box model 330 so that the bounding box model 330, with the selected panoramic image 710 as its texture, is rendered after the hole 140 in the 3D model 150.

[0063] Using the exemplary implementation of this disclosure, since the selected panoramic image 710 is captured near the user's current location, the content of the panoramic image 710 more closely matches the surrounding scenery as seen by the user from their current location. In this way, virtual scenes can be presented with higher realism, thereby improving the user experience of virtual reality applications.

[0064] Figure 8 A block diagram 800 illustrates a process for compensating for holes 140 in a 3D model 150 using a bounding box model, according to some implementations of this disclosure. For example... Figure 8 As shown, a bounding box model 330 (with adjusted dimensions matching the dimensions of the 3D model 150) exists outside the 3D model 150. During the user's movement from point P1 to point P2, if the user is located between points P1 and PM, the texture of the bounding box model 330 is the panoramic image 220 taken at point P1. If the user is located between points PM and P2, the texture of the bounding box model 330 is the panoramic image 230 taken at point P2.

[0065] Because the distance between the bounding box model 330 and the 3D model 150 is small, the bounding box model 330 can wrap around the 3D model 150 at close range. In this way, the problem of inconsistent texture movement speed inside and outside the hole can be alleviated when the user moves, thereby improving the user experience of virtual reality applications.

[0066] According to an exemplary implementation of this disclosure, the method described above can be used in indoor roaming scenes to address the problem of voids in the outer contour objects (e.g., walls, ceilings, floors, etc.) of 3D models in indoor scenes. Using the exemplary implementation of this disclosure, it is not necessary to detect whether voids actually exist in each 3D model to be loaded; instead, the size of the bounding box model needs to be adjusted accordingly based on the size of each 3D model. Furthermore, the bounding box model can be adjusted to a size that can accommodate the 3D models, and the texture of the bounding box model can be determined according to the user's current position in the virtual space. In this way, all 3D models to be loaded into the virtual roaming application are processed uniformly.

[0067] If the outer contour of a 3D model contains holes, a bounding box model can be used to compensate for the undesirable visual effect of these holes. If the 3D model does not contain holes, adjusting the size of the bounding box model will not increase the computational resource overhead of the virtual tour application. In this way, compensation measures can be provided for potential holes without needing to check whether the 3D model contains holes, thereby improving the user experience of virtual tour applications.

[0068] Example process

[0069] Figure 9 A flowchart of a method 900 for managing a 3D model according to some implementations of this disclosure is shown. Specifically, at block 910, a panoramic image and a 3D model of a real scene to be presented in a virtual roaming application are acquired. At block 920, the 3D model and a bounding box model of the 3D model are loaded at the viewpoint of the panoramic image. The bounding box model has an initial size that is larger than the size of the 3D model. At block 930, the size of the bounding box model is adjusted based on the size of the 3D model such that the distance between the outer contour of the adjusted bounding box model and the outer contour of the 3D model satisfies a predetermined threshold condition.

[0070] According to an exemplary implementation of this disclosure, adjusting the bounding box model includes: reducing the size of the bounding box model from an initial size to an adjusted size, wherein the adjusted size is larger than the size of the 3D model.

[0071] According to an exemplary implementation of this disclosure, loading a bounding box model includes: mapping a panoramic image as a texture of the bounding box model to the bounding box model; and loading a bounding box model with texture.

[0072] According to an exemplary implementation of this disclosure, the method 900 further includes: determining the shape of a bounding box model; and generating a bounding box model having the shape.

[0073] According to an exemplary implementation of this disclosure, determining the shape of the bounding box model includes at least one of the following: determining the shape of the bounding box model based on the type of the panoramic image; determining the shape of the bounding box model based on the outer contour of the 3D model.

[0074] According to one exemplary implementation of this disclosure, the three-dimensional model includes a three-dimensional model of an outer contour object in a real scene, the outer contour object including at least one of the following: a wall, a ceiling, a floor, and at least one facet of the three-dimensional model includes a hole.

[0075] According to an exemplary implementation of this disclosure, the method 900 further includes: presenting a 3D model and a bounding box model based on the position of a virtual camera in a virtual roaming application.

[0076] According to an exemplary implementation of this disclosure, rendering a bounding box model includes: selecting a panoramic image that matches the position of a virtual camera; mapping the selected panoramic image as a texture onto the bounding box model; and rendering the bounding box model with texture.

[0077] According to an exemplary implementation of this disclosure, the method 900 further includes: determining a predetermined threshold based on the dimensions of the three-dimensional model.

[0078] Example devices and equipment

[0079] Figure 10 A block diagram of an apparatus 1000 for managing a 3D model according to some implementations of the present disclosure is shown. The apparatus 1000 includes: an acquisition module 1010 configured to acquire a panoramic image and a 3D model of a real scene to be presented in a virtual roaming application; a loading module 1020 configured to load the 3D model and a bounding box model of the 3D model at the viewpoint of the panoramic image, the bounding box model having an initial size larger than the size of the 3D model; and an adjustment module 1030 configured to adjust the size of the bounding box model based on the size of the 3D model, such that the distance between the outer contour of the adjusted bounding box model and the outer contour of the 3D model satisfies a predetermined threshold condition.

[0080] According to an exemplary implementation of this disclosure, the adjustment module 1030 includes: a shrinking module configured to shrink the size of the bounding box model from an initial size to an adjusted size, the adjusted size being larger than the size of the three-dimensional model.

[0081] According to an exemplary implementation of this disclosure, the loading module 1020 includes: a mapping module configured to map the panoramic image as a texture of the bounding box model to the bounding box model; and a model loading module configured to load the bounding box model with texture.

[0082] According to an exemplary implementation of this disclosure, it further includes: a shape determination module configured to determine the shape of the bounding box model; and a generation module configured to generate a bounding box model having the shape.

[0083] According to an exemplary implementation of this disclosure, the shape determination module includes at least one of the following: a first shape determination module configured to determine the shape of the bounding box model based on the type of the panoramic image; and a second shape determination module configured to determine the shape of the bounding box model based on the outer contour of the three-dimensional model.

[0084] According to one exemplary implementation of this disclosure, the three-dimensional model includes a three-dimensional model of an outer contour object in a real scene, the outer contour object including at least one of the following: a wall, a ceiling, a floor, and at least one facet of the three-dimensional model includes a hole.

[0085] According to one exemplary implementation of this disclosure, it further includes: a rendering module configured to render a 3D model and a bounding box model based on the position of a virtual camera in a virtual roaming application.

[0086] According to an exemplary implementation of this disclosure, the rendering module includes: a selection module configured to select a panoramic image that matches the position of a virtual camera; a mapping module configured to map the selected panoramic image as a texture onto a bounding box model; and a model rendering module configured to render a bounding box model with texture.

[0087] According to an exemplary implementation of this disclosure, it further includes: a threshold determination module configured to determine a predetermined threshold based on the dimensions of the 3D model.

[0088] Figure 11 A block diagram of a device 1100 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 11 The computing device 1100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. Figure 11 The computing device 1100 shown can be used to implement the method described above.

[0089] like Figure 11As shown, computing device 1100 is in the form of a general-purpose computing device. Components of computing device 1100 may include, but are not limited to, one or more processors or processing units 1110, memory 1120, storage devices 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processing unit 1110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 1100.

[0090] Computing device 1100 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1130 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 1100.

[0091] The computing device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 11 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1120 may include computer program product 1125 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0092] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 1100 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1100 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0093] Input device 1150 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1160 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1100 can also communicate as needed with one or more external devices (not shown) via communication unit 1140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1100, or with any device that enables computing device 1100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0094] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0095] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0096] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0097] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0099] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for managing a three-dimensional model, comprising: obtaining a panoramic image and a three-dimensional model of a real scene to be presented in a virtual roaming application, the three-dimensional model including a hole; loading the three-dimensional model and a bounding box model of the three-dimensional model at a viewpoint of the panoramic image, the bounding box model having an initial size and the initial size being larger than a size of the three-dimensional model; determining a predetermined threshold based on a predetermined proportion of the size of the three-dimensional model; determining an adjusted size of the bounding box model based on the size of the three-dimensional model and the predetermined threshold; scaling down the size of the bounding box model from the initial size to the adjusted size such that a distance between an outer contour of the adjusted bounding box model and an outer contour of the three-dimensional model satisfies the predetermined threshold; updating the predetermined threshold based on a distance between a position of a virtual camera in the virtual roaming application and a face of the outer contour of the three-dimensional model; and updating the size of the bounding box based on the updated predetermined threshold such that the distance between the outer contour of the updated bounding box model and the outer contour of the three-dimensional model satisfies the updated predetermined threshold.

2. The method of claim 1, wherein the adjusted size is larger than the size of the three-dimensional model.

3. The method of claim 1, wherein loading the bounding box model comprises: mapping the panoramic image as a texture to the bounding box model; and loading the bounding box model with the texture.

4. The method of claim 1, further comprising: determining a shape of the bounding box model; and generating the bounding box model with the shape.

5. The method of claim 4, wherein determining the shape of the bounding box model comprises at least one of: determining the shape of the bounding box model based on a type of the panoramic image; determining the shape of the bounding box model based on the outer contour of the three-dimensional model.

6. The method of claim 1, wherein the three-dimensional model includes a three- dimensional model of an outer contour object in the real scene, the outer contour object including at least one of: a wall, a ceiling, a floor, and at least one face of the three-dimensional model including a hole. presenting the three-dimensional model and the bounding box model based on a position of a virtual camera in the virtual roaming application.

8. The method of claim 7, wherein presenting the bounding box model comprises: selecting a panoramic image matching the position of the virtual camera based on the position of the virtual camera; 7. The method of claim 6, further comprising: mapping the selected panoramic image as a texture to the bounding box model; and presenting the bounding box model with the texture.

9. An apparatus for managing a three-dimensional model, comprising: an obtaining module configured to obtain a panoramic image and a three- dimensional model of a real scene to be presented in a virtual roaming application, the three-dimensional model including a hole; ​ ​ ​ ​ a loading module configured to load the three-dimensional model and a bounding box model of the three-dimensional model at a viewpoint of the panoramic image, the bounding box model having an initial size and the initial size being greater than a size of the three-dimensional model; a threshold determining module configured to determine a predetermined threshold based on a predetermined proportion of the size of the three-dimensional model; a size determining module configured to determine an adjusted size of the bounding box model based on the size of the three-dimensional model and the predetermined threshold; a reducing module configured to reduce the size of the bounding box model from the initial size to the adjusted size such that a distance between an outer contour of the adjusted bounding box model and an outer contour of the three-dimensional model satisfies the predetermined threshold; a threshold updating module configured to update the predetermined threshold based on a distance between a position of a virtual camera in the virtual roaming application and a face of the outer contour of the three-dimensional model; and an updating module configured to update the size of the bounding box based on the updated predetermined threshold such that the distance between the outer contour of the updated bounding box model and the outer contour of the three-dimensional model satisfies the updated predetermined threshold.

10. The apparatus of claim 9, wherein the adjusted size is greater than the size of the three-dimensional model.

11. The apparatus of claim 9, wherein the loading module comprises: a mapping module configured to map the panoramic image as a texture of the bounding box model to the bounding box model; and a model loading module configured to load the bounding box model with the texture.

12. The apparatus of claim 9, further comprising: a shape determining module configured to determine a shape of the bounding box model; and a generating module configured to generate the bounding box model with the shape.

13. The apparatus of claim 12, wherein the shape determining module comprises at least one of: a first shape determining module configured to determine the shape of the bounding box model based on a type of the panoramic image; a second shape determining module configured to determine the shape of the bounding box model based on the outer contour of the three-dimensional model.

14. The apparatus of claim 9, wherein the three-dimensional model comprises a three-dimensional model of an outer contour object in the real scene, the outer contour object comprising at least one of: a wall, a ceiling, a floor, and at least one face of the three-dimensional model comprises a hole. a presenting module configured to present the three-dimensional model and the bounding box model based on a position of a virtual camera in the virtual roaming application.

16. The apparatus of claim 15, wherein the presenting module comprises: a selecting module configured to select a panoramic image matching the position of the virtual camera based on the position of the virtual camera; 15. The apparatus of claim 14, further comprising: a mapping module configured to map the selected panoramic image as a texture to the bounding box model; and a rendering module configured to render the bounding box model with the texture. ​ ​ ​ a model rendering module configured for rendering the bounding box model with the texture.

17. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1 to 8.

18. A computer-readable storage medium having stored thereon a computer program, the computer program, when executed by a processor, causing the processor to implement the method according to any one of claims 1 to 8.

Images