Method, computer device, storage medium and computer program for presenting a virtual representation

Abstract

The present application provides a method and apparatus, a computer device, a storage medium, and a computer program product for presenting a virtual representation of a real space, the method including the steps of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, superimposing a color image corresponding to an arbitrary observation point among a plurality of observation points and a depth image corresponding to the color image to obtain a superimposed image, mapping each of the superimposed images corresponding to the plurality of observation points onto a plurality of spheres in a virtual space, respectively, performing spatial transformation on each vertex of the plurality of spheres in the virtual space, and performing spatial editing and coloring processing on an arbitrary vertex of an arbitrary sphere to obtain and present a corresponding virtual representation in the virtual space of a corresponding local scene in the real space. [Representative diagram] Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001]

[0001] This application claims priority to Chinese Patent Application No. CN202110171986.9, filed on February 5, 2021, the contents of which are incorporated by reference in their entirety into this specification for all purposes. [Technical field]

[0002] The present disclosure relates to virtual reality (VR) technology, and in particular to methods and apparatus, computer devices, storage media and computer program products for presenting a virtual representation of a real space. [Background technology]

[0003]

[0003] Virtual reality technology has been applied in many fields. For example, this technology has been used to display interior scenes of real spaces (e.g., houses for sale / rent), allowing users to intuitively understand various information in the real space. In the currently emerging technology for presenting real spaces, the real space needs to be 3D modeled to generate a virtual reality image. The 3D modeling process is complicated, requiring a lot of computing power and processing time. In addition, the virtual reality image cannot be viewed before generating the 3D model.

[0004]

[0004] This can cause the following problems: 1. When viewing virtual reality images online, users need to wait for a long time, resulting in a poor user experience; 2. If the 3D model cannot be generated correctly, the virtual reality image cannot be generated, which is of low practicality. Summary of the Invention

[0005] It would be advantageous to provide a mechanism for mitigating, alleviating, or even eliminating one or more of the above problems.

[0006]

[0006] A method for presenting a virtual representation of a real space according to one aspect of the present disclosure includes the steps of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, where the plurality of color images correspond to corresponding local scenes in the real space observed at a plurality of observation points in the real space, and the plurality of depth images each contain depth information of the corresponding local scenes; superimposing, for any observation point, a color image from the plurality of color images corresponding to that observation point and a depth image from the plurality of depth images corresponding to that color image to obtain a superimposed image; and mapping the superimposed images corresponding to the plurality of observation points onto a plurality of spheres in the virtual space, respectively. The method includes the steps of: mapping an arbitrary sphere to a corresponding observation point among a plurality of observation points, the arbitrary sphere including a plurality of vertices, the arbitrary vertex having corresponding color information and corresponding depth information; performing spatial transformation in the virtual space for each vertex of the plurality of spheres based on the relative spatial relationship between the plurality of observation points in the real space; performing spatial editing for an arbitrary vertex of the arbitrary sphere based on the depth information of the vertex; and obtaining and presenting a corresponding virtual representation in the virtual space of the corresponding local scene in the real space by performing a coloring process for an arbitrary vertex of the arbitrary sphere based on the color information of the vertex.

[0007]

[0007] According to another aspect of the present disclosure, an apparatus for presenting a virtual representation of a real space includes an image acquisition unit configured to acquire a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, where the plurality of color images correspond to corresponding local scenes in the real space observed at a plurality of observation points in the real space, and the plurality of depth images each include depth information of the corresponding local scenes; an image superimposition unit configured to, for an arbitrary observation point, superimpose a color image among the plurality of color images corresponding to the observation point and a depth image among the plurality of depth images corresponding to the color image to obtain a superimposed image; and a virtual representation unit configured to map the superimposed images corresponding to the plurality of observation points onto a plurality of spheres in a virtual space, respectively, to obtain a superimposed image of an arbitrary observation point. the mapping unit configured such that each sphere corresponds to a corresponding observation point among the plurality of observation points and includes a plurality of vertices, with any vertex having corresponding color information and corresponding depth information; a spatial transformation unit configured to perform spatial transformation in the virtual space for each vertex of the plurality of spheres based on a relative spatial relationship between the plurality of observation points in the real space; a vertex editing unit configured to perform spatial editing for any vertex of any sphere based on the depth information of the vertex; and a coloring unit configured to obtain and present a corresponding virtual representation in the virtual space of the corresponding local scene in the real space by performing a coloring process on any vertex of any sphere based on the color information of the vertex.

[0008]

[0008] According to yet another aspect of the present disclosure, a computing device includes a memory, a processor, and a computer program stored in the memory, the processor configured to execute the computer program to implement the steps of the above method.

[0009]

[0009] According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the above-described method when executed by a processor.

[0010]

[0010] According to yet another aspect of the present disclosure, a computer program product includes a computer program which, when executed by a processor, implements the steps of the above method.

[0011]

[0011] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0012]

[0012] Hereinafter, further details, features and advantages of the present disclosure will be disclosed in the following description of exemplary embodiments, with reference to the accompanying drawings, in which:

[0013] [Figure 1] 1 is a flowchart of a method for presenting a virtual representation of a real space according to an exemplary embodiment.

[0014] [Figure 2A] 2A and 2B show an example color image and an example depth image, respectively, acquired by the method of FIG. 1 in accordance with an example embodiment. [Figure 2B] 2A and 2B show an example color image and an example depth image, respectively, acquired by the method of FIG. 1 in accordance with an example embodiment.

[0015] [Diagram 3] FIG. 2 is a diagram showing an example image obtained by mapping superimposed images corresponding to multiple observation points onto multiple spheres in a virtual space, respectively, in the method of FIG. 1 according to an example embodiment.

[0016] [Figure 4] FIG. 4 illustrates an example image obtained by rotating and translating the multiple spheres of FIG.

[0017] [Diagram 5]FIG. 5 shows an exemplary image obtained by performing spatial editing on the vertices of the arbitrary sphere of FIG.

[0018] [Figure 6] FIG. 4 illustrates an example image after spatial editing has been performed on the vertices of the spheres corresponding to each observation point in FIG. 3.

[0019] [Figure 7] 4 is a flowchart of a method for presenting a virtual representation of a real space according to another exemplary embodiment.

[0020] [Figure 8]

[0020] FIG. 1 illustrates an exemplary view presented by a view window while a user views a virtual representation image in accordance with an exemplary embodiment.

[0021] [Figure 9]

[0021] FIG. 1 illustrates an exemplary view presented by a view window after a user switches observation points in accordance with an exemplary embodiment.

[0022] [Figure 10]

[0022] FIG. 1 illustrates an exemplary view presented by a view window after a user switches the observation point in accordance with an exemplary embodiment.

[0023] [Figure 11] FIG. 1 is a structural block diagram of an apparatus for presenting a virtual representation of a real space according to an exemplary embodiment.

[0024] [Figure 12]

[0024] FIG. 1 is a structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025]

[0025] In this disclosure, unless otherwise specified, terms such as "first", "second", etc. are used to describe various elements and are not intended to limit the positional, timing or importance relationships of these elements, and the terms are only used to distinguish one element from another. In some instances, the first element and the second element may refer to the same instance of the elements, and in some cases, they may refer to different instances based on the context description.

[0026]

[0026] The terms used in the description of various examples in this disclosure are intended only to describe specific examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, when the number of elements is not specifically limited, the elements may be one or more. As used herein, the term "plurality" means two or more, and the term "based on" should be interpreted as "based at least in part on." Additionally, the terms "and / or" and "at least one of" include any one and all possible combinations of the listed items.

[0027]

[0027] Exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0028]

[0028] Figure 1 is a flowchart of a method 100 for presenting a virtual representation of a real space in accordance with an example embodiment. As shown in Figure 1, the method 100 generally includes steps 110 through 160, which may be performed, for example, on a virtual reality terminal device, although the disclosure is not limited in this respect.

[0029]

[0029] In step 110, a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images are obtained, the plurality of color images corresponding to corresponding local scenes in a real space observed at a plurality of observation points in the real space, and the plurality of depth images each include depth information of the corresponding local scenes.

[0030]

[0030] In each embodiment, the real space may be a space in various buildings, such as a residential building, an office building, a factory, a warehouse, etc. In one example, an image capture device (e.g., a professional image collection device or a general camera) and a depth camera can be used to take panoramic shots at multiple observation points in the real space, respectively, to collect the above multiple color images and depth images. The number of observation points can depend on the spatial size of the real space. According to some embodiments, multiple observation points can be uniformly set in the real space, so that all details of the interior scene of the real space can be seen from these multiple observation points without blind spots. For ease of interpretation, a suite room in a residential building is described below as an example of a real space, but the realization of the present disclosure is not limited by the specific type of real space.

[0031]

[0031] FIG. 2A illustrates an exemplary color image 200A acquired in step S110. This exemplary color image may be a single color image obtained by synthesizing (e.g., by Gauss-Kruger projection) six color images collected from six directions, namely, up, down, left, right, front, and back, at one observation point in the suite. FIG. 2B illustrates an exemplary depth image 200B acquired in step S110. This exemplary depth image is a depth image corresponding to the color image of FIG. 2A, collected at the observation point for collecting the color image 200A. According to some embodiments, the color image and depth image collected by the image capture device and the depth camera may be directly transmitted to a virtual reality terminal device for subsequent use. According to some embodiments, the color image and depth image collected by the image capture device and the depth camera may be stored in a server, in which case, when a user requests to view a virtual representation of a real space, the terminal device can obtain the corresponding color image and depth image from the server and complete the processing of the virtual representation image for the user to view.

[0032] In step 120, for any observation point, a color image from among the plurality of color images corresponding to that observation point and a depth image from among the plurality of depth images corresponding to that color image are superimposed to obtain a superimposed image.

[0033]

[0033] For example, by superimposing the color image 200A of FIG. 2A and the depth image 200B of FIG. 2B, a superimposed image corresponding to the observation point for collecting the color image 200A and the depth image 200B can be obtained. The superimposed image includes a number of pixel points, and any pixel point includes color information of the corresponding pixel point in the color image and depth information of the corresponding pixel point in the depth image. The superimposition process may be performed by image processing software such as Web GL / open GL. Specifically, first, the color image and the depth image are adjusted to two images with the same resolution so that the color image and the depth image include the same number of pixel points. This adjustment step may be realized by operations such as enlarging, reducing, stretching, etc. the image. Next, the color image and the depth image are superimposed so that the pixel points of the color image correspond one by one to the pixel points of the depth image to obtain a superimposed image.

[0034]

[0034] In step 130, the superimposed images corresponding to the multiple observation points are respectively mapped onto multiple spheres in the virtual space, such that any sphere corresponds to one corresponding observation point among the multiple observation points and includes multiple vertices, with any vertex having corresponding color information and corresponding depth information.

[0035]

[0035] In some embodiments, a virtual space can be created using image processing software such as Web GL / open GL, and multiple spheres can be provided in the virtual space to present the multiple superimposed images in the virtual space. The number of spheres is the same as the number of observation points, and each superimposed image is projected onto a corresponding sphere by mapping, thereby obtaining multiple spherical images. Projecting a planar image onto a sphere is a technique well known to those skilled in the art of image processing, and will not be described in detail here.

[0036]

[0036] Referring to Fig. 3, an exemplary image 300 is shown obtained by mapping each superimposed image corresponding to five observation points in the real space onto a plurality of spheres 301 to 305 in the virtual space, respectively. In this example, the spheres 301 to 305 are initially positioned at the same spherical centers O1 to O5 and have the same radius. Thus, the spheres 301 to 305 are shown as superimposed spheres in Fig. 3. In some embodiments, the radii of the plurality of spheres 301 to 305 may not be equal, and the spherical centers may not be at the same location. In the present disclosure, the initial radii and spherical center positions of the plurality of spheres are not limited.

[0037]

[0037] Images on the spheres 301 to 305 are presented in the virtual space in the form of a point cloud. As shown in FIG. 3, any one of the spheres 301 to 305 includes multiple vertices, and any one of the vertices has corresponding color information and corresponding depth information. In computer graphics, a three-dimensional object (including a sphere) is usually shown as a collection of triangular polyhedrons. The term "vertex" as used herein refers to the geometric vertices of such triangular polyhedrons. By operating on multiple vertices, adjustment of the position and shape of the sphere can be realized. Specifically, the coordinates and number of vertices can be determined by functions built into image processing software such as Web GL / open GL, and the position of the entire sphere can be accurately adjusted.

[0038] In step 140, a spatial transformation is performed in the virtual space for each vertex of the plurality of spheres based on the relative spatial relationships between the plurality of observation points in the real space.

[0039]

[0039] The purpose of the spatial transformation is to make the distribution of the multiple superimposed images mapped onto the multiple spheres in the virtual space correspond to the relative spatial relationship between the multiple observation points in the real space. In this embodiment, a spatial transformation matrix built into the image processing software such as Web GL / open GL can be called to perform spatial transformation on the coordinates of the vertices. The transformation parameters of this spatial transformation matrix can be determined when collecting the color image and the depth image at each observation point and can be stored for later provision to the image processing software. The spatial transformation may include scaling, rotation, and / or translation. For example, in some embodiments, the spatial transformation includes both of scaling, rotation, and translation. In other embodiments, the spatial transformation may include three of scaling, rotation, and translation at the same time.

[0040]

[0040] Figure 4 is a diagram showing an example image 400 obtained by rotating and translating the multiple spheres 301 to 305 of Figure 3. As shown in Figure 4, due to spatial transformation, the spheres 301 to 305 that were originally overlapped in Figure 3 are now positioned at different sphere centers 01 to 05, respectively, and the relative spatial relationship between the sphere centers 01 to 05 coincides with the relative spatial relationship between the corresponding observation points in real space.

[0041] In step 150, for any vertex of any sphere, spatial editing is performed on that vertex based on the depth information of that vertex.

[0042]

[0042] According to some embodiments, spatial editing may be performed by a vertex shader built into image processing software such as Web GL / open GL. The vertex shader is a set of instruction codes that are executed when a vertex is rendered. In one example, the vertex shader first obtains the vertex coordinates of any vertex of a sphere, and then moves the coordinates of any vertex by an offset distance along the normal direction of the vertex, respectively, and this offset distance corresponds to the depth information of the vertex. The depth information is actually information for representing the distance from an observation point to a real space scene, so that the contour shape of the real space interior scene can be obtained after offsetting the corresponding vertices along the normal direction of the vertex based on the depth information, respectively.

[0043]

[0043] Figure 5 illustrates an exemplary image 500 resulting from spatial editing on the vertices of spheres 301 to 305 of Figure 4, where sphere centers 01 to 05 represent the first to fifth observation points in real space, respectively. From Figure 5, it can be intuitively seen that the relative spatial relationships between sphere centers 01 to 05 now reflect the relative spatial relationships between the corresponding observation points in real space.

[0044]

[0044] It is also possible to reverse the order of steps 140 and 150. That is, spatial editing can be performed on each vertex of the arbitrary sphere first, and then spatial transformation of the arbitrary sphere can be performed. FIG. 6 is a diagram showing an exemplary image 600 obtained after spatial editing is performed on the vertices of the spheres 301 to 305 in FIG. 3. As shown in FIG. 6, since no spatial transformation such as translation is performed, the sphere centers 01 to 05 of the spheres 301 to 305 are still positioned at the same positions. However, since spatial editing is performed on the vertices of the arbitrary sphere (e.g., the coordinates of the arbitrary vertex are moved along the normal direction of the vertex by an offset distance corresponding to the depth information of the vertex), the spheres 301 to 305 originally shown as "spheres" in FIG. 3 are now not spheres. Even after the spatial transformation described above with respect to step 140 is performed on the virtual space image 600, the virtual space image 500 shown in FIG. 5 can be obtained.

[0045] In step 160, for any vertex of any sphere, a coloring process is performed on the vertex based on the color information of the vertex, thereby obtaining and presenting a corresponding virtual representation in the virtual space of the corresponding local scene in the real space.

[0046]

[0046] According to some embodiments, a fragment shader built into image processing software such as Web GL / open GL can be used to perform coloring processing on any vertex of a sphere. In one example, when the color information of the vertex and the coordinates of the vertex are input to the fragment shader, the fragment shader colors the vertex based on the color information, and actually restores the color distribution of the local scene in the real space. After the vertex is colored, a final virtual representation of the real space is obtained. This virtual representation reflects various local scenes in the real space, including the contour shape and color distribution, and can be presented to a user on a terminal device.

[0047]

[0047] According to the embodiment of the present disclosure, by processing color images and depth images collected at multiple observation points in a real space, a virtual representation image of the real space can be obtained without constructing a three-dimensional model of the real space. Therefore, the amount of calculation of the terminal device is greatly reduced, and the time required to generate the virtual representation image is shortened. This is favorable for shortening the waiting time of the user and greatly improving the user experience. At the same time, the amount of original image data required is small, and the process of generating the virtual representation image is simplified, which is highly practical.

[0048]

[0048] Figure 7 is a flow chart of a method 700 for presenting a virtual representation of a real space according to another exemplary embodiment. As shown in Figure 7, the method 700 includes steps 701 to 780.

[0049]

[0049] In step 701, a plurality of groups of original color images are obtained, and the original color images of any group are color images collected from different directions at one corresponding observation point among the plurality of observation points. In one example, six color images of the real space can be collected from six directions, namely, up, down, left, right, front, and back, at any observation point. It can be understood that more / fewer color images can be obtained from more / fewer directions, for example, four, eight, or twelve color images can be collected, and the present disclosure is not limited in this respect.

[0050]

[0050] In step 702, the original color images of any group are synthesized into a corresponding single synthesized color image, which is used as the color image corresponding to the local scene in the real space observed at the corresponding observation point. In each embodiment, various image projection methods can be used to project the six original color images of any group onto the same plane to obtain one synthesized color image. According to some embodiments, the original color images of the group can be synthesized into the above single synthesized color image by Gauss-Kruger projection. Specifically, first, the color images of six directions are combined into a cubic skybox map. Then, an elliptical cylinder surface is fitted to the outer surface of the cube of the skybox map, and is tangent to the cube, and it is assumed that the central axis of the elliptical cylinder passes through the center of the cube, and then the images within a certain range of the skybox map are projected onto the elliptical cylinder surface, and the cylindrical surface is expanded to become the projection surface. In other embodiments, other suitable image projection methods can also be used to synthesize the six original color images into one original color image, which will not be listed here one by one.

[0051]

[0051] In step 703, a plurality of groups of original depth images are obtained, where the original depth images of any group are depth images collected from different directions at a corresponding observation point of one of the plurality of observation points. The operation of step 703 may be similar to that of step 701, except that it involves the use of a depth camera (or other depth capture device).

[0052]

[0052] In step 704, the original depth images of any group are synthesized into a corresponding single synthesized depth image, which is used as the depth image containing the depth information corresponding to the local scene in the real space observed at the corresponding observation point. The operation of step 704 may be the same as that of step 702, and the description is omitted here.

[0053]

[0053] In step 710, a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images are obtained. The operations in step 701 are the same as those in step 110 described above with respect to Figure 1, and the description will be omitted for brevity.

[0054]

[0054] In step 720, for any observation point, a color image among the plurality of color images corresponding to the observation point and a depth image among the plurality of depth images corresponding to the color image are superimposed to obtain a superimposed image. The operations of step 720 are the same as those of step 120 described above with reference to Figure 1, and will not be described again for the sake of brevity.

[0055]

[0055] In step 730, the superimposed images corresponding to the multiple observation points are respectively mapped onto multiple spheres in the virtual space, such that any sphere corresponds to one corresponding observation point among the multiple observation points and includes multiple vertices. The operation of step 730 is the same as that of step 130 described above with reference to FIG. 1, and the description is omitted for brevity.

[0056] Steps 741 to 742 correspond to step 140 described above with respect to FIG. 1 and are described in detail below.

[0057]

[0057] In step 741, each vertex of the multiple spheres is rotated, and the angle of rotation is based on the degree of overlap between the local scene observed at the observation point corresponding to the sphere on which the vertex is located and the local scenes observed at other observation points among the multiple observation points.

[0058]

[0058] According to some embodiments, a reference observation point can be set in advance, for example, an observation point located in the center of the real space can be set as the reference observation point. Then, the vertices of the spheres corresponding to the other observation points are rotated relative to the sphere corresponding to this reference observation point (hereinafter abbreviated as the reference sphere). In order to form a virtual image that can realize the viewpoint walk function, the rotation angle of the spheres corresponding to each observation point relative to the reference sphere should be consistent with the degree of overlap of the local scenes observed at these two observation points in the real world, thereby accurately switching the viewing angle in the walk process of these two observation points. In one example, it is determined that the sphere corresponding to observation point 1 is rotated 30° relative to the reference sphere according to the degree of overlap of the other observation point (denoted as observation point 1) and the local scene observed at the reference observation point. Then, this rotation angle can be input into the image processing software as a rotation parameter, and the image processing software generates a rotation matrix according to this rotation parameter, and applies this rotation matrix to all the vertices of the sphere corresponding to observation point 1, thereby realizing the rotation operation on the entire sphere of observation point 1. The spheres of the remaining observation points are rotated in the same manner, and a virtual space image after the rotation is completed is obtained.

[0059]

[0059] In step 742, each vertex of the multiple spheres is translated, the translation distance being based on the relative spatial position between the observation point corresponding to the sphere on which the vertex is located and other observation points of the multiple observation points.

[0060]

[0060] According to some embodiments, the spheres corresponding to all the observation points are translated relative to the reference sphere. In order to form a virtual image that can realize the viewpoint walk function, the translation distance of the sphere corresponding to any observation point relative to the reference sphere should be consistent with the spatial distance of these two observation points in the real world, thereby accurately expressing the movement distance in the walk process of these two observation points. For example, assuming that 1 m in the real world corresponds to one unit distance in the virtual space, if the observation point 1 and the reference observation point are 5 m apart in the real world, the sphere of the observation point 1 is translated by 5 unit distances relative to the reference sphere. The direction of the translation should also be consistent with the direction of the observation point 1 relative to the reference observation point in the real world. This translation distance can be input into the program of the image processing software as a translation parameter, and the image processing software generates a translation matrix according to this translation parameter, and applies this translation matrix to all vertices of the sphere corresponding to the observation point 1 to realize the translation operation on the entire sphere corresponding to the observation point 1. In a similar manner, the spheres of the remaining observation points are translated in sequence, and a virtual space image after rotation and translation is obtained, such as the image 400 shown in FIG.

[0061] It is understood that it is possible to reverse the order of execution of steps 741 and 242. In other words, if the sphere is first translated and then rotated, the virtual space image 400 shown in FIG. 4 can be obtained in the same manner.

[0062] Steps 751 through 754 correspond to step 150 described above with respect to FIG. 1 and are described in detail below.

[0063]

[0063] In step 751, depth information of an arbitrary vertex is obtained. According to some embodiments, since the vertices contain the depth information, in step 751, a vertex shader of the image processing software can directly obtain the depth information of an arbitrary vertex.

[0064] In step 752, the depth values ​​indicated in the depth information are normalized.

[0065] In step 753, the normalized depth value is multiplied by the radius of the sphere in which the vertex is located to obtain the offset distance.

[0066]

[0066] Normalization is a dimensionless processing method. The normalization process can change the absolute depth value of the depth information to a value relative to a preset depth value. For example, if the preset depth value in the real space can have a corresponding relationship with the radius of a sphere in the virtual space, the normalization process can convert the depth value of the vertex of the sphere into a ratio between it and the preset depth value, and then multiply the ratio by the radius of the sphere to obtain the offset distance to be moved.

[0067] In step 754, the coordinates of any vertex are moved along the normal direction of the vertex by an offset distance, which corresponds to the depth information of the vertex.

[0068]

[0068] By the operations of steps 752 to 754, spatial editing is sequentially performed on any vertices of the sphere of the remaining observation points, and a virtual space image after spatial editing is completed, such as the image 500 shown in FIG. 5, is obtained.

[0069]

[0069] In step 760, for any vertex of any sphere, a corresponding virtual representation in the virtual space of the corresponding local scene in the real space is obtained and presented by performing a coloring process on the vertex based on the color information of the vertex. The operation of step 760 is the same as that of step 160 described above with reference to Figure 1, and the description will be omitted for brevity.

[0070]

[0070] In step 770, a first one of the corresponding virtual representations is presented in the view, the first virtual representation corresponding to a current observation point of the plurality of observation points.

[0071]

[0071] According to some embodiments, a user can launch a virtual reality application pre-installed on a terminal device and view a virtual representation image of a real space. In one example, the virtual reality application can be an application for viewing an interior scene of a house source using VR technology, for example, checking a house source list on the Internet. After entering the application, a house source list can be first presented to the user for selection. After the user selects a house source he or she wants to view, a corresponding virtual representation can be presented in a view window for the user to view. Also, an observation point can be pre-selected or automatically selected, and when viewed from the selected observation point, the view window presents the corresponding virtual representation as the current observation point. FIG. 8 illustrates an exemplary view 800 presented by a view window during a period when a user views a virtual representation image. A third observation point (not shown in FIG. 8) is the current observation point. The user can realize switching between observation points, such as switching to a first observation point OB1 or a second observation point OB2, by operating on an input device (e.g., a mouse, a keyboard, etc.) of the terminal device, thereby viewing a local scene of a real space from different viewing angles.

[0072]

[0072] In step 780, in response to detecting a user action indicating movement from the current observation point to another one of the plurality of observation points, the view is updated to present a second virtual representation of the corresponding virtual representations, the second virtual representation corresponding to the other observation point.

[0073]

[0073] In the process of viewing, the user can realize walking between different observation points by operating on the input device. In one example, as shown in FIG. 8, when the user clicks the first observation point OB1 or the second observation point OB2 in the view 800, the view window presents a virtual representative image corresponding to the first observation point OB1 or the second observation point OB2. As shown in FIG. 9, when the viewing angle is switched to the first observation point OB1, the view window updates the view and displays a virtual representation image 900 corresponding to the first observation point OB1. As shown in FIG. 10, when the viewing angle is switched to the second observation point OB2, the view window updates the view and displays a virtual representation image 1000 corresponding to the second observation point OB2. At the second observation point OB2, the viewing angle can be further switched to the fourth observation point OB4 or the fifth observation point OB5, as shown in FIG. 10. Through the above operation, the user can switch the viewing angle between all the observation points, thereby achieving the simulation effect of the user walking in the entire real space. In some embodiments, the process of updating a view in a view window can perform a gradient process on the two views, one before the other, to make the viewing angle switching process more natural.

[0074]

[0074] Although operations are depicted in a particular order in the figures, this should not be understood as requiring that these operations be performed in the particular order shown, or sequentially, nor should it be understood that all of the operations shown are required to achieve desirable results.

[0075]

[0075] Fig. 11 is a structural block diagram of an apparatus 1100 for presenting a virtual representation of a real space according to an exemplary embodiment. As shown in Fig. 11, the apparatus 1100 includes an image acquisition unit 1110, an image overlay unit 1120, a mapping unit 1130, a spatial transformation unit 1140, a vertex editing unit 1150 and a coloring unit 1160.

[0076]

[0076] The image acquisition unit 1110 is configured to acquire a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, where the plurality of color images correspond to corresponding local scenes in a real space observed at a plurality of observation points in the real space, and the plurality of depth images each include depth information of the corresponding local scenes.

[0077]

[0077] The image overlay unit 1120 is configured to overlay, for any observation point, a color image corresponding to the observation point among the multiple color images and a depth image corresponding to the observation point among the multiple depth images to obtain a overlay image, which includes multiple pixel points, and for any pixel point, color information of the corresponding pixel point in the color image and depth information of the corresponding pixel point in the depth image.

[0078]

[0078] The mapping unit 1130 is configured to map the superimposed images corresponding to the multiple observation points onto multiple spheres in the virtual space, respectively, so that any sphere corresponds to one corresponding observation point among the multiple observation points and includes multiple vertices, and any vertex has corresponding color information and corresponding depth information.

[0079]

[0079] The spatial transformation unit 1140 is configured to perform spatial transformation in the virtual space for each vertex of the multiple spheres based on the relative spatial relationships between the multiple observation points in the real space.

[0080]

[0080] Vertex edit unit 1150 is configured to perform spatial edits on any vertex of any sphere based on the depth information of that vertex.

[0081]

[0081] The coloring unit 1160 is configured to obtain and present, for any vertex of any sphere, a corresponding virtual representation in the virtual space of the corresponding local scene in the real space by performing a coloring process on the vertex based on the color information of the vertex.

[0082]

[0082] The vertex editing unit 1150 is further configured to move the coordinates of the vertex to be edited by an offset distance along the normal direction of the vertex, where the offset distance corresponds to the depth information of the vertex.

[0083]

[0083] In some embodiments, the vertex editing unit 1150 is further configured to obtain depth information for the vertex to be edited, normalize the depth value indicated in the depth information, and multiply the normalized depth value by the radius of the sphere in which the vertex is located to obtain the offset distance.

[0084]

[0084] In some embodiments, the spatial transformation unit 1140 is further configured to, for any vertex of any sphere, perform a spatial transformation on the coordinates of that vertex using a spatial transformation matrix. The spatial transformation may include scaling, rotation and / or translation.

[0085]

[0085] In some embodiments, the spatial transformation unit 1140 may include a rotation unit 1141 and a translation unit 1142. The rotation unit 1141 is configured to rotate the coordinates of the vertex using a rotation matrix, where the rotation angle is based on the degree of overlap between the local scene observed at the observation point corresponding to the sphere on which the vertex is located and the local scene observed at the other observation points of the multiple observation points. The translation unit 1142 is configured to translate the coordinates of the vertex using a translation matrix, where the translation distance is based on the relative spatial positions between the observation point corresponding to the sphere on which the vertex is located and the other observation points of the multiple observation points.

[0086]

[0086] In some embodiments, the coloring unit 1160 is further configured to input the vertex color information and the coordinates of the vertex to a fragment shader for coloring processing.

[0087]

[0087] In some embodiments, the device 1110 may further include a view presenting unit 1170 configured to present a first virtual representation of the corresponding virtual representations in the view, the first virtual representation corresponding to a current observation point of the multiple observation points.

[0088]

[0088] In some embodiments, the view presenting unit 1170 is further configured to update the view to present a second virtual representation of the corresponding virtual representations in response to detecting a user action indicating movement from a current observation point to another observation point of the plurality of observation points, the second virtual representation corresponding to the other observation point.

[0089]

[0089] In some embodiments, the image acquisition unit 1110 is further configured to receive a plurality of color images and a plurality of depth images from a server.

[0090]

[0090] In some embodiments, the apparatus 1100 may further include an image synthesis unit 1180 configured to acquire multiple groups of original color images, where the original color images of any group are color images collected from different directions at a corresponding observation point among the multiple observation points, and to synthesize the original color images of any group into a corresponding single synthetic color image to use as the color image corresponding to the local scene in real space observed at the corresponding observation point.

[0091]

[0091] In some embodiments, the image synthesis unit 1180 is further configured to acquire multiple groups of original depth images, where the original depth images of any group are depth images collected from different directions at a corresponding observation point among the multiple observation points, and to synthesize the original depth images of any group into a corresponding single synthesized depth image to use as the depth image containing depth information corresponding to the local scene in real space observed at the corresponding observation point.

[0092]

[0092] It should be understood that each unit of the apparatus 1100 shown in Fig. 11 may correspond to each step of the method 700 described with reference to Fig. 7. Thus, the operations, features and advantages described above for the method 700 apply equally to the apparatus 1100 and the units contained therein. For the sake of brevity, some operations, features and advantages will not be described here.

[0093]

[0093] Although specific functions have been described above with reference to specific units, it should be noted that the functionality of each unit described herein may be divided into multiple units, and / or at least some of the functionality of multiple units may be combined into a single unit. A specific unit described herein performing an operation includes the specific unit itself performing the operation, or the specific unit instead calling or otherwise accessing another component or unit that performs the operation (or cooperates with the specific unit to perform the operation). Thus, a specific unit performing an operation may include the specific unit itself performing the operation, and / or another unit performing the operation that is called by, or accessed in another manner by, the specific unit.

[0094]

[0094] It should also be understood that various techniques may be described herein in the general context of software and hardware elements or program modules. Each unit described above with respect to FIG. 11 may be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units may be implemented as computer program codes / instructions configured to be executed by one or more processors or stored in a computer-readable storage medium. Alternatively, these units may be implemented as hardware logic / circuitry. For example, in some embodiments, one or more of the units described above with respect to FIG. 11 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (including a processor (e.g., including a Central Processing Unit (CPU), microcontroller, microprocessor, Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components in other circuits) and may include embedded firmware to selectably execute received program code and / or perform functions.

[0095] According to one aspect of the present disclosure, there is provided a computer apparatus including a memory, a processor, and a computer program stored in the memory, the processor configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0096]

[0096] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is provided having a computer program stored thereon, the computer program implementing the steps of any of the method embodiments described above when executed by a processor.

[0097]

[0097] According to one aspect of the present disclosure, there is provided a computer program product comprising a computer program, the computer program implementing the steps of any of the method embodiments described above when executed by a processor.

[0098]

[0098] Examples of such a computer device, a non-transitory computer-readable storage medium and a computer program product are described below with reference to FIG.

[0099] FIG. 12 is a structural block diagram of an exemplary electronic device 1200 that may be used to implement embodiments of the present disclosure. The electronic device 1200 may be an example of a hardware device applicable to each aspect of the present disclosure. The term electronic device is intended to represent various forms of digital electronic computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0100] 12, the device 1200 includes a computing unit 1201 that can perform various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 1202 or loaded from a storage unit 1208 into a random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of the device 1200. The processing unit 1201, the ROM 1202 and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0101]

[0101] A number of components in the device 1200, including an input unit 1206, an output unit 1207, a storage unit 1208 and a communication unit 1209, are connected to the I / O interface 1205. The input unit 1206 may be any type of device capable of inputting information to the device 1200, the input unit 1206 may receive input numeric or character information and generate key signal inputs for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone and / or a remote controller. The output unit 1207 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 1208 may include, but is not limited to, a magnetic disk, an optical disk. The communications unit 1209 allows the device 1200 to exchange information / data with other devices over a computer network such as the Internet and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communications device, a wireless communications transceiver and / or chipset, such as a Bluetooth™ device, a 1302.11 device, a WiFi device, a WiMax device, a cellular communications device, and / or the like.

[0102]

[0102] The processing unit 1201 may be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processing unit 1201 executes each of the methods and processes described above, such as the method for presenting a virtual representation for presenting a virtual representation of a real space. For example, in some embodiments, the method for presenting a virtual representation of a real space may be implemented as a computer software program tangibly contained in a machine-readable medium, such as the storage unit 1208. In some embodiments, some or all of the computer program may be loaded and / or attached to the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the above-described method for presenting a virtual representation of a real space may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured in any other suitable manner (e.g., by firmware) to perform a method for presenting a virtual representation of a real space.

[0103]

[0103] Various embodiments of the systems and techniques described herein above may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that may be executed and / or interpreted in a programmable system that includes at least one programmable processor, which may be an application specific or general purpose programmable processor, and that may receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0104]

[0104] The program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when the program codes are executed by the processor or controller, the functions / operations defined in the flowcharts and / or block diagrams are performed. The program codes may be entirely or partially executed on a machine, partly executed on a machine and partly executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0105]

[0105] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program that can be used by, or in combination with, an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above contents. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CDs), optical storage devices, magnetic storage devices, or any suitable combination of the above contents.

[0106]

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, a keyboard, and a pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices may be used to provide interaction with a user. For example, feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and may receive input from the user in any form (including acoustic, speech, and tactile input).

[0107]

[0107] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0108]

[0108] A computer system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is created by computer programs running on the corresponding computers and having a client-server relationship to each other.

[0109]

[0109] It should be understood that steps can be rearranged, added, or deleted using the various forms of flow shown above. For example, each step described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this disclosure are achieved, but this specification is not limited thereto.

[0110]

[0110] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, the above-mentioned methods, systems, and devices are merely illustrative embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but only by the scope of the allowed claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by equivalent elements. It should be noted that these steps may be performed in a different order than that described in this disclosure. Furthermore, the various elements in the embodiments or examples may be combined in various ways. It is important to note that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements appearing later in this disclosure. [Item of invention] [Item 1] 1. A method for presenting a virtual representation of a real space, comprising: acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, the plurality of color images corresponding to corresponding local scenes in the real space observed at a plurality of observation points in the real space, the plurality of depth images each including depth information of the corresponding local scenes; for any one of the observation points, superimposing a color image corresponding to the observation point among the plurality of color images and a depth image corresponding to the observation point among the plurality of depth images to obtain a superimposed image; Mapping the superimposed images corresponding to the plurality of observation points onto a plurality of spheres in a virtual space, respectively, such that any of the spheres corresponds to one corresponding observation point among the plurality of observation points and includes a plurality of vertices, and any of the vertices has corresponding color information and corresponding depth information; performing a spatial transformation in a virtual space on each vertex of the plurality of spheres based on a relative spatial relationship between the plurality of observation points in the real space; performing spatial editing on any of the vertices of any of the spheres based on the depth information of the vertex; and for any vertex of any of the spheres, obtaining and presenting a corresponding virtual representation in the virtual space of a corresponding local scene in the real space by performing a coloring process on the vertex based on color information of the vertex. [Item 2] The above step of performing spatial editing on the vertex based on the depth information of the vertex is 2. The method of claim 1, further comprising the step of moving the coordinates of the vertex by an offset distance along a normal direction of the vertex, the offset distance corresponding to depth information of the vertex. [Item 3] The above step of performing spatial editing on the vertex based on the depth information of the vertex is Before moving the coordinates of that vertex, obtaining depth information for the vertices; normalizing the depth values ​​indicated in the depth information; 3. The method of claim 2, further comprising the step of multiplying the normalized depth value by the radius of the sphere in which the vertex is located to obtain the offset distance. [Item 4] The step of performing spatial transformation in the virtual space for each vertex of the plurality of spheres based on a relative spatial relationship between the plurality of observation points in the real space includes: For any said vertex of any said sphere, 2. The method according to claim 1, comprising a step of performing a spatial transformation on the coordinates of the vertices using a spatial transformation matrix, the spatial transformation including at least one selected from the group consisting of scaling, rotation and translation. [Item 5] 5. The method according to item 4, wherein the spatial transformation includes a rotation, and the angle of the rotation is based on the degree of overlap between the local scene observed at an observation point corresponding to a sphere on which the vertex is located and local scenes observed at other observation points among the plurality of observation points. [Item 6] 5. The method according to claim 4, wherein the spatial transformation includes a translation, and the distance of the translation is based on a relative spatial position between an observation point corresponding to a sphere on which the vertex is located and other observation points among the plurality of observation points. [Item 7] The above step of performing coloring processing on the vertices based on the color information of the vertices is The method according to item 1, further comprising the step of inputting the color information of the vertex and the coordinates of the vertex to a fragment shader to perform the coloring process. [Item 8] Item 14. The method of item 1, further comprising the step of presenting in a view a first one of the corresponding virtual representations, the first virtual representation corresponding to a current observation point of the plurality of observation points. [Item 9] 9. The method of claim 8, further comprising the step of updating the view to present a second virtual representation of the corresponding virtual representation in response to detecting a user action indicating movement from the current observation point to another observation point of the plurality of observation points, the second virtual representation corresponding to the other observation point. [Item 10] 2. The method according to item 1, wherein the step of performing spatial editing on the vertices based on the depth information of the vertices is performed before the step of performing spatial transformation in the virtual space for each vertex of the plurality of spheres. [Item 11] 2. The method of claim 1, wherein the plurality of spheres have the same radius. [Item 12] The step of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, 12. The method according to any one of claims 1 to 11, comprising receiving the plurality of color images and the plurality of depth images from a server. [Item 13] Prior to the step of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, acquiring a plurality of groups of original color images, each of the original color images of any group being color images collected from different directions at a corresponding observation point of one of the plurality of observation points; The method according to any one of items 1 to 11, further comprising a step of combining any group of original color images into a corresponding single combined color image, and using the combined color image as a color image corresponding to a local scene in the real space observed at the corresponding observation point. [Item 14] 14. The method according to item 13, wherein the original color images of any group include six color images of the real space collected from six directions, respectively, above, below, left, right, front, and back, at the corresponding observation point. [Item 15] The above steps of combining any group of original color images into a corresponding single composite color image include: 15. The method according to claim 14, comprising a step of combining the original color images of the group into the combined color image by Gauss-Kruger projection. [Item 16] Prior to the step of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, acquiring a plurality of groups of original depth images, each of the original depth images of any group being depth images collected from different directions at a corresponding observation point of one of the plurality of observation points; Item 14. The method of item 13, further comprising the step of: synthesizing any group of original depth images into a corresponding single synthesized depth image, which is used as a depth image containing depth information corresponding to a local scene in the real space observed at the corresponding observation point. [Item 17] 1. An apparatus for presenting a virtual representation of a real space, comprising: an image acquisition unit configured to acquire a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, the plurality of color images corresponding to corresponding local scenes in a real space observed at a plurality of observation points in the real space, the plurality of depth images each including depth information of the corresponding local scenes; an image overlay unit configured to overlay, for any observation point, a color image corresponding to the observation point among the plurality of color images and a depth image corresponding to the observation point among the plurality of depth images to obtain a overlay image; a mapping unit configured to map the superimposed images corresponding to the plurality of observation points onto a plurality of spheres in the virtual space, respectively, such that any sphere corresponds to one corresponding observation point among the plurality of observation points, includes a plurality of vertices, and any vertex has corresponding color information and corresponding depth information; a spatial transformation unit configured to perform spatial transformation in the virtual space for each vertex of the plurality of spheres based on a relative spatial relationship between the plurality of observation points in the real space; a vertex editing unit configured to perform spatial editing on any vertex of any sphere based on the depth information of the vertex; a coloring unit configured to perform a coloring process on any vertex of any sphere based on color information of the vertex, thereby obtaining and presenting a corresponding virtual representation in the virtual space of the corresponding local scene in the real space; 16. An apparatus for presenting a virtual representation of a real space comprising: [Item 18] The vertex editing unit is further configured to move the coordinates of the vertex by an offset distance along a normal direction of the vertex, the offset distance corresponding to depth information of the vertex. Item 18. The apparatus according to item 17. [Item 19] The vertex editing unit further comprises: Get the depth information of the vertex waiting to be edited, normalizing the depth values ​​indicated in the depth information; configured to multiply the normalized depth value by the radius of the sphere in which the vertex is located to obtain the offset distance. Item 19. The apparatus according to item 18. [Item 20] The spatial transformation unit further comprises: For any of the vertices of any of the spheres, a spatial transformation is performed on the coordinates of the vertex using a spatial transformation matrix, the spatial transformation being configured to include at least one selected from the group consisting of scaling, rotation, and translation. Item 18. The apparatus according to item 17. [Item 21] The spatial transformation unit includes a rotation unit; The rotation unit is configured to rotate the coordinates of the vertex using a rotation matrix, and the angle of the rotation is based on the degree of overlap between the local scene observed at an observation point corresponding to the sphere on which the vertex is located and the local scene observed at another observation point among the plurality of observation points. 21. The apparatus according to item 20. [Item 22] the spatial transformation unit includes a translation unit; The translation unit is configured to perform a translation on the coordinates of the vertex using a translation matrix, and the distance of the translation is based on a relative spatial position between an observation point corresponding to a sphere on which the vertex is located and other observation points of the plurality of observation points. 21. The apparatus according to item 20. [Item 23] The coloring unit is further configured to input color information of the vertices and coordinates of the vertices to a fragment shader to perform the coloring process. Item 18. The apparatus according to item 17. [Item 24] Further included is a view presentation unit; The view presenting unit is configured to present in a view a first one of the corresponding virtual representations, the first virtual representation corresponding to a current observation point of the plurality of observation points. Item 18. The apparatus according to item 17. [Item 25] The view presenting unit further comprises: updating the view to present a second virtual representation of the corresponding virtual representations in response to detecting a user action indicating movement from the current observation point to another one of the plurality of observation points, the second virtual representation being configured to correspond to the other observation point. 25. The apparatus according to item 24. [Item 26] The image acquisition unit is further configured to receive the plurality of color images and the plurality of depth images from a server. The device according to any one of items 17 to 25. [Item 27] Further included is an image synthesis unit; The image synthesis unit includes: Acquire a plurality of groups of original color images, and the original color images of any group are color images collected from different directions at a corresponding observation point among the plurality of observation points; configured to combine any group of original color images into a corresponding single composite color image to serve as a color image corresponding to a local scene in said real space as observed at said corresponding observation point. The device according to any one of items 17 to 25. [Item 28] The image synthesis unit further comprises: Obtaining a plurality of groups of original depth images, each of the original depth images of any group being depth images collected from different directions at a corresponding observation point among the plurality of observation points; configured to synthesize any group of original depth images into a corresponding single synthesized depth image and use it as a depth image containing depth information corresponding to a local scene in the real space observed at the corresponding observation point. 28. The apparatus according to item 27. [Item 29] A computing device comprising: a memory; a processor; and a computer program stored in the memory; Here, the processor is a computer device configured to execute the computer program to implement the steps of the method according to any one of items 1 to 16. [Item 30] A non-transitory computer-readable storage medium having a computer program stored therein, the computer program implementing the steps of the method according to any one of items 1 to 16 when executed by a processor. [Item 31] A computer program product including a computer program, the computer program product implementing the steps of the method according to any one of items 1 to 16 when the computer program is executed by a processor.

Claims

1. 1. A method for presenting a virtual representation of a real space, comprising: acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, the plurality of color images corresponding to corresponding local scenes in the real space observed at a plurality of observation points in the real space, the plurality of depth images each including depth information of the corresponding local scenes; for any one of the observation points, superimposing a color image corresponding to the observation point among the plurality of color images and a depth image corresponding to the observation point among the plurality of depth images to obtain a superimposed image; a step of mapping the superimposed images corresponding to the plurality of observation points onto a plurality of spheres in a virtual space, each of the spheres corresponding to one of the plurality of observation points and including a plurality of vertices, each of the vertices having corresponding color information and corresponding depth information; performing a spatial transformation in a virtual space on each vertex of the plurality of spheres based on a relative spatial relationship between the plurality of observation points in the real space; performing spatial editing on any of the vertices of any of the spheres based on the depth information of the vertex; performing a coloring process on any of the vertices of any of the spheres based on color information of the vertices, thereby obtaining and presenting a corresponding virtual representation in the virtual space of a corresponding local scene in the real space; 16. A method for presenting a virtual representation of a real space, comprising:

2. The above step of performing spatial editing on the vertex based on the depth information of the vertex is The method of claim 1 , comprising: shifting coordinates of the vertex by an offset distance along a normal direction of the vertex, the offset distance corresponding to depth information of the vertex.

3. The above step of performing spatial editing on the vertex based on the depth information of the vertex is Before moving the coordinates of that vertex, obtaining depth information for the vertices; normalizing the depth values ​​indicated in the depth information; multiplying the normalized depth value by the radius of the sphere in which the vertex is located to obtain the offset distance; The method of claim 2 , further comprising:

4. The step of performing spatial transformation in the virtual space for each vertex of the plurality of spheres based on a relative spatial relationship between the plurality of observation points in the real space includes: For any said vertex of any said sphere, performing a spatial transformation on the coordinates of the vertices using a spatial transformation matrix, the spatial transformation including at least one selected from the group consisting of scaling, rotation, and translation. The method of claim 1 , comprising:

5. 5. The method of claim 4, wherein the spatial transformation includes a rotation, the angle of the rotation being based on the degree of overlap of the local scene observed at an observation point corresponding to a sphere on which the vertex is located with local scenes observed at other observation points among the plurality of observation points.

6. 5. The method of claim 4, wherein the spatial transformation includes a translation, the distance of the translation being based on a relative spatial position between an observation point corresponding to a sphere on which the vertex is located and other observation points of the plurality of observation points.

7. The above step of performing coloring processing on the vertices based on the color information of the vertices is The method of claim 1 , further comprising the step of inputting the color information of the vertices and the coordinates of the vertices to a fragment shader to perform the coloring process.

8. 2. The method of claim 1, further comprising presenting in a view a first one of the corresponding virtual representations, the first virtual representation corresponding to a current observation point of the plurality of observation points.

9. 10. The method of claim 8, further comprising: updating the view to present a second virtual representation of the corresponding virtual representation in response to detecting a user action indicating movement from the current observation point to another observation point of the plurality of observation points, the second virtual representation corresponding to the other observation point.

10. The method of claim 1 , wherein the step of performing spatial editing on the vertices based on depth information of the vertices is performed before the step of performing a spatial transformation in the virtual space for each vertex of the plurality of spheres.

11. The method of claim 1 , wherein the spheres have the same radius.

12. The step of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, The method of claim 1 , comprising receiving the plurality of color images and the plurality of depth images from a server.

13. Prior to the step of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, acquiring a plurality of groups of original color images, each of the original color images of any group being color images collected from different directions at a corresponding observation point of one of the plurality of observation points; combining any group of original color images into a corresponding single composite color image, which serves as a color image corresponding to the local scene in the real space observed at the corresponding observation point; The method of claim 1 further comprising:

14. The method of claim 13 , wherein any group of original color images includes six color images of the real space collected from six directions, respectively: above, below, left, right, front, and back, at the corresponding observation point.

15. The above steps of combining any group of original color images into a corresponding single composite color image include:

15. The method of claim 14, further comprising the step of combining the original color images of the group into the composite color image by Gauss-Kruger projection.

16. Prior to the step of acquiring a plurality of color images and a plurality of depth images respectively corresponding to the plurality of color images, acquiring a plurality of groups of original depth images, where the original depth images of any group are depth images collected from different directions at a corresponding observation point of one of the plurality of observation points; synthesizing the original depth images of any group into a corresponding single synthesized depth image, which serves as a depth image containing depth information corresponding to a local scene in the real space observed at the corresponding observation point; The method of claim 13 further comprising:

17. A computing device comprising: a memory; a processor; and a computer program stored in the memory; A computing device, wherein said processor is configured to execute said computer program to implement the steps of the method according to any one of claims 1 to 16.

18. A non-transitory computer readable storage medium having a computer program stored thereon, the computer program implementing the steps of the method according to any one of claims 1 to 16 when executed by a processor.

19. A computer program comprising instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 16.

Images