Method, device and electronic device for panoramic scene switching

Abstract

The present disclosure provides a panoramic scene switching method, device and electronic equipment, relates to the technical field of artificial intelligence, specifically to the technical field of augmented reality, virtual reality, computer vision, deep learning and the like, and can be applied to scenarios such as metaverse and smart city. The specific scheme is: a first depth map of a current scene and a second depth map of a target scene are generated; coordinates of vertices in a first panoramic sphere model are determined according to the first depth map, and coordinates of vertices in the second depth map are determined according to the second depth map; a viewpoint and a corresponding visible area are obtained to determine a target vertex, and the first panoramic sphere model and the second panoramic sphere model are rendered; the viewpoint is moved to a target direction, and states of the first panoramic sphere model and the second panoramic sphere model are adjusted. According to the present disclosure, the panoramic sphere model is rendered according to depth information, so as to realize the effect of switching from the current scene to the target scene. According to the present disclosure, the switching process can be made more real according to the depth information, and the display effect is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically augmented reality, virtual reality, computer vision, deep learning, and other technical fields, and can be applied to scenarios such as metaverse and smart cities. Background Technology

[0002] Panoramic virtual display systems are increasingly being used in various fields due to their advantages such as low hardware requirements and high realism. Panoramic technology is a type of virtual reality technology that can simulate the visual experience of a user being in a specific location in a real scene, providing a strong sense of immersion and a truly immersive user experience.

[0003] In related technologies, during the process of switching from a panoramic image corresponding to one scene to a panoramic image corresponding to another scene, all areas in the user's field of vision deform simultaneously and to the same degree, which does not conform to the laws of movement in the real world and results in poor display effects. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and electronic device for panoramic scene switching.

[0005] According to a first aspect of this disclosure, a method for panoramic scene switching is provided, comprising:

[0006] The depth information of each pixel in the panoramic image data of the current scene and the target scene is obtained to generate a first depth map corresponding to the current scene and a second depth map corresponding to the target scene. The panoramic image data contains multiple tile images.

[0007] Create a first panoramic sphere model corresponding to the current scene and a second panoramic sphere model corresponding to the target scene. Determine the coordinates of the vertices in the first panoramic sphere model based on the first depth map, and determine the coordinates of the vertices in the second panoramic sphere model based on the second depth map.

[0008] Obtain the viewpoint and the corresponding visible area, and determine the vertices of the first panoramic sphere model and the second panoramic sphere model within the visible area as target vertices. Render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the target vertices.

[0009] The viewpoint is moved toward the target direction, and the states of the first panoramic sphere model and the second panoramic sphere model are adjusted to display the target scene.

[0010] Optionally, determining the coordinates of vertices in the first panoramic sphere model based on the first depth map, and determining the coordinates of vertices in the second panoramic sphere model based on the second depth map, includes:

[0011] Obtain the first mapping point of the vertex in the first panoramic sphere model corresponding to the first depth map, and adjust the coordinates of the vertex according to the depth information of the first mapping point;

[0012] Obtain the second mapping point of the vertex in the second panoramic sphere model corresponding to the second depth map, and adjust the coordinates of the vertex according to the depth information of the second mapping point.

[0013] Optionally, obtaining the viewpoint and the corresponding visible area includes:

[0014] Determine the coordinates and visible direction of the viewpoint, and draw a ray along the visible direction starting from the viewpoint;

[0015] The visible area is defined as the region where the ray intersects with the first panoramic sphere model or the second panoramic sphere model.

[0016] Optionally, rendering the first panoramic sphere model and the second panoramic sphere model based on the tile map corresponding to the target vertex includes:

[0017] Load the tile map into the cache space, wherein the tile map includes a first tile map and a second tile map, and the resolution of the first tile map is smaller than that of the second tile map;

[0018] Obtain the vertex coordinates of the first and second tile maps;

[0019] The target tile map vertex is determined from the tile map vertices based on the target vertex;

[0020] Render the first panoramic sphere model and the second panoramic sphere model based on the tile maps corresponding to the vertices of the target tile map.

[0021] Optionally, determining the target tile map vertex from the tile map vertices based on the target vertex includes:

[0022] Obtain the mapping table between the target vertex and the tile graph vertices, and determine the tile graph vertex corresponding to the target vertex as the target tile graph vertex according to the mapping table.

[0023] Optionally, rendering the first panoramic sphere model and the second panoramic sphere model based on the tile maps corresponding to the vertices of the target tile map includes:

[0024] In response to the cache space containing a second tile map corresponding to the target tile map vertex, the target vertex is rendered according to the second tile map;

[0025] In response to the cache space not containing a second tile map corresponding to the target tile map vertex, but containing a first tile map corresponding to the target tile map vertex, the target vertex is rendered based on the first tile map corresponding to the target tile map vertex.

[0026] Optionally, the method includes:

[0027] In response to the fact that the cache space does not contain the second tile map and the first tile map corresponding to the target tile map vertex, the first tile map is loaded into the cache space, and the target vertex is rendered according to the first tile map corresponding to the target tile map vertex.

[0028] Optionally, moving the viewpoint towards the target direction and adjusting the states of the first panoramic sphere model and the second panoramic sphere model includes:

[0029] The first panoramic sphere model is changed from an opaque state to a transparent state, while the second panoramic sphere model remains opaque. A switching animation is generated based on the rendered images corresponding to the first and second panoramic sphere models within the viewpoint.

[0030] According to a second aspect of this disclosure, a device for panoramic scene switching is provided, comprising:

[0031] The depth acquisition module is used to acquire the depth information of each pixel in the panoramic image data of the current scene and the target scene, so as to generate a first depth map corresponding to the current scene and a second depth map corresponding to the target scene, wherein the panoramic image data contains multiple tile images;

[0032] The model creation module is used to create a first panoramic sphere model corresponding to the current scene and a second panoramic sphere model corresponding to the target scene. It determines the coordinates of the vertices in the first panoramic sphere model based on the first depth map and the coordinates of the vertices in the second panoramic sphere model based on the second depth map.

[0033] The rendering module is used to acquire the viewpoint and the corresponding visible area, and to determine the vertices of the first panoramic sphere model and the second panoramic sphere model within the visible area as target vertices, and to render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the target vertices.

[0034] The switching module is used to move the viewpoint toward the target direction and adjust the state of the first panoramic sphere model and the second panoramic sphere model to display the target scene.

[0035] Optionally, the model creation module includes:

[0036] The first coordinate adjustment submodule is used to obtain the first mapping point corresponding to the vertex in the first panoramic sphere model in the first depth map, and adjust the coordinates of the vertex according to the depth information of the first mapping point;

[0037] The second coordinate adjustment submodule is used to obtain the second mapping point corresponding to the vertex in the second panoramic sphere model in the second depth map, and adjust the coordinates of the vertex according to the depth information of the second mapping point.

[0038] Optionally, the rendering module includes:

[0039] The visual method determination submodule is used to determine the coordinates and visual direction of the viewpoint, and to draw a ray along the visual direction starting from the viewpoint;

[0040] The visible area determination submodule is used to determine the area where the ray intersects with the first panoramic sphere model or the second panoramic sphere model as the visible area.

[0041] Optionally, the rendering module includes:

[0042] A tile image loading submodule is used to load the tile image into a cache space, wherein the tile image includes a first tile image and a second tile image, and the resolution of the first tile image is smaller than that of the second tile image;

[0043] The tile map vertex acquisition submodule is used to acquire the tile map vertex coordinates of the first tile map and the second tile map;

[0044] The target vertex determination submodule is used to determine the target tile map vertex from the tile map vertices based on the target vertex;

[0045] The rendering submodule is used to render the first panoramic sphere model and the second panoramic sphere model based on the tile map corresponding to the vertex of the target tile map.

[0046] Optionally, the target vertex determination submodule includes:

[0047] The target vertex determination unit is used to obtain a mapping table between the target vertex and the tile graph vertices, and determine the tile graph vertex corresponding to the target vertex as the target tile graph vertex according to the mapping table.

[0048] Optionally, the rendering submodule includes:

[0049] The first rendering unit is configured to render the target vertex according to the second tile map in response to the cache space containing a second tile map corresponding to the target tile map vertex;

[0050] The second rendering unit is configured to render the target vertex based on the first tile map corresponding to the target tile map vertex in response to the cache space not containing a second tile map corresponding to the target tile map vertex and containing a first tile map corresponding to the target tile map vertex.

[0051] Optionally, the device further includes:

[0052] The third rendering unit, in response to the fact that the cache space does not contain the second tile map and the first tile map corresponding to the vertex of the target tile map, loads the first tile map into the cache space and renders the target vertex according to the first tile map corresponding to the vertex of the target tile map.

[0053] Optionally, the switching module includes:

[0054] The state adjustment module is used to change the first panoramic sphere model from an opaque state to a transparent state, while keeping the second panoramic sphere model in an opaque state, and to generate a switching animation based on the rendered images corresponding to the first and second panoramic sphere models within the viewpoint.

[0055] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0056] At least one processor; and

[0057] A memory communicatively connected to the at least one processor; wherein,

[0058] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of the first aspects above.

[0059] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any one of the first aspects above.

[0060] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any one of the first aspects described above.

[0061] The disclosure of this content can achieve the following beneficial effects:

[0062] By obtaining a depth map from the panoramic image data and adjusting the vertex coordinates in the panoramic sphere model based on the depth information in the depth map, the panoramic sphere model rendered from the tile image becomes visually more realistic. Using a panoramic sphere model with added depth information results in a more realistic visual effect during panoramic scene transitions, improving the user experience.

[0063] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0064] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0065] Figure 1 This is a flowchart illustrating a panoramic scene switching method provided according to an embodiment of the present disclosure;

[0066] Figure 2 This is a flowchart illustrating a panoramic scene switching method provided according to an embodiment of the present disclosure;

[0067] Figure 3 This is a schematic diagram of a visible area provided according to an embodiment of the present disclosure.

[0068] Figure 4 This is a flowchart illustrating a panoramic scene switching method provided according to an embodiment of the present disclosure;

[0069] Figure 5 This is a schematic diagram of a panoramic sphere rendering process provided according to an embodiment of the present disclosure;

[0070] Figure 6 This is a schematic diagram of the structure of a panoramic scene switching device provided according to an embodiment of the present disclosure;

[0071] Figure 7 This is a block diagram of an electronic device used to implement the panoramic scene switching method of the embodiments of this disclosure. Detailed Implementation

[0072] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0073] In related technologies, users can select a coordinate on a map on their terminal device and view a panoramic view of the scene at that coordinate. They can also switch scenes by selecting a specific direction, i.e., switching from the panoramic view of the current scene to the panoramic view of the next scene along a specific direction. The current implementation involves creating panoramic sphere models of the current scene and the next scene. Based on the correspondence between the panoramic image and the panoramic sphere model, the panoramic image is mapped onto the panoramic sphere model. The panoramic image texture within the viewpoint is then used for rendering the panoramic sphere model. A virtual camera is moved within the panoramic sphere model; the virtual camera simulates the human eye, and the animation captured by the virtual camera allows the user to experience the sensation of moving between two scenes.

[0074] However, the panoramic sphere model in related technologies is a unit sphere, which means that the deformation of objects in the animation captured during the movement of the virtual camera is similar or even the same. In real-world scenarios, when a person moves from one scene to another, the changes in objects in the sky seen by the human eye are relatively small, while the changes in objects on the ground are relatively large. This results in a gap between the travel effect and the human's experience in the actual scene, reducing the realism of the travel effect for the user.

[0075] This disclosure provides a method for panoramic scene switching to solve problems existing in related technologies. Figure 1 This is a flowchart illustrating a panoramic scene switching method according to an embodiment of this disclosure, as shown below. Figure 1 As shown, the method is applied to scenarios such as the metaverse and smart cities, and the method includes:

[0076] Step 101: Obtain the depth information of each pixel in the panoramic image data of the current scene and the target scene to generate a first depth map corresponding to the current scene and a second depth map corresponding to the target scene, wherein the panoramic image data contains multiple tile images.

[0077] In this embodiment, the panoramic image data used is in the form of tile maps. The panoramic image data of a scene is divided into multiple tile maps. The tile maps include two types: grid tiles and vector tiles. This application uses grid tiles, which are map tiles that are segmented from all layers in the panoramic image data and stored in raster format. When using tile maps, the tile maps can be loaded into the cache space during the rendering process. The tile maps that need to be rendered are then selected from the cache space and rendered, resulting in high rendering efficiency.

[0078] To improve the display effect of panoramic scenes, this embodiment extracts depth information from the panoramic image data and combines it with the depth information during the rendering process. A pre-trained depth extraction network is used to input the panoramic image data into the depth extraction network to obtain the depth information of each pixel in the panoramic image data. The depth extraction network can be a deep learning network model. Each pixel in the depth map contains values ​​(u, v, z), where u and v are the x and y coordinates of the pixel in a two-dimensional Cartesian coordinate system, respectively, and z is the depth value of the pixel. The depth value reflects the distance of the pixel from the camera; the larger the z value, the farther the corresponding pixel is from the camera.

[0079] Step 102: Create a first panoramic sphere model corresponding to the current scene and a second panoramic sphere model corresponding to the target scene. Determine the coordinates of the vertices in the first panoramic sphere model based on the first depth map, and determine the coordinates of the vertices in the second panoramic sphere model based on the second depth map.

[0080] In this embodiment of the disclosure, when switching from the current scene to the target scene, a panoramic sphere model corresponding to the current scene and the target scene is first created. The panoramic sphere model is used as a skeleton. The tile image is attached to the panoramic sphere model. Then, a virtual camera is used to capture images of the panoramic sphere model, thus achieving the effect of observing a panoramic image.

[0081] The spherical surface of the panoramic sphere model contains multiple vertices and faces formed by the connections between these vertices. The panoramic sphere model data includes vertex coordinates and vertex indices, with the vertex indices indicating which vertices form one of the faces. Initially, the first and second panoramic sphere models are unit spheres, meaning all vertices are equidistant from the center. This disclosure obtains the correspondence between pixels in the first depth map and vertices in the first panoramic sphere model to acquire the depth information corresponding to each vertex, and adjusts the distance from the vertex to the center based on this depth information. Similarly, it obtains the correspondence between pixels in the second depth map and vertices in the second panoramic sphere model to acquire the depth information corresponding to each vertex, and adjusts the distance from the vertex to the center based on this depth information. After processing, the first and second panoramic sphere models are no longer unit spheres.

[0082] Step 103: Obtain the viewpoint and the corresponding visible area, and determine the vertices of the first panoramic sphere model and the second panoramic sphere model within the visible area as target vertices. Render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the target vertices.

[0083] In this embodiment, after obtaining the tile map from the panoramic sphere model and panoramic image data, the panoramic sphere model can be rendered, and the tile map can be mapped onto the corresponding vertices of the panoramic image model. First, the viewpoint and the corresponding visible area are obtained; the viewpoint is used to simulate a human eye or a camera. In practical applications, the viewport size is usually different for electronic devices of different sizes. The visible area can be related to the screen of the electronic device or the size of the window displayed by the electronic device. Optionally, the electronic device can be a mobile phone, an augmented reality (AR) device, or a virtual reality (VR) device.

[0084] The vertices of the first panoramic sphere model and the second panoramic sphere model within the visible area are determined as target vertices. Once the target vertices fall into the visible area, only these target vertices need to be rendered.

[0085] In a panoramic sphere, each vertex and the tile map in the panoramic image data have a one-to-one correspondence. During rendering, the target vertex needs to be mapped with the corresponding tile map for rendering. The rendered image is then obtained and displayed on the monitor to achieve the visual effect of a panoramic image.

[0086] Step 104: Move the viewpoint toward the target direction and adjust the state of the first panoramic sphere model and the second panoramic sphere model to display the target scene.

[0087] In this embodiment of the disclosure, achieving the visual effect of panoramic scene switching requires rendering multiple images to form an animation. To visually achieve the effect of traversing from the current scene to the target scene, a target direction needs to be determined. This target direction can be preset or set by the user. The viewpoint moves along the target direction, the rendered images are played sequentially, and the states of the first and second panoramic sphere models are adjusted. The first panoramic sphere model gradually becomes transparent, and the second panoramic sphere model is finally revealed. The resulting animation naturally gives the viewer the visual effect of traversing from the current scene to the target scene.

[0088] Optional, Figure 1 Step 102 specifically includes: obtaining the first mapping point of the vertex in the first panoramic sphere model corresponding to the first depth map, and adjusting the coordinates of the vertex according to the depth information of the first mapping point;

[0089] Optional, Figure 1 Step 102 further includes: obtaining the second mapping point of the vertex in the second panoramic sphere model corresponding to the second depth map, and adjusting the coordinates of the vertex according to the depth information of the second mapping point.

[0090] In this embodiment, each vertex in the first panoramic sphere model has a unique corresponding point in the first depth map. This point is used as the first mapping point, and the coordinates of the corresponding vertex are adjusted according to the depth information in the first mapping point, that is, the distance from the corresponding vertex to the center of the first panoramic sphere model is adjusted. The larger the depth value of the first mapping point, the farther the pixel is from the camera, and the corresponding vertex in the first panoramic sphere model should also be farther from the center of the first panoramic sphere model. In this way, the visual effect of the rendered panoramic sphere model is closer to the real situation, improving the immersive experience of the panoramic display.

[0091] Similarly, in the second panoramic sphere model, each vertex has a unique corresponding point in the second depth map. This point is used as the second mapping point, and the coordinates of the corresponding vertex are adjusted according to the depth information in the second mapping point, that is, the distance from the corresponding vertex to the center of the second panoramic sphere model is adjusted. The larger the depth value of the second mapping point, the farther the pixel is from the camera, and the corresponding vertex in the second panoramic sphere model should also be farther from the center of the second panoramic sphere model. In this way, the visual effect of the rendered panoramic sphere model is closer to the real situation, improving the immersive experience of the panoramic display.

[0092] In panoramic data, objects in the sky, such as clouds, birds, and airplanes, are relatively far from the camera. Therefore, even if the camera moves a certain distance, the change in the sky scene will not be too significant. However, objects on the ground, such as streets, shops, and pedestrians, are closer to the camera, and the ground scene will change significantly after the camera moves a certain distance. Therefore, in order to more realistically simulate the effect of scene transitions, the vertices corresponding to pixels with lower depth values ​​on the ground need to be closer to the center of the panoramic sphere model, while the vertices corresponding to pixels with higher depth values ​​need to be farther away from the center of the panoramic sphere model.

[0093] Figure 2 This is a flowchart illustrating a panoramic scene switching method according to an embodiment of this disclosure, as shown below. Figure 2 As shown, the method is applied to scenarios such as the metaverse and smart cities. Figure 1 Step 103 specifically includes:

[0094] Step 201: Determine the coordinates and visible direction of the viewpoint, and draw a ray along the visible direction starting from the viewpoint;

[0095] Step 202: Determine the visible area as the region where the ray intersects with the first panoramic sphere model or the second panoramic sphere model.

[0096] In this embodiment, the viewpoint is located within the first or second panoramic sphere model. The viewpoint can also be used as a camera to capture images of the first or second panoramic sphere model. The visible direction includes multiple directions. Rays drawn from the viewpoint along the visible direction intersect the first or second panoramic sphere model at multiple points. The area formed by these intersections is the range visible from the viewpoint, i.e., the visible area.

[0097] Figure 3 This is a schematic diagram of a visible area provided according to an embodiment of the present disclosure, such as... Figure 3 As shown, after determining viewpoint 301, multiple rays are drawn along a certain visible direction. These rays form a cone, called the view cone 302. Viewpoint 301 is set within the panoramic sphere model, and the area encompassed by the view cone 302 is the visible range. The region formed by the multiple intersections of the view cone 302 and the panoramic sphere model is the visible area 303. Since the image seen by the viewpoint is a planar image, during rendering, the panoramic sphere model within the visible area needs to be projected onto a two-dimensional plane, namely the view plane 304. The image on the view plane 304 is then presented to the user.

[0098] Figure 4 This is a flowchart illustrating a panoramic scene switching method according to an embodiment of this disclosure, as shown below. Figure 4 As shown, the method is applied to scenarios such as the metaverse and smart cities. Figure 1 Step 103 specifically includes:

[0099] Step 401: Load the tile map into the cache space, wherein the tile map includes a first tile map and a second tile map, and the resolution of the first tile map is smaller than that of the second tile map;

[0100] In this embodiment, the tile image is first loaded into the cache space before rendering. Due to limitations in the cache space size and loading speed, the tile image contains two resolutions: a first tile image, which is a thumbnail of the panoramic view, and a second tile image, which is a high-resolution version of the panoramic view. To improve rendering efficiency, the first tile image is loaded first, and then the second tile image is loaded as needed.

[0101] Step 402: Obtain the vertex coordinates of the first and second tile maps;

[0102] The vertex coordinates of the tile map have a one-to-one correspondence with the vertices in the panoramic sphere model.

[0103] Step 403: Determine the target tile map vertex from the tile map vertices based on the target vertex;

[0104] The target vertex is the vertex of the panoramic sphere model within the viewport range. Based on the correspondence between the tile map vertex coordinates and the vertices in the panoramic sphere model, the tile map vertex corresponding to the target vertex is determined as the target tile map vertex.

[0105] Step 404: Render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the vertex of the target tile map.

[0106] This disclosure renders the entire first panoramic sphere model and the second panoramic sphere model by default. In order to improve rendering efficiency, only the target vertices within the viewport range need to be rendered with a relatively clear second tile map, and the vertices outside the viewport range need to be rendered with a first tile map.

[0107] Optional, Figure 4 Step 404 in the process renders the first panoramic sphere model and the second panoramic sphere model based on the tile maps corresponding to the vertices of the target tile map, specifically including:

[0108] In response to the cache space containing a second tile map corresponding to the target tile map vertex, the target vertex is rendered according to the second tile map;

[0109] In response to the cache space not containing a second tile map corresponding to the target tile map vertex, but containing a first tile map corresponding to the target tile map vertex, the target vertex is rendered based on the first tile map corresponding to the target tile map vertex.

[0110] Optional, Figure 4 Step 404 in the process of rendering the first panoramic sphere model and the second panoramic sphere model based on the tile map corresponding to the vertex of the target tile map further includes:

[0111] In response to the fact that the cache space does not contain the second tile map and the first tile map corresponding to the target tile map vertex, the first tile map is loaded into the cache space, and the target vertex is rendered according to the first tile map corresponding to the target tile map vertex.

[0112] If the cache space does not contain the second tile map and the first tile map corresponding to the target tile map vertex, the rendering of the entire panoramic sphere model cannot be completed. The first tile map needs to be loaded into the cache space and the target vertex needs to be rendered based on the first tile map.

[0113] Optionally, moving the viewpoint towards the target direction and adjusting the states of the first panoramic sphere model and the second panoramic sphere model includes:

[0114] The first panoramic sphere model is changed from an opaque state to a transparent state, while the second panoramic sphere model remains opaque. A switching animation is generated based on the rendered images corresponding to the first and second panoramic sphere models within the viewpoint.

[0115] In this embodiment, to achieve the panoramic scene switching effect, the viewpoint first needs to be moved. Then, during the movement, the transparency state of the first and second panoramic sphere models needs to be changed. The transparency state is controlled by the α value of the panoramic sphere model, which ranges from [0,1]. The larger the α value, the lower the transparency of the panoramic sphere model. When α=1, the panoramic sphere model is opaque; when α=0, it is fully transparent. During the viewpoint movement, the α value of the second panoramic sphere model is kept at 1, while the α value of the first panoramic sphere model is gradually reduced from 1 to 0. Thus, the first panoramic sphere model gradually becomes transparent within the viewpoint, while the second panoramic sphere model gradually becomes clear. The switching animation generated from the rendered images corresponding to the first and second panoramic sphere models can provide the viewing user with a scene-switching experience.

[0116] Figure 5 This is a schematic diagram of a panoramic sphere rendering process provided according to an embodiment of the present disclosure, such as... Figure 5 As shown, the method is applied to scenarios such as the metaverse and smart cities, and the method includes:

[0117] Step 501: Load the thumbnail. This embodiment of the disclosure renders the entire panoramic sphere model by default, so the thumbnail, i.e., the first tile image, is first loaded into the cache space. Since the thumbnail occupies less memory, the loading speed is relatively fast.

[0118] Step 502: Create the full Mesh. A Mesh is a polygonal mesh, a data structure used in computer graphics for modeling various irregular objects. In a polygonal mesh, triangular faces are the smallest unit of division, and their representation is relatively simple, flexible, and convenient for topological description, hence their widespread use. Therefore, Mesh often refers to triangular faces. In this embodiment, the Mesh refers to the data in the panoramic sphere model, consisting of vertex coordinates and vertex indices. The vertex indices indicate which vertices form a face.

[0119] Step 503: Load high-definition tiles on demand. The high-definition tiles are the second tile map. On demand means that the target vertices are obtained according to the viewpoint and viewport, and the second tile map corresponding to the vertices of the target tile map is loaded.

[0120] Step 504: Render the Mesh, which involves mapping the tile image onto the corresponding vertices of the panoramic sphere model for rendering.

[0121] Then determine whether the cache space stores high-resolution tile images corresponding to each vertex in the panoramic sphere model.

[0122] Step 505a: If the second tile map is stored in the cache space, then render the position corresponding to the panoramic sphere model according to the second tile map.

[0123] Step 505b: If the second tile image is not stored in the cache space, render the position corresponding to the panoramic sphere model based on the thumbnail.

[0124] Figure 6 This is a structural schematic diagram of a panoramic scene switching device provided according to an embodiment of the present disclosure, such as... Figure 6 As shown, the method is applied to scenarios such as the metaverse and smart cities, and the device includes:

[0125] The depth acquisition module 610 is used to acquire the depth information of each pixel in the panoramic image data of the current scene and the target scene, so as to generate a first depth map corresponding to the current scene and a second depth map corresponding to the target scene, wherein the panoramic image data includes multiple tile images.

[0126] The model creation module 620 is used to create a first panoramic sphere model corresponding to the current scene and a second panoramic sphere model corresponding to the target scene. It determines the coordinates of the vertices in the first panoramic sphere model based on the first depth map and the coordinates of the vertices in the second panoramic sphere model based on the second depth map.

[0127] The rendering module 630 is used to acquire the viewpoint and the corresponding visible area, and to determine the vertices of the first panoramic sphere model and the second panoramic sphere model within the visible area as target vertices, and to render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the target vertices.

[0128] The switching module 640 is used to move the viewpoint toward the target direction and adjust the state of the first panoramic sphere model and the second panoramic sphere model to display the target scene.

[0129] Optionally, the model creation module includes:

[0130] The first coordinate adjustment submodule is used to obtain the first mapping point corresponding to the vertex in the first panoramic sphere model in the first depth map, and adjust the coordinates of the vertex according to the depth information of the first mapping point;

[0131] The second coordinate adjustment submodule is used to obtain the second mapping point corresponding to the vertex in the second panoramic sphere model in the second depth map, and adjust the coordinates of the vertex according to the depth information of the second mapping point.

[0132] Optionally, the rendering module includes:

[0133] The visual method determination submodule is used to determine the coordinates and visual direction of the viewpoint, and to draw a ray along the visual direction starting from the viewpoint;

[0134] The visible area determination submodule is used to determine the area where the ray intersects with the first panoramic sphere model or the second panoramic sphere model as the visible area.

[0135] Optionally, the rendering module includes:

[0136] A tile image loading submodule is used to load the tile image into a cache space, wherein the tile image includes a first tile image and a second tile image, and the resolution of the first tile image is smaller than that of the second tile image;

[0137] The tile map vertex acquisition submodule is used to acquire the tile map vertex coordinates of the first tile map and the second tile map;

[0138] The target vertex determination submodule is used to determine the target tile map vertex from the tile map vertices based on the target vertex;

[0139] The rendering submodule is used to render the first panoramic sphere model and the second panoramic sphere model based on the tile map corresponding to the vertex of the target tile map.

[0140] Optionally, the target vertex determination submodule includes:

[0141] The target vertex determination unit is used to obtain a mapping table between the target vertex and the tile graph vertices, and determine the tile graph vertex corresponding to the target vertex as the target tile graph vertex according to the mapping table.

[0142] Optionally, the rendering submodule includes:

[0143] The first rendering unit is configured to render the target vertex according to the second tile map in response to the cache space containing a second tile map corresponding to the target tile map vertex;

[0144] The second rendering unit is configured to render the target vertex based on the first tile map corresponding to the target tile map vertex in response to the cache space not containing a second tile map corresponding to the target tile map vertex and containing a first tile map corresponding to the target tile map vertex.

[0145] Optionally, the device further includes:

[0146] The third rendering unit, in response to the fact that the cache space does not contain the second tile map and the first tile map corresponding to the vertex of the target tile map, loads the first tile map into the cache space and renders the target vertex according to the first tile map corresponding to the vertex of the target tile map.

[0147] Optionally, the switching module includes:

[0148] The state adjustment module is used to change the first panoramic sphere model from an opaque state to a transparent state, while keeping the second panoramic sphere model in an opaque state, and to generate a switching animation based on the rendered images corresponding to the first and second panoramic sphere models within the viewpoint.

[0149] The steps corresponding to the steps described above are explained in detail in the aforementioned method and will not be repeated here.

[0150] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0151] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0152] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0153] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0154] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the panoramic scene switching method. For example, in some embodiments, the panoramic scene switching method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the panoramic scene switching method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the panoramic scene switching method by any other suitable means (e.g., by means of firmware).

[0155] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0157] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0159] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0160] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0161] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0162] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for panoramic scene switching, comprising: The depth information of each pixel in the panoramic image data of the current scene and the target scene is obtained to generate a first depth map corresponding to the current scene and a second depth map corresponding to the target scene. The panoramic image data contains multiple tile images. Create a first panoramic sphere model corresponding to the current scene and a second panoramic sphere model corresponding to the target scene. Determine the coordinates of the vertices in the first panoramic sphere model based on the first depth map, and determine the coordinates of the vertices in the second panoramic sphere model based on the second depth map. Obtain the viewpoint and the corresponding visible area, and determine the vertices of the first panoramic sphere model and the second panoramic sphere model within the visible area as target vertices. Render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the target vertices. In response to a scene switching command, the viewpoint is moved toward the target direction, and the states of the first panoramic sphere model and the second panoramic sphere model are adjusted to display the target scene; Rendering the first panoramic sphere model and the second panoramic sphere model based on the tile map corresponding to the target vertex includes: Load the tile map into the cache space, wherein the tile map includes a first tile map and a second tile map, and the resolution of the first tile map is smaller than that of the second tile map; Obtain the vertex coordinates of the first and second tile maps; The target tile map vertex is determined from the tile map vertices based on the target vertex; The first panoramic sphere model and the second panoramic sphere model are rendered based on the tile map corresponding to the vertex of the target tile map, wherein, in response to the cache space containing the second tile map corresponding to the vertex of the target tile map, the target vertex is rendered based on the second tile map.

2. The method according to claim 1, wherein, Determining the coordinates of the vertices in the first panoramic sphere model based on the first depth map includes: Obtain the first mapping point of the vertex in the first panoramic sphere model corresponding to the first depth map, and adjust the coordinates of the vertex according to the depth information of the first mapping point.

3. The method according to claim 1, wherein, Determining the coordinates of vertices in the second panoramic sphere model based on the second depth map includes: Obtain the second mapping point of the vertex in the second panoramic sphere model corresponding to the second depth map, and adjust the coordinates of the vertex according to the depth information of the second mapping point.

4. The method according to claim 1, wherein, The acquisition of the viewpoint and the corresponding visible area includes: Determine the coordinates and visible direction of the viewpoint, and draw a ray along the visible direction starting from the viewpoint; The visible area is defined as the region where the ray intersects with the first panoramic sphere model or the second panoramic sphere model.

5. The method according to claim 1, wherein, The step of determining the target tile map vertex from the tile map vertices based on the target vertex includes: Obtain the mapping table between the target vertex and the tile graph vertices, and determine the tile graph vertex corresponding to the target vertex as the target tile graph vertex according to the mapping table.

6. The method according to claim 1, wherein, The step of rendering the first panoramic sphere model and the second panoramic sphere model based on the tile map corresponding to the vertex of the target tile map further includes: In response to the cache space not containing a second tile map corresponding to the target tile map vertex, but containing a first tile map corresponding to the target tile map vertex, the target vertex is rendered based on the first tile map corresponding to the target tile map vertex.

7. The method according to claim 1, wherein, The method includes: In response to the fact that the cache space does not contain the second tile map and the first tile map corresponding to the target tile map vertex, the first tile map is loaded into the cache space, and the target vertex is rendered according to the first tile map corresponding to the target tile map vertex.

8. The method according to claim 1, wherein, Moving the viewpoint toward the target direction and adjusting the states of the first and second panoramic sphere models includes: The first panoramic sphere model is changed from an opaque state to a transparent state, while the second panoramic sphere model remains opaque. A switching animation is generated based on the rendered images corresponding to the first and second panoramic sphere models within the viewpoint.

9. A device for panoramic scene switching, comprising: The depth acquisition module is used to acquire the depth information of each pixel in the panoramic image data of the current scene and the target scene, so as to generate a first depth map corresponding to the current scene and a second depth map corresponding to the target scene, wherein the panoramic image data contains multiple tile images; The model creation module is used to create a first panoramic sphere model corresponding to the current scene and a second panoramic sphere model corresponding to the target scene. It determines the coordinates of the vertices in the first panoramic sphere model based on the first depth map and the coordinates of the vertices in the second panoramic sphere model based on the second depth map. The rendering module is used to acquire the viewpoint and the corresponding visible area, and to determine the vertices of the first panoramic sphere model and the second panoramic sphere model within the visible area as target vertices, and to render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the target vertices. The switching module is used to move the viewpoint toward the target direction and adjust the state of the first panoramic sphere model and the second panoramic sphere model to display the target scene; The rendering module includes a tile map loading submodule, a tile map vertex acquisition submodule, a target vertex determination submodule, and a rendering submodule; The tile image loading submodule is used to load the tile image into the cache space, wherein the tile image includes a first tile image and a second tile image, and the resolution of the first tile image is smaller than that of the second tile image; The tile map vertex acquisition submodule is used to acquire the tile map vertex coordinates of the first tile map and the second tile map; The target vertex determination submodule is used to determine the target tile map vertex from the tile map vertices based on the target vertex; The rendering submodule is used to render the first panoramic sphere model and the second panoramic sphere model according to the tile map corresponding to the vertex of the target tile map. The rendering submodule includes a first rendering unit, which is used to render the target vertex according to the second tile map in response to the cache space containing the second tile map corresponding to the vertex of the target tile map.

10. The apparatus according to claim 9, wherein, The model creation module includes: The first coordinate adjustment submodule is used to obtain the first mapping point of the vertex in the first panoramic sphere model in the first depth map, and adjust the coordinates of the vertex according to the depth information of the first mapping point.

11. The apparatus according to claim 9, wherein, The model creation module includes: The second coordinate adjustment submodule is used to obtain the second mapping point corresponding to the vertex in the second panoramic sphere model in the second depth map, and adjust the coordinates of the vertex according to the depth information of the second mapping point.

12. The apparatus according to claim 9, wherein, The rendering module includes: The visual method determination submodule is used to determine the coordinates and visual direction of the viewpoint, and to draw a ray along the visual direction starting from the viewpoint; The visible area determination submodule is used to determine the area where the ray intersects with the first panoramic sphere model or the second panoramic sphere model as the visible area.

13. The apparatus according to claim 9, wherein, The target vertex determination submodule includes: The target vertex determination unit is used to obtain a mapping table between the target vertex and the tile graph vertices, and determine the tile graph vertex corresponding to the target vertex as the target tile graph vertex according to the mapping table.

14. The apparatus according to claim 9, wherein, The rendering submodule also includes: The second rendering unit is configured to render the target vertex based on the first tile map corresponding to the target tile map vertex in response to the cache space not containing a second tile map corresponding to the target tile map vertex and containing a first tile map corresponding to the target tile map vertex.

15. The apparatus according to claim 9, wherein, The device further includes: The third rendering unit, in response to the fact that the cache space does not contain the second tile map and the first tile map corresponding to the vertex of the target tile map, loads the first tile map into the cache space and renders the target vertex according to the first tile map corresponding to the vertex of the target tile map.

16. The apparatus according to claim 9, wherein, The switching module includes: The state adjustment module is used to change the first panoramic sphere model from an opaque state to a transparent state, while keeping the second panoramic sphere model in an opaque state, and to generate a switching animation based on the rendered images corresponding to the first and second panoramic sphere models within the viewpoint.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images