360-degree indoor dynamic VR panoramic roaming browsing seamless transition method

Abstract

According to the 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method, panoramic VR and a three-dimensional field model are loaded and fused, the three-dimensional field model is established as a reference architecture, a panoramic site is set in the three-dimensional field model, and the three-dimensional field model is naturally linked with two-dimensional panoramic roaming; therefore, the roaming system not only can browse all panoramic roaming sites, but also can jump out of the limitation of a panoramic ball, visit the whole three-dimensional space and join two heterogeneous browsing interaction modes of panoramic roaming and model browsing, and seamless transition switching is achieved by generating a temporary observation camera and interpolating the spatial pose of the temporary observation camera. Smooth switching transition between 360-degree browsing scenes is achieved, two schemes of dynamic panoramic VR and real-time panoramic sphere are included, a 360-degree indoor dynamic full-scene seamless roaming prototype system is established, a method for compressing panoramic VR and layering and slicing panoramic VR to ensure reusability of codes is constructed, and seamless transition of 360-degree dynamic panoramic roaming browsing is achieved.

Description

Technical Field

[0001] The present application relates to a method for seamless transition of indoor panoramic roaming browsing, and in particular to a method for seamless transition of 360-degree indoor dynamic VR panoramic roaming browsing, and belongs to the field of virtual reality three-dimensional image technology. Background Art

[0002] 360-degree panoramic roaming is a virtual reality technology based on static images to create virtual reality scenes. It has the advantages of low hardware requirements for playback devices, high display resolution, good navigation, strong interactivity, strong authenticity, easy data collection and processing, and small data volume for easy network transmission. The popularity of the panoramic roaming system is because it overcomes the three unresolved problems of true three-dimensional virtual reality technology: i) difficulty in obtaining three-dimensional data; ii) difficulty in network transmission due to large data volume; iii) high hardware requirements, low cost performance for display effect and interactivity. In terms of data acquisition, the indoor panoramic roaming system only needs to use a digital camera with a fisheye lens to collect multiple image data at the desired roaming location, and then use tools such as PanoTools and PTGui to stitch them together to obtain a panoramic VR. In the absence of a fisheye lens, an ordinary digital camera can also be used to collect more image data, and no special acquisition equipment is required. In network transmission, the generated panoramic VR is compressed using the JPEG standard. Generally, a panoramic VR with a resolution of 8192×4096 can be compressed to less than 2MB, achieving an excellent experience. If the image pyramid is used to reduce the resolution to 2048×1024, the data volume is less than 200KB, and there is no obvious loss in the display effect under normal viewing angles.

[0003] Due to the lack of real three-dimensional information, the 360-degree panoramic roaming system also has some problems compared with general virtual reality systems, which are reflected in: jump-like switching occurs when the scene transitions between roaming points, the effect is not natural, and the roaming experience is seriously affected. There are generally two solutions. The first is based on light field information. By collecting a large amount of data and creating a full light function, the view of any viewpoint can be calculated; the second is based on image deformation technology, and some image processing is performed between two panoramic views. In solving the problem that the spatial relationship of panoramic roaming sites is destroyed, resulting in users being unable to perceive the position change during panoramic roaming and feeling lost in space, it can be considered from two perspectives: global and local. From a global perspective, the introduction of model data as a global reference framework to govern all panoramic roaming sites can transform the problem into 1) the fusion problem of model and panorama in space; and 2) the connection problem of model browsing and panoramic browsing, two different interaction and browsing methods, in time. Locally, it is necessary to solve the problem of smooth transition between panoramic roaming scenes. From the perspectives of texture and model, the former can make the transition natural by generating a gradient texture map, and the latter can deform the panoramic scene by moving the panoramic ball nodes to achieve a natural transition effect.

[0004] The problems that need to be solved in the seamless transition of indoor panoramic roaming browsing in the prior art and the key technical difficulties of this application include:

[0005] (1) In the existing panoramic roaming system, when the scene transitions between roaming points, there will be a jump-like switching, which is not natural. In the panoramic roaming system, there is a common problem that the spatial association between panoramic sites is broken, resulting in a poor roaming experience. The existing technology lacks the idea and method of using a three-dimensional model as a global reference framework and placing panoramic sites in it, and lacks a panorama-model transition solution that connects the two observation interaction methods by interpolating temporary observation camera postures. The existing technology does not have a dynamic panoramic VR and dynamic panoramic texture solution for seamless transition of 360-degree browsing scenes, lacks a method based on guiding graphic deformation with matching point pairs with the same name, and generally has jumpy scene switching, which cannot be as natural as walking in a three-dimensional scene. The feeling of spatial loss is obvious. At the same time, there is a lack of the design, software platform and technology of the 360-degree browsing scene seamless transition prototype system, and it is impossible to achieve a seamless transition of 360-degree indoor panoramic roaming browsing.

[0006] (2) The unnatural transition problem of panoramic site switching in the 360-degree roaming system is becoming more and more prominent and needs to be solved urgently. Due to the scene switching problem, the panoramic roaming system cannot effectively connect the panoramic sites, making it difficult for users to grasp the changes in spatial position when switching scenes, resulting in a sense of disorientation, which greatly affects the browsing experience of the roaming system. At the same time, there is a contradiction between panoramic VR and three-dimensional field models. In the virtual three-dimensional space, panoramic VR uses a panoramic ball to realize panoramic roaming browsing. When integrating the panoramic ball and the three-dimensional model in space, it is necessary to consider the positional relationship between them and the setting of the proportion. If the panoramic ball is too large, it will appear very inconsistent when viewed together with the model in the three-dimensional space; if the panoramic ball is too small, it is difficult to enter the browsing of the panoramic site when browsing the model. If the positions of the two are not placed correctly, it will mislead the user and affect his grasp of the spatial position. There are significant differences between these panoramic VR and three-dimensional field models, resulting in the loading and fusion of the two is not simply loading the two into the three-dimensional environment. It is necessary to consider how to transform the two coordinate spaces into the fused world coordinates. In addition, there are also obvious differences in the observation method between the two. There is an unsolvable contradiction in time and space between panoramic VR and three-dimensional field models.

[0007] (3) The existing indoor panoramic roaming systems have not been able to solve the unnatural phenomenon when switching between roaming, and the scene switching problem has become the shortcoming and deficiency of the panoramic roaming system. The existing technology lacks a 360-degree seamless scene transition solution, a transition between three-dimensional models and panorama, a transition between panoramic sites, and an interpolated temporary observation camera pose, dynamic panoramic VR, and a real-time panoramic ball solution to effectively solve the above problems; there is a lack of a loading and fusion method for panoramic VR and three-dimensional field models, and it is impossible to achieve a smooth switching transition between 360-degree browsing scenes. There is a lack of a 360-degree indoor dynamic full-scene seamless roaming prototype system, and it is impossible to achieve a seamless transition between panoramic sites. The visual effect is unnatural, and there is a performance bottleneck, resulting in insufficient view rendering update frame rate during transition, a freeze problem, and an inability to achieve a 360-degree dynamic panoramic roaming seamless transition. Summary of the invention

[0008] In view of the closed and local characteristics of indoor scenes, this application has established a 360-degree seamless scene transition solution, which is used for the transition between three-dimensional models and panoramas, and the transition between panoramic sites. By interpolating temporary observation camera postures, dynamic panoramic VR and real-time panoramic ball solutions, the unnatural roaming switching jump and scene switching problems that the existing technology has not been able to solve well are effectively solved; for the seamless transition problem between three-dimensional models and panoramic VR, by introducing additional temporary observation cameras, linear and nonlinear interpolation from the initial state to the target state is performed to achieve the purpose of connecting the two heterogeneous interaction modes; for the seamless transition problem between panoramic sites, the deformation process is guided by introducing the same-name control points to make the visual effect more natural. The real-time panoramic ball is sensitive to the selection of the same-name control points. This solution is used to transition the panoramic view, and the rendering frame rate is almost not reduced, and the transition effect is smoother. This application has good navigation and strong authenticity. The transition of panoramic site switching is natural. The panoramic roaming system effectively connects the panoramic sites. Users can grasp the changes in spatial position when switching scenes, and will not feel disoriented, and have a good browsing experience.

[0009] In order to achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0010] 360-degree indoor dynamic VR panoramic roaming seamless transition method, for the seamless transition problem between 3D model and panoramic VR, by introducing an additional temporary observation camera, linear and nonlinear interpolation from the initial state to the target state is performed to achieve the purpose of connecting the two heterogeneous interaction modes; for the seamless transition problem between panoramic sites, by introducing the same-name control points to guide the deformation process to make the visual effect more natural, the real-time panoramic ball solution effect and the dynamic panoramic VR solution are integrated to transition the panoramic view;

[0011] 1) Loading and fusion method of panoramic VR and three-dimensional field model: Establish a three-dimensional field model as a reference framework, set up panoramic sites in the three-dimensional field model, and naturally connect the three-dimensional field model with the two-dimensional panoramic roaming; expand the panoramic roaming system by introducing the three-dimensional model of the scene, so that the roaming system can not only browse various panoramic roaming sites, but also jump out of the limitations of the panoramic ball and tour the entire three-dimensional space, connecting the two heterogeneous browsing interaction modes of panoramic roaming and model browsing, and establish a solution for seamless transition switching by generating a temporary observation camera and interpolating its spatial posture;

[0012] 2) Smooth switching transition between 360-degree browsing scenes: including dynamic panoramic VR and real-time panoramic sphere solutions. Dynamic panoramic VR forms a dynamic triangulated grid by extracting the same-name control points to update the original panoramic VR to form a dynamic panoramic VR for panoramic transition. Dynamic panorama is applied when the scene is switched, and a natural and smooth transition is made from the current site to the next panoramic site. The real-time panoramic sphere forms a dynamic triangulated grid by the same-name control points to cover the original panoramic sphere, and the smooth transition of the panoramic scene is achieved through the deformation of the panoramic sphere itself. In the dynamic panoramic VR and real-time panoramic sphere solutions, objects in the scene move slowly to the side, and the view is naturally stretched and enlarged, giving people a feeling of natural walking and observation, achieving a natural transition;

[0013] 3) 360-degree indoor dynamic full-scene seamless roaming prototype system: A single-frame application based on Three.js, using JavaScript for code implementation, Makefile for project management, plain text JSON and panoramic VR file system for data access, establishing a full-scene data processing flow, and constructing panoramic VR compression and panoramic VR layered slicing methods to ensure code reusability.

[0014] Preferably, the three-dimensional field model is used as a reference framework: if the indoor panoramic roaming system does not need to be linked with other systems, its main reference framework, i.e., the world coordinate system, is directly selected as the building or several rooms of the roaming location. First, the three-dimensional field model is loaded into the roaming system. The data formats of different model data are different, but the information provided is consistent. The model data is a combination of geometric structure and material. The former records the position information of each node of the model, and the latter records the texture map of the model.

[0015] When loading, it is parsed according to the corresponding data specifications and the model is drawn in three-dimensional space. First, a local coordinate system is generated to load the points, lines, and surfaces in the model data. For points, it is important to correctly render the size and color of the points; for lines, one-dimensional textures need to be specified to make color transitions; for surfaces, the texture and texture coordinates of each node need to be specified. Finally, if the coordinate system of the model data has a different rotation from the coordinate system used by the rendering system, one of the directions needs to be reversed. If the model is not directly used as a world coordinate, the position and posture of the model also need to be specified.

[0016] Preferably, the 3D scene model is naturally connected with the 2D panoramic roaming: a plurality of cameras are kept in the 3D environment, and the 3D scene model is naturally connected with the 2D panoramic roaming by transitioning between the cameras;

[0017] Assume that the panoramic site P is in the model M, and the camera C p and C m Observation, C. m The current observation result is roughly in the positive direction of the x-axis. The sphere in the center is the panoramic sphere where the panoramic site P is located. p The observation results are roughly towards the negative direction of the x-axis. When the user clicks the panoramic ball, the viewing angle changes from C m The current state gradually transitions to C p The current state of the system and make corresponding adjustments in the interaction mode;

[0018] The following logic is introduced to make the transition more natural: First, a temporary observation camera C is introduced x Used as a transition, during which the user observes C x When the user starts to switch from model viewing mode to panoramic viewing mode, C x Start with C m Get the pose and camera configuration, C x The observations are consistent with C m Consistent, at the same time, when enabling C x After that, the original C m The interaction mode is invalid. At this time, it no longer responds to the user's mobile viewing operation, and the user interface enters the automatic roaming state. x From the initial state C m Towards the target state C p The transition from the model to the panoramic view is completed. The posture is expressed by quaternion in C m and C p Interpolating SLERP between the two poses yields the camera orientation at each moment.

[0019] Preferably, a solution A for smooth switching transition between 360-degree browsing scenes: Dynamic panoramic VR: Generate a dynamic panoramic VR with gradual transition, play the dynamic panorama during the panoramic transition, and realize natural scene transition;

[0020] First, the input is two panoramic VRs. The goal is to transition naturally from the first panoramic VR to the second panoramic VR. Consider the transition process from panoramic site a to panoramic site b. The panoramic VRs of the two panoramic sites are panoramic VRa and panoramic VRb, respectively, denoted as I (a) and I (b) , in I (a) and I (b)Find N pairs of points P with the same name in the corresponding area i , i∈{1,…,N}, the same-name point pair P i in I (a) The upper coordinate is P i (a) =(x i (a) ,y i (a) ), in I (b) Upper coordinate P i (b) =(x i (b) ,y i (b) ), {P i} Form a point set {P i (a)}, which is conducive to the Delaunay triangulation algorithm to maintain the grid structure cluster, and the point set {P i (a)}Use Delaunay triangulation to triangulate into M triangles T j (a) , j∈{1,…,M};

[0021] For each triangle T j (a) =△ABC, its three vertices are A=P j1 (a) , B=P j2 (a) and C = P j3 (a) , A, B, C are all from the point set {P i (a)}, j1, j2, j3∈{1,…,N}, find the points in I according to the same name. (b) The corresponding point on the j1 (b) , B=P j2 (b) and C = P j3 (b) , A', B', C' form a triangle T j (b) =△A'B'C';

[0022] For each T j (b) , generate a triangular face Face j , its vertices are initialized to P1 = A, P2 = B and P3 = C, and I is added (a) and I (b) Two texture maps, the transparency of the former is set to TRANSPARENT (I(a) )=0, the latter transparency is set to TRANSPARENT(I (b) )=1, triangular face j The six UV texture coordinates of the three vertices are taken from A, B and C in I (a) The texture coordinates of A', B' and C' in I (b) The texture coordinates of

[0023] Assume parameter α is the progress of the scene transition from panoramic site a to panoramic site b. When α = 0, the panoramic roaming scene a is displayed; when α = 1, the panoramic roaming scene b is displayed. α is gradually adjusted over time. The process of α changing from 0 to 1 is the process of the panoramic roaming scene transition. The transition scheme is described as follows: at any moment of scene transition, the triangular face Face j The coordinates of the three vertices are P1 = LERP (A, A', α), P2 = LERP (B, B', α) and P3 = LERP (C, C', α), where LERP is a linear interpolation function that satisfies LERP (f, t, α) = (1-α) × f + α × t. At the same time, the transparency of the texture on the triangle surface satisfies TRANSPARENT (I a )=α、TRANSPARENT(I b )=1-α, the triangle patch is covered on the original texture and resampled to form a new texture frame. These texture frames are a dynamic texture that can be used for natural transition of panoramic scenes as a whole.

[0024] Preferably, obtain the control points with the same name: extract the point pairs with the same name through the feature point extraction operator and the matching algorithm, and conduct manual review, open the two panoramas with a customized point editor with the same name, add some control point pairs with the same name through the mouse and keyboard, and in the editor, the points with the same name of the two panoramas are connected. This editor provides basic operations of loading, saving, adding, deleting and editing the control points with the same name. When manually adding the control point pairs with the same name, the visual focus of the user when browsing the panorama should be considered.

[0025] Preferably, triangulation seamless transition: the control points with the same name are placed at I (a) The point set {P i (a)}Triangulation, a triangular face Face transitions to another triangular face. When a triangle △ABC transitions to position △A'B'C', at time α, the three vertex coordinates P1, P2 and P3 of Face are linear interpolations of A and A', B and B', C and C' respectively. At the same time, the color transition of the internal points is achieved through Interpolation is obtained, where is the pixel value with the centroid coordinates (i, j) on △ABC, is the pixel value with the centroid coordinates (i, j) on △A'B'C', for {T j Each triangle in} generates a triangular patch and records the texture at each α moment, recorded as Generated from I (a) To I (b) A series of texture frames Create a dynamic texture Playing dynamic textures on the panoramic sphere can achieve the effect of natural scene transition.

[0026] Preferably, solution B for smooth switching transition between 360-degree browsing scenes: Real-time panoramic sphere: Use the same-name control point pairs to construct matching triangles, and perform linear transition of position and color based on the same-name points to form each frame of dynamic texture, and then apply these texture frames on the panoramic sphere to achieve natural panoramic scene transition;

[0027] The real-time panoramic sphere solution avoids the process of regenerating textures and remapping by moving the nodes associated with the panoramic sphere;

[0028] Based on the dynamic texture method, the control point pairs with the same name are generated, and {P i (a)} and {P i (b)}, based on the dynamic texture method to generate triangulated networks, generate {T j (a)}, for each triangle T j (a) ∈{T j (a)}, find the nodes on the panoramic sphere within this triangle, assuming there are K nodes, set as {V k (a,j)}, where k∈{1,…,K}, for each node V k (a,j) , find its j (a) = Coordinates of the barycentric coordinate system in △ABC And find I (b) The corresponding triangle T j (b) =△A'B'C's same centroid coordinates Cartesian coordinates of the point V k (b,j) , determine a node V on the panoramic sphere k (a,j) The deformation target position V k (b,j) In the dynamic texture, the position of △ABC gradually transitions to △A'B'C'. In the real-time panoramic ball, V k (a,j)The position gradually transitions to V k (b,j) ;

[0029] At the panorama sphere node V k (a,j) The geometric position gradually transitions to V k (b,j) When the texture is deformed, another texture is generated for the original panoramic sphere. This texture uses the panoramic VRI (b) , the texture coordinates remain unchanged, the node V k (a,j) There are two texture coordinates (u (a，j，k) , v (a，j，k) ) and (u (b，j，k) , v (b，j，k) ) According to V k (a,j) The target position V k (b,j) Calculate the new texture coordinates and assign them to the second texture coordinates, that is lonlat2uv converts the spherical longitude and latitude coordinates into texture UV coordinates. Similarly, when moving the nodes, the transparency of the two sets of textures of the panoramic sphere needs to be changed accordingly.

[0030] Preferably, improvement 1: introducing triangular facets in three-dimensional space: based on the dynamic panoramic texture generation process, introducing triangular facets in three-dimensional space: for each triangle T j (a) ∈{T j (a)}, Generate triangular face Face j , and apply I on it according to the triangle position (a) and I (b) Two panoramic VRs are used as texture maps, and then the triangular face is mapped according to the control points of the same name. j The position of T j (a) Transition to T j (b) The color interpolation ratio of the two textures is gradually changed accordingly to achieve a natural transition effect; the position transition of the triangle facets in the dynamic panoramic VR solution adopts LERP linear interpolation, and the position transition of the triangle facets in the dynamic panoramic solution in three-dimensional space adopts SLERP spherical interpolation;

[0031] Improvement 2: Use finely enhanced triangles: further subdivide the triangles and perform more detailed texture mapping to reduce distortion; for each edge of each triangle, let n be the number of nodes on the edge. When n=2, there is no need to interpolate other nodes; when n>2, it is necessary to interpolate some nodes on the edges and inside of the triangles to form triangles.

[0032] Preferably, the whole scene data processing flow: extracting the bounding box model from the panoramic VR requires finding the coordinates of the eight corners in the panoramic VR, and the steps of generating the bounding box model are: 1) manually determining the coordinates of the eight corner points p i , i∈{1,…,8}; 2) Determine the ground and ceiling based on the relative comparison between the panoramic VR zenith direction and the camera height in the room; 3) The eight corner points p i The ground and ceiling projected into the three-dimensional space correspond to P i ,i∈{1,…,8}; 4) According to P i Generate six planes: front, back, left, right, top, and bottom, and project the panoramic VR onto these six planes to form a bounding box model;

[0033] The site information is organized in JSON text, which records the information of each panoramic site in a scene, including: 1) ID; 2) Name; 3) Introduction; 4) Panoramic VR path; 5) Panoramic posture information (north direction and zenith direction); 6) Other roaming points that can be jumped to;

[0034] The data processing flow of the indoor panoramic roaming system with 360-degree seamless scene transition includes: 1) using PTGui to stitch the original images into a panoramic VR with an aspect ratio of 2:1 using PlateCarree projection; 2) manually locating the corner points in the panoramic VR, generating a bounding box model, and making a roaming environment model; 3) manually determining the control point pairs with the same name between the roamable sites and writing them into the configuration file.

[0035] Preferably, the layered slicing of the panoramic VR ensures the reusability of the code: generate a new panoramic VR, and use the CanvasRenderer of Three.js to map the Canvas element on the HTML page as a texture to the panoramic sphere. After obtaining the new slice, the virtual panoramic VR on the Canvas element is updated, and then the global rendering is redrawn to achieve the effect of updating a part of the spherical surface on the panoramic sphere. The specific steps are: 1) Write a script to layer and slice a high-definition panoramic VR according to certain rules to obtain a series of sub-images; 2) Write the corresponding virtual panoramic VR adaptation code according to the above rules, which is responsible for the drawing and updating of a Canvas element; 3) Bind the user's field of view changes to the adapter, and update the virtual panoramic VR in time according to the changes in the field of view.

[0036] Compared with the prior art, the innovations and advantages of this application are:

[0037] (1) Aiming at the closed and local characteristics of indoor scenes, this application establishes a 360-degree seamless scene transition solution, which is used for the transition between three-dimensional models and panoramas, and between panoramic sites. By interpolating temporary observation camera positions, dynamic panoramic VR and real-time panoramic sphere solutions, the unnaturalness of roaming switching and scene switching problems that the existing technology has not solved well are effectively solved; for the seamless transition problem between three-dimensional models and panoramic VR, an additional temporary observation camera is introduced to perform linear and nonlinear interpolation from the initial state to the target state to achieve the purpose of connecting the two heterogeneous interaction modes; for the seamless transition problem between panoramic sites, the same-name control points are introduced to guide the deformation process to make the visual effect more natural. The dynamic panoramic VR solution has a realistic and natural effect, but there is a performance bottleneck; the real-time panoramic sphere solution has a similar effect to the dynamic panoramic VR solution, which is sensitive to the selection of the same-name control points, but has good performance. When this solution is used to transition the panoramic view, the rendering frame rate is almost not reduced, and the transition effect is smoother. This application has good navigation and strong authenticity. The transition between panoramic site switches is natural. The panoramic roaming system effectively connects the panoramic sites. Users can grasp the changes in spatial position when switching scenes, will not feel disoriented, and have a good browsing experience.

[0038] (2) In order to solve the problem that the spatial relationship of panoramic roaming sites is destroyed, which causes users to be unable to perceive the change of position during panoramic roaming and feel lost in space, this application considers it from both global and local perspectives. From a global perspective, the model data is introduced as a global reference framework to govern all panoramic roaming sites, and the problem is transformed into the spatial fusion problem of model and panorama, as well as the temporal connection problem of two different interaction and browsing methods, model browsing and panoramic browsing. Locally, the problem of smooth transition between panoramic roaming scenes is solved. From the perspectives of texture and model, the former makes the transition natural by generating gradient texture maps, and the latter deforms the panoramic scene by moving the panoramic ball nodes to achieve the effect of natural transition. The innovation lies in: 1) Mixing and organizing panoramic data and 3D model data, and introducing 3D field models into the roaming system; 2) Applying color mixing strategies to the triangles of the front and back views that match each other instead of the entire view without registration, solves the problem of unnatural scene transition; 3) Implementing a usable prototype system with high configurability, generating configuration information according to predetermined rules, and conveniently calling the front-end rendering module to experience the roaming environment.

[0039] (3) For the roaming transition of indoor scenes, this application establishes a loading and fusion method for panoramic VR and three-dimensional field models, establishes a three-dimensional field model as a reference framework, sets up panoramic sites in the three-dimensional field model, and naturally connects the three-dimensional field model with the two-dimensional panoramic roaming; so that the roaming system can not only browse various panoramic roaming sites, but also jump out of the limitations of the panoramic ball and tour the entire three-dimensional space, connecting the two heterogeneous browsing interaction methods of panoramic roaming and model browsing, and establishing a solution for seamless transition by generating a temporary observation camera and interpolating its spatial posture; it realizes smooth switching transition between 360-degree browsing scenes, including two solutions: dynamic panoramic VR and real-time panoramic ball. Objects in the scene slowly move to the side, and the view is naturally stretched and enlarged, giving people a feeling of natural walking and observation, achieving a good natural transition effect. A 360-degree indoor dynamic full-scene seamless roaming prototype system was established, and a method for compressing panoramic VR and layering and slicing panoramic VR to ensure code reusability was constructed, realizing a seamless transition of 360-degree indoor dynamic VR panoramic roaming browsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of the panoramic sphere nodes and texture coordinates.

[0041] Figure 2 This is a schematic diagram of the panoramic sphere texture map.

[0042] Figure 3 It is a schematic diagram of the transition from the scene to the panorama of this application.

[0043] Figure 4 It is a schematic diagram of dynamic panoramic VR input image.

[0044] Figure 5 It is a dynamic panoramic VR panoramic browsing effect picture.

[0045] Figure 6 This is a schematic diagram of the problems existing in the original dynamic panoramic sphere solution.

[0046] Figure 7 This is a schematic diagram of the triangular patch result after fine enhancement.

[0047] Figure 8 It is a schematic diagram of generating pseudo model data from a panoramic image.

[0048] Fig. 9 It is the data processing flow chart of the 360-degree seamless scene transition panoramic roaming system. DETAILED DESCRIPTION

[0049] The following, in conjunction with the accompanying drawings, further describes the technical solution of the 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method provided by the present application, so that those skilled in the art can better understand the present application and implement it.

[0050] The 360-degree panoramic roaming system is a primary virtual reality technology based on static images. It is widely used because of its good navigation and high authenticity. With its popularity, the unnatural transition problem of panoramic site switching has become more prominent and needs to be solved urgently. Due to the scene switching problem, the panoramic roaming system cannot effectively connect the panoramic sites, making it difficult for users to grasp the changes in spatial position and feel disoriented when switching scenes, which greatly affects the browsing experience of the roaming system. To address this problem, especially the roaming transition of indoor scenes, this application includes:

[0051] 1) Loading and fusion method of panoramic VR and three-dimensional field model: Establish a three-dimensional field model as a reference framework, set up panoramic sites in the three-dimensional field model, and naturally connect the three-dimensional field model with the two-dimensional panoramic roaming; expand the panoramic roaming system by introducing the three-dimensional model of the scene, so that the roaming system can not only browse various panoramic roaming sites, but also jump out of the limitations of the panoramic ball and tour the entire three-dimensional space, connecting the two heterogeneous browsing interaction modes of panoramic roaming and model browsing, and establish a solution for seamless transition switching by generating a temporary observation camera and interpolating its spatial posture;

[0052] 2) Smooth switching transition between 360-degree browsing scenes: including two solutions: dynamic panoramic VR and real-time panoramic sphere. Dynamic panoramic VR forms a dynamic triangulated grid by extracting control points with the same name to update the original panoramic VR to form a dynamic panoramic VR for panoramic transition. Dynamic panorama is applied when the scene is switched, and a natural and smooth transition is made from the current site to the next panoramic site. The real-time panoramic sphere forms a dynamic triangulated grid through control points with the same name and covers the original panoramic sphere. The smooth transition of the panoramic scene is achieved through the deformation of the panoramic sphere itself. In the dynamic panoramic VR and real-time panoramic sphere solutions, objects in the scene move slowly to the side, and the view is naturally stretched and enlarged, giving people the feeling of natural walking and observation, achieving a good natural transition effect.

[0053] 3) 360-degree indoor dynamic full-scene seamless roaming prototype system: A single-frame application based on Three.js, using JavaScript for code implementation, Makefile for project management, using plain text JSON files and panoramic VR direct file system for data access, establishing a full-scene data processing flow, and constructing panoramic VR compression and panoramic VR layered slicing methods to ensure code reusability.

[0054] 1. Loading and fusion method of panoramic VR and 3D field model

[0055] The three-dimensional field model is introduced to control all panoramic browsing roaming sites to improve the overall sense of the roaming system, and the three-dimensional field model and the panoramic browsing ball are integrated in space, and the two different interaction modes related to this are connected in time.

[0056] 1. Contradictions between panoramic VR and 3D scene models

[0057] In a virtual three-dimensional space, panoramic VR uses a panoramic ball to achieve panoramic roaming browsing. To spatially fuse the panoramic ball and the three-dimensional model, it is necessary to consider the positional relationship between them and the setting of the ratio. If the panoramic ball is too large, it will appear inconsistent when viewed together with the model in three-dimensional space; if the panoramic ball is too small, it will be difficult to enter the panoramic site when browsing the model. If the positions of the two are not placed correctly, it will mislead the user and affect his grasp of the spatial position. There are significant differences between these panoramic VR and three-dimensional field models, resulting in the loading and fusion of the two is not a simple loading of both into the three-dimensional environment. It is necessary to consider how to transform the two coordinate spaces into the fused world coordinates.

[0058] In addition, there are obvious differences in the way of observation. There is a contradiction in time and space between panoramic VR and three-dimensional field models, which can be resolved through the following three steps: 1) the three-dimensional field model is used as a reference framework; 2) the panoramic site is located in the three-dimensional field model; 3) the natural connection between the three-dimensional field model and the two-dimensional panoramic roaming.

[0059] (II) 3D Field Model as a Reference Framework

[0060] If the indoor panoramic roaming system does not need to be linked with other systems, its main reference framework, i.e., the world coordinate system, is directly selected as the building or several rooms of the roaming location. First, the 3D field model is loaded into the roaming system. The data formats of different model data are different, but the information provided is consistent. The model data is a combination of geometric structure and material. The former records the position information of each node of the model, and the latter records the texture map of the model.

[0061] When loading, it is parsed according to the corresponding data specifications and the model is drawn in three-dimensional space. First, a local coordinate system is generated to load the points, lines, and surfaces in the model data. For points, it is important to correctly render the size and color of the points; for lines, one-dimensional textures need to be specified to make color transitions; for surfaces, the texture and texture coordinates of each node need to be specified. Finally, if the coordinate system of the model data has a different rotation from the coordinate system used by the rendering system, one of the directions needs to be reversed. If the model is not directly used as a world coordinate, the position and posture of the model also need to be specified.

[0062] (III) Setting up panoramic sites in the 3D field model

[0063] A panoramic VR is rendered on a panoramic ball to form a roamable panoramic site. Just record the position and posture of the panoramic VR during acquisition, and place the panoramic ball in the correct position under the model reference framework through coordinate transformation.

[0064] The panoramic ball is an observation ball formed by attaching panoramic VR to a sphere. Figure 1 (a) and Figure 1 (b) Generate a series of regular nodes on the plane point, attach the panoramic VR, and set the texture coordinates. Then move the nodes to the sphere, such as Figure 2 (a) Figure 2 (b) Figure 2 (c), forming a sphere, and finally pasting the texture to browse the panoramic site, such as Figure 2 (d) When converting a plane into a spherical surface, the panoramic VR should be attached to the inside, otherwise the observation effect will be incorrect, which can be achieved by reversing the x-axis;

[0065] Finally, the camera is placed at the center of the panoramic sphere and the camera posture is adjusted to simulate the effect of a person roaming around in a three-dimensional scene.

[0066] (IV) The 3D field model naturally connects to the 2D panoramic tour

[0067] Many cameras are kept in the 3D environment, and the natural connection between the 3D scene model and the 2D panoramic roaming is ensured by transitioning between the cameras.

[0068] Assume that the panoramic site P is in the model M, and the camera C p and C m Observation, C. m The current observation result is roughly in the positive direction of the x-axis. The sphere in the center is the panoramic sphere where the panoramic site P is located. p The observation results are roughly towards the negative direction of the x-axis. When the user clicks the panoramic ball, the viewing angle changes from C m The current state gradually transitions to C p The current state of the system and make corresponding adjustments in the interaction mode;

[0069] The following logic is introduced to make the transition more natural: First, a temporary observation camera C is introduced x Used as a transition, during which the user observes C x When the user starts to switch from model viewing mode to panoramic viewing mode (or vice versa), C x Start with C m Get the posture and camera configuration (such as camera field of view, near and far cutting plane parameters, etc.), C x The observations are consistent with C m Consistent, at the same time, when enabling C x After that, the original C m The interaction mode is invalid. At this time, it no longer responds to the user's mobile viewing operation, and the user interface enters the automatic roaming state. x From the initial state C m Towards the target state Cp The transition from the model to the panoramic view is completed. The posture is expressed by quaternion in C m and C p Interpolate SLERP between the two poses to get the camera orientation at each moment. The final effect is as follows Figure 3 .

[0070] 2. Smooth switching transition between 360-degree viewing scenes

[0071] The panoramic transition problem is a key point and difficulty in the panoramic roaming system. If users are given enough visual guidance, the transition effect will be natural and realistic, the authenticity of the system will be better, and the browsing experience will be better. If this problem cannot be solved well, users may not be able to understand the changes in spatial position and feel confused. Regarding this issue, the present application proposes two smooth switching transition solutions between 360-degree browsing scenes: dynamic panoramic VR and real-time panoramic sphere. The former forms a dynamic panoramic VR for panoramic transition by extracting the same-name control points to form a dynamic triangulated grid to update the original panoramic VR. The dynamic panorama is applied when the scene is switched to naturally and smoothly transition from the current site to the next panoramic site; the latter forms a dynamic triangulated grid through the same-name control points to cover the original panoramic sphere, and realizes the smooth transition of the panoramic scene through the deformation of the panoramic sphere itself.

[0072] (I) Solution A: Dynamic Panoramic VR

[0073] Generate a dynamic panorama VR with gradual transition, play the dynamic panorama during the panorama transition, and achieve natural scene transition. First, the input is two panorama VR (see Figure 4 ), the goal is to naturally transition from the first panoramic VR to the second panoramic VR. Considering the transition process from panoramic site a to panoramic site b, the panoramic VRs of the two panoramic sites are panoramic VRa( Figure 4 (a)) and panoramic VRb( Figure 4 (b)), denoted as I (a) and I (b) , in I (a) and I (b) Find N pairs of points P with the same name in the corresponding area (for example, two rectangles are formed by taking 60° on each side of the roaming direction) i , i∈{1,…,N}, the same-name point pair P i in I (a) The upper coordinate is P i (a) =(x i (a) ,y i (a) ), in I (b) Upper coordinate P i (b) =(xi (b) ,y i (b) ), {P i} Form a point set {P i (a)}, which is conducive to the Delaunay triangulation algorithm to maintain the grid structure cluster, and the point set {P i (a)}Use Delaunay triangulation to triangulate into M triangles T j (a) , j∈{1,…,M};

[0074] For each triangle T j (a) =△ABC, its three vertices are A=P j1 (a) , B=P j2 (a) and C = P j3 (a) , A, B, C are all from the point set {P i (a)}, j1, j2, j3∈{1,…,N}, find the points in I according to the same name. (b) The corresponding point on the j1 (b) , B=P j2 (b) and C = P j3 (b) , A', B', C' form a triangle T j (b) =△A'B'C';

[0075] For each T j (b) , generate a triangular face Face j , its vertices are initialized to P1 = A, P2 = B and P3 = C, and I is added (a) and I (b) Two texture maps, the transparency of the former is set to TRANSPARENT (I (a) )=0 (fully visible), the latter transparency is set to TRANSPARENT(I (b) )=1 (completely invisible), triangular face Face j The six UV texture coordinates of the three vertices are taken from A, B and C in I (a) The texture coordinates of A', B' and C' in I (b) The texture coordinates of

[0076] Assume parameter α is the progress of the scene transition from panoramic site a to panoramic site b. When α = 0, the panoramic roaming scene a is displayed; when α = 1, the panoramic roaming scene b is displayed. α is gradually adjusted over time. The process of α changing from 0 to 1 is the process of the panoramic roaming scene transition. The transition scheme is described as follows: at any moment of scene transition, the triangular face Face j The coordinates of the three vertices are P1 = LERP (A, A', α), P2 = LERP (B, B', α) and P3 = LERP (C, C', α), where LERP is a linear interpolation function that satisfies LERP (f, t, α) = (1-α) × f + α × t. At the same time, the transparency of the texture on the triangle surface satisfies TRANSPARENT (I a )=α、TRANSPARENT(I b )=1-α, the triangle patch is covered on the original texture and resampled to form a new texture frame. These texture frames are a dynamic texture that can be used for natural transition of panoramic scenes as a whole.

[0077] 1. Get the control point with the same name

[0078] The feature point extraction operator and matching algorithm are used to extract pairs of points with the same name. Although many feature points are extracted and matched by the SIFT feature point operator, there are many problems with these points: 1) The extracted feature points are unevenly distributed and deviate from the central direction of the scene; 2) These points are not what the human eye pays attention to. For example, many feature points are extracted in the door frame that the human eye does not pay attention to, but only one feature point is extracted on the door frame. Because the feature extraction operator cannot extract high-quality control points, manual methods have to be used to do this part of the work. Use a customized same-name point editor to open the two panoramas, and add some same-name control point pairs using the mouse and keyboard. In the editor, the same-name points of the two panoramas are connected. This editor provides basic same-name control point loading, saving, adding, deleting, and editing operations. When manually adding same-name control point pairs, the visual focus of the user when browsing the panorama should be considered.

[0079] 2. Triangulation seamless transition

[0080] Place the control point with the same name in I (a) The point set {P i (a)}Triangulation, a triangular face Face transitions to another triangular face. When a triangle △ABC transitions to position △A'B'C', at time α, the three vertex coordinates P1, P2 and P3 of Face are linear interpolations of A and A', B and B', C and C' respectively. At the same time, the color transition of the internal points is achieved through Interpolation is obtained, where is the pixel value with the centroid coordinates (i, j) on △ABC, is the pixel value with the centroid coordinates (i, j) on △A'B'C', for {T j Each triangle in} generates a triangular patch and records the texture at each α moment, recorded as Generated from I (a) To I (b) A series of texture frames Create a dynamic texture Playing dynamic textures on the panoramic sphere can achieve the effect of natural scene transition.

[0081] 3. Effect and evaluation

[0082] The above method is used to sample and generate a 4096×2048 dynamic texture. The effect of panoramic browsing is as follows: Figure 5 The first column is the α value, and the second column is the triangulation effect of the control point with the same name. When the α value changes from 0 to 1, the triangle {T j (a)}Gradually towards {T j (b)} transition, each triangle is covered on the original texture image, the pixels below are updated to form a new texture, and this new texture is mapped to the panoramic sphere, which is the observed scene switching process, as shown in the third column of the figure. As a reference, the fourth column is the ordinary transition effect using a simple gradient. It can be seen that compared with simple gradients, dynamic textures have no overlapping ghosts. Whether it is the lamp above, the picture on the left wall, the table on the ground, or the equipment on the right, they all move slowly to the side. The view is naturally stretched and enlarged, giving people a natural feeling of walking and observing. The use of dynamic textures can indeed achieve the effect of a natural transition scene.

[0083] However, the dynamic panoramic VR solution has potential performance issues. After α changes, it needs to be re-rendered to form a new panoramic texture. The amount of calculation in this process (triangulation of the point set and linear interpolation of the vertex positions) is not large, but the on-site rendering of the Canvas texture consumes a lot of memory, and applying the updated Canvas texture to the panoramic sphere also has a certain performance burden. Furthermore, rendering dynamic panoramic VR requires a separate rendering loop, which competes for resources with the main rendering loop for panoramic viewing. Therefore, under the dynamic panoramic VR solution, it is very likely to be unsmooth or even stuck; in mobile browsers, it may also freeze due to insufficient memory and video memory.

[0084] In summary, compared with jump, gradient, stretching, interference and parallax methods, dynamic panoramic VR can achieve extremely natural scene transition switching with only panoramic data, and is an effective solution for seamless transition of indoor 360-degree browsing scenes.

[0085] (II) Solution B: Real-time panoramic ball

[0086] Use the same-name control point pairs to build matching triangles, and perform linear transitions of position and color based on the same-name points to form each frame of the dynamic texture. Then apply these texture frames on the panoramic sphere to achieve natural panoramic scene transitions.

[0087] The biggest problem with the dynamic panoramic VR solution is that it needs to constantly redraw new textures and re-map them. This process takes up a lot of memory and may cause the frame rate to drop during the transition process, or even cause problems such as unsmoothness and freezes. The real-time panoramic sphere solution avoids the process of regenerating textures and re-mapping by moving the nodes associated with the panoramic sphere.

[0088] Based on the dynamic texture method, the control point pairs with the same name are generated, and {P i (a)} and {P i (b)}, based on the dynamic texture method to generate triangulated networks, generate {T j (a)}, for each triangle T j (a) ∈{T j (a)}, find the nodes in this triangle on the panoramic sphere (the definition of nodes refers to Figure 1 (a)), assuming there are K, set {V k (a,j)}, where k∈{1,…,K}, for each node V k (a,j) , find its j (a) = Coordinates of the barycentric coordinate system in △ABC And find I (b) The corresponding triangle T j (b) =△A'B'C's same centroid coordinates Cartesian coordinates of the point V k (b,j) , determine a node V on the panoramic sphere k (a,j) The deformation target position V k (b,j) In the dynamic texture, the position of △ABC gradually transitions to △A'B'C'. In the real-time panoramic ball, V k (a,j) The position gradually transitions to V k (b,j) .

[0089] At the panorama sphere node V k (a,j) The geometric position gradually transitions to V k(b,j) When the texture is deformed, another texture is generated for the original panoramic sphere. This texture uses the panoramic VRI (b) , the texture coordinates remain unchanged, the node V k (a,j) There are two texture coordinates (u (a，j，k) , v (a，j，k) ) and (u (b，j，k) , v (b，j，k) ) According to V k (a,j) The target position V k (b,j) Calculate the new texture coordinates and assign them to the second texture coordinates, that is lonlat2uv converts the spherical longitude and latitude coordinates into texture UV coordinates. Similarly, when moving the nodes, the transparency of the two sets of textures of the panoramic sphere needs to be changed accordingly;

[0090] However, after actual testing, it was found that this real-time panoramic ball solution had some irreconcilable loopholes. Figure 6 (a), each node correctly transitions from the initial position to the target position, but the panoramic sphere is partially folded, which results in the failure of color fusion and an unnatural effect. In addition, the door frame is also bent and distorted, reflecting that this solution cannot effectively maintain the Manhattan characteristics of common three-dimensional objects. The reason is that if the panoramic sphere is directly operated, some nodes will be pulled apart, which will inevitably lead to the folding and overlapping of the surface (because the view is basically zoomed in when the panoramic roaming site is switched); and when selecting the same-name control point pairs, it is assumed that the triangular facets remain unchanged as a whole when the image is deformed, but the triangles formed by the nodes on the panoramic sphere are not overlapped with the original triangles. Operating the view in this way cannot maintain the structural characteristics of objects in real space.

[0091] Improvement 1: Introducing triangular patches in three-dimensional space

[0092] Although the real-time panoramic ball solution has some major problems, it avoids the real-time rendering of panoramic VR. Now, based on the above problems, we introduce triangular patches in three-dimensional space based on the dynamic panoramic texture generation process: for each triangle T j (a) ∈{T j (a)}, Generate triangular face Face j , and apply I on it according to the triangle position (a) and I (b) Two panoramic VRs are used as texture maps, and then the triangular face is mapped according to the control points of the same name. j The position of T j (a) Transition to T j(b) The color interpolation ratio of the two textures is gradually changed accordingly to achieve a natural transition effect; the position transition of the triangle facets in the dynamic panoramic VR solution adopts LERP linear interpolation, and the position transition of the triangle facets in the dynamic panoramic solution in three-dimensional space adopts SLERP spherical interpolation;

[0093] The visual effects of the dynamic panoramic VR and real-time panoramic sphere solutions are almost the same when α≥0.2. Only when α<0.2, the left door frame of the real-time panoramic sphere solution has obvious deformation, while the dynamic panoramic VR solution is relatively normal.

[0094] Improvement 2: Using finely enhanced triangle patches

[0095] The triangles are further subdivided and more detailed texture mapping is performed to reduce distortion; for each edge of each triangle, let n be the number of nodes on the edge. When n = 2, there is no need to interpolate other nodes (such as Figure 7 (a); When n>2, some nodes need to be interpolated on the edges and inside of the triangle patch to form a triangle patch (node ​​interpolation can be conveniently performed in the barycentric coordinate system). Figure 7 It can be seen that as the triangular facets are continuously subdivided, the facets fit the panoramic sphere more closely and appear more refined. However, the test found that when viewing the transition effect from the center of the panoramic sphere, the visual effect is slightly improved only when a < 0.2.

[0096] Improvement 3: Using refined control points with the same name

[0097] The real-time panoramic sphere solution solves the performance bottleneck problem caused by the dynamic panoramic VR solution that requires continuous rendering of dynamic panoramic textures.

[0098] 1. Effect and evaluation

[0099] Both dynamic panoramic VR and real-time panoramic sphere achieve relatively natural panoramic scene transition effects. Both require the same-name control point pairs to guide the transition and deformation of triangular facets. The former achieves natural transition by playing dynamic panoramic VR, while the latter achieves similar effects by dynamically adjusting the panoramic sphere. Compared with the common transition processing in panoramic roaming systems, these two solutions have obvious advantages in visual effects.

[0100] Compared with the real-time panoramic ball solution, the dynamic panoramic VR solution is intuitive in theory and easy to implement. Although the real-time rendering of dynamic panoramic textures will take up a lot of memory, affect the redrawing frame rate of the view during the transition period, and may cause unnatural or even stuck phenomena, if the texture generation is completed in the server background, it can reduce the calculation and memory pressure of the browser front end, and can also hide the implementation method of the panoramic natural scene transition from the user to avoid plagiarism.

[0101] Compared with the dynamic panoramic VR solution, the real-time panoramic sphere does not need to dynamically generate texture maps for each frame and redraw the panoramic sphere, which has great performance advantages, and the effect is basically the same as the dynamic texture solution. Because there is no need to render textures, one rendering cycle is reduced. The real-time panoramic sphere solution has the characteristics of high redrawing frame rate, small memory usage, and smoother scene transition. However, compared with the dynamic panoramic VR solution, the real-time panoramic sphere solution needs to operate the spatial position, shape and texture of multiple triangles in real time, and the interpolation of the vertex position of the triangles also needs to use SLERP, which makes the code implementation of the real-time panoramic sphere solution more difficult. Finally, the real-time panoramic sphere solution is more sensitive to the selection of control point pairs with the same name. Improper control point selection may cause distortion of objects in the scene and affect the visual effect when switching between scene roaming sites.

[0102] 3. 360-degree Indoor Dynamic Full-Scene Seamless Roaming Prototype System

[0103] JavaScript is used for code implementation, Makefile is used for project management, and plain text JSON files and panoramic VR are directly placed in the file system for data access, instead of using back-end servers and databases.

[0104] (I) Single-frame application based on Three.js

[0105] The prototype system is built using the 3D display library Three.js based on JavaScript and WebGL. The steps include:

[0106] 1) Create a new scene;

[0107] 2) Add some 3D mesh objects to the scene, including the 3D object panoramic sphere in the indoor panoramic system, which is a 3D sphere with panoramic VR attached;

[0108] 3) Add a camera to the scene;

[0109] 4) The renderer uses the camera as a viewport to render the 3D scene into a 2D image and presents it on the Canvas element in the page;

[0110] (II) Prototype system engineering management based on Makefile

[0111] 1) Determine the final output file (target) of the prototype system, including the main interface index.html of this prototype system;

[0112] 2) Determine the target's dependencies, including the JavaScript library and main.js and main.css.

[0113] 3) Clarify the processing flow from dependencies to targets, including directly copying index.html to the target folder, compressing and mixing JavaScript code, and then outputting it to the target file;

[0114] After writing the project processing flow through Makefile, monitor the changes of related files in the folder, automatically re-make when the file is modified, and update the effect in real time.

[0115] (II) Full-scenario data processing flow

[0116] To extract the bounding box model from the panoramic VR, you need to find the coordinates of the eight corners in the panoramic VR, such as Figure 8 The steps to generate the bounding box model are: 1) Manually determine the coordinates of the eight corner points p i , i∈{1,…,8}; 2) Determine the ground and ceiling based on the relative comparison between the panoramic VR zenith direction and the camera height in the room; 3) The eight corner points p i The ground and ceiling projected into the three-dimensional space correspond to P i ,i∈{1,…,8}; 4) According to P i Generate six planes: front, back, left, right, top, and bottom, and project the panoramic VR onto these six planes to form a bounding box model;

[0117] The site information is organized in JSON text, which records the information of each panoramic site in a scene, including: 1) ID; 2) Name; 3) Introduction; 4) Panoramic VR path; 5) Panoramic posture information (north direction and zenith direction); 6) Other roaming points that can be jumped to;

[0118] The data processing flow chart of the indoor panoramic roaming system with 360-degree seamless scene transition is shown in Fig. 9 . Including: 1) Use PTGui to stitch the original images into a panoramic VR with an aspect ratio of 2:1 using PlateCarree projection; 2) Manually locate the corner points in the panoramic VR, generate a bounding box model (due to the lack of 3D data), and use it as a roaming environment model; 3) Manually determine the control point pairs with the same name between the roamable sites and write them into the configuration file;

[0119] A series of original images are transformed into panoramic roaming data that can be used by the front end of the 360-degree panoramic roaming system. By loading the panoramic VR at the roaming site, you can browse the panoramic site freely; jumping out of the panorama, you can also browse the three-dimensional environment of the roaming point; between panoramic sites, you can also load the switching configuration (the same-name control point pair) and achieve a seamless and natural transition of the panoramic scene through the dynamic panoramic texture solution or the real-time panoramic ball solution.

[0120] 3. Compression of Panoramic VR

[0121] Optimizing the volume of panoramic VR data is a very meaningful part of the panoramic system. Reducing the size of image files without reducing image quality provides smoother loading.

[0122] Panoramic VR is stored in JPEG format to reduce file size. When saving panoramic VR, the panoramic VR capacity is compressed by setting JPEG saving parameters. First, the same image is stored in Progressive JPEG and Baseline JPEG file formats respectively. The difference in file size between the two is linearly related to the image size: 1) When the original image is around 10KB, the file size of the output image under the two saving methods is similar; 2) When the original image is larger than 10KB, the file size is smaller when saved in Progressive mode (about 94% probability). The size of panoramic VR is significantly larger than 10KB, and Progressive saving mode is used to reduce the file size; secondly, blurred images have a higher compression rate. Appropriate Gaussian blurring of the original image can significantly reduce the file size under the same settings, which is especially suitable for situations where the image quality itself is not high; finally, there are comments or Exif tag information in the JPEG image, which are directly removed.

[0123] (IV) Panoramic VR layered slicing ensures code reusability

[0124] In some application scenarios that require high resolution of panoramic scenes, it is not advisable to improve the loading speed of panoramic textures and the smoothness of panoramic rendering by reducing the quality of panoramic VR and reducing the file size. At this time, the original panoramic VR with higher resolution should be collected and spliced, and layered and sliced. When browsing the panoramic site, first load the relatively blurry panoramic VR, and then gradually load the clearer image slices suitable for the current field of view. Such a strategy is bound to affect the code implementation of the panoramic viewing tool. Once the original panoramic VR resolution is adjusted, or the layering and slicing methods are changed, the source code of the panoramic viewing tool needs to be modified accordingly: recalculate the spherical range corresponding to each slice, remap and render. Such high coupling is not conducive to code integration and project maintenance.

[0125] This application no longer considers how to update the new slice to the new position of the panoramic sphere, but generates a new panoramic VR. Through the CanvasRenderer of Three.js, the Canvas element on the HTML page is used as a texture map to the panoramic sphere. As long as the virtual panoramic VR on the Canvas element is updated after obtaining the new slice, the global rendering is redrawn to achieve the effect of updating a part of the spherical surface on the panoramic sphere. The specific steps are: 1) Write a script to layer and slice a high-definition panoramic VR according to certain rules to obtain a series of sub-images; 2) Write the corresponding virtual panoramic VR adaptation code according to the above rules, which is responsible for the drawing and updating of a Canvas element; 3) Bind the user's field of view changes to the adapter, and update the virtual panoramic VR in time according to the changes in the field of view.

Claims

1. A 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method, characterized in that: To solve the problem of seamless transition between 3D models and panoramic VR, an additional temporary observation camera is introduced to perform linear and nonlinear interpolation from the initial state to the target state to achieve the purpose of connecting the two heterogeneous interaction modes; to solve the problem of seamless transition between panoramic sites, the same-name control points are introduced to guide the deformation process to make the visual effect more natural, and the real-time panoramic ball solution effect and the dynamic panoramic VR solution are integrated to transition the panoramic view; 1) Loading and fusion method of panoramic VR and three-dimensional field model: Establish a three-dimensional field model as a reference framework, set up panoramic sites in the three-dimensional field model, and naturally connect the three-dimensional field model with the two-dimensional panoramic roaming; expand the panoramic roaming system by introducing the three-dimensional model of the scene, so that the roaming system can not only browse various panoramic roaming sites, but also jump out of the limitations of the panoramic ball and tour the entire three-dimensional space, connecting the two heterogeneous browsing interaction modes of panoramic roaming and model browsing, and establish a solution for seamless transition switching by generating a temporary observation camera and interpolating its spatial posture; 2) Smooth switching transition between 360-degree browsing scenes: including dynamic panoramic VR and real-time panoramic sphere solutions. Dynamic panoramic VR extracts the same-name control points to form a dynamic triangulated grid to update the original panoramic VR to form a dynamic panoramic VR for panoramic transition. Dynamic panorama is applied when switching scenes to smoothly transition from the current site to the next panoramic site. The real-time panoramic sphere forms a dynamic triangulated grid through the control points of the same name and covers the original panoramic sphere. The panoramic sphere's own deformation is used to achieve a smooth transition of the panoramic scene. In the dynamic panoramic VR and real-time panoramic sphere solutions, objects in the scene slowly move to the side, and the view is naturally stretched and enlarged, giving people a feeling of natural walking and observation, achieving a natural transition. 3) 360-degree indoor dynamic full-scene seamless roaming prototype system: A single-frame application based on Three.js, using JavaScript for code implementation, Makefile for project management, plain text JSON and panoramic VR file system for data access, establishing a full-scene data processing flow, and constructing panoramic VR compression and panoramic VR layered slicing methods to ensure code reusability.

2. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 1 is characterized in that: 3D field model as reference framework: If the indoor panoramic roaming system does not need to be linked with other systems, its main reference framework, i.e. the world coordinate system, is directly selected as the building or several rooms of the roaming location. First, the 3D field model is loaded into the roaming system. The data formats of different model data are different, but the information provided is consistent. The model data is a combination of geometric structure and material. The former records the position information of each node of the model, and the latter records the texture map of the model. When loading, it is parsed according to the corresponding data specifications and the model is drawn in three-dimensional space. First, a local coordinate system is generated to load the points, lines, and surfaces in the model data. For points, it is important to correctly render the size and color of the points; for lines, one-dimensional textures need to be specified to make color transitions; for surfaces, the texture and texture coordinates of each node need to be specified. Finally, if the coordinate system of the model data has a different rotation from the coordinate system used by the rendering system, one of the directions needs to be reversed. If the model is not directly used as a world coordinate, the position and posture of the model also need to be specified.

3. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 1 is characterized in that: The 3D field model is naturally connected to the 2D panoramic roaming: many cameras are retained in the 3D environment, and the 3D field model is naturally connected to the 2D panoramic roaming by transitioning between the cameras; Assume that the panoramic site P is in the model M, and the camera C p and C m Observation, C. m The current observation result is roughly in the positive direction of the x-axis. The sphere in the center is the panoramic sphere where the panoramic site P is located. p The observation results are roughly towards the negative direction of the x-axis. When the user clicks the panoramic ball, the viewing angle changes from C m The current state gradually transitions to C p The current state of the system and make corresponding adjustments in the interaction mode; The following logic is introduced to make the transition more natural: First, a temporary observation camera C is introduced x Used as a transition, during which the user observes C x When the user starts to switch from model viewing mode to panoramic viewing mode, C x Start with C m Get the pose and camera configuration, C x The observations are consistent with C m Consistent, at the same time, when enabling C x After that, the original C m The interaction mode is invalid. At this time, it no longer responds to the user's mobile viewing operation, and the user interface enters the automatic roaming state. x From the initial state C m Towards the target state C p The transition from model to panorama is completed, and the posture is expressed by quaternion in C m and C p Interpolating SLERP between the two poses yields the camera orientation at each moment.

4. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 1 is characterized in that: Solution A for smooth switching transition between 360-degree browsing scenes: Dynamic panoramic VR: Generate a dynamic panoramic VR with gradual transition, play the dynamic panorama during the panorama transition, and realize natural scene transition; First, the input is two panoramic VRs. The goal is to transition naturally from the first panoramic VR to the second panoramic VR. Consider the transition process from panoramic site a to panoramic site b. The panoramic VRs of the two panoramic sites are panoramic VRa and panoramic VRb, respectively, denoted as I (a) and I (b) , in I (a) and I (b) Find N pairs of points P with the same name in the corresponding area i , i∈{1,…,N}, the same-name point pair P i in I (a) The upper coordinate is P i (a) =(x i (a) ,y i (a) ), in I (b) Upper coordinate P i (b) =(x i (b) ,y i (b) ), {P i } Form a point set {P i (a) }, which is conducive to the Delaunay triangulation algorithm to maintain the grid structure cluster, and the point set {P i (a) }Use Delaunay triangulation to triangulate into M triangles T j (a) , j∈{1,…,M}; For each triangle T j (a) =△ABC, its three vertices are A=P j1 (a) , B=P j2 (a) and C = P j3 (a) , A, B, C are all from the point set {P i (a) }, j1, j2, j3∈{1,…,N}, find the points in I according to the same name. (b) The corresponding point on the j1 (b) , B=P j2 (b) and C = P j3 (b) , A', B', C' form a triangle T j (b) =△A'B'C'; For each T j (b) , generate a triangular face Face j , its vertices are initialized to P1 = A, P2 = B and P3 = C, and I is added (a) and I (b) Two texture maps, the transparency of the former is set to TRANSPARENT (I (a) )=0, the latter transparency is set to TRANSPARENT(I (b) )=1, triangular face j The six UV texture coordinates of the three vertices are taken from A, B and C in I (a) The texture coordinates of A', B' and C' in I (b) The texture coordinates of Assume parameter α is the progress of the scene transition from panoramic site a to panoramic site b. When α = 0, the panoramic roaming scene a is displayed; when α = 1, the panoramic roaming scene b is displayed. α is gradually adjusted over time. The process of α changing from 0 to 1 is the process of the panoramic roaming scene transition. The transition scheme is described as follows: at any moment of scene transition, the triangular face Face j The coordinates of the three vertices are P1 = LERP (A, A', α), P2 = LERP (B, B', α) and P3 = LERP (C, C', α), where LERP is a linear interpolation function that satisfies LERP (f, t, α) = (1-α) × f + α × t. At the same time, the transparency of the texture on the triangle surface satisfies TRANSPARENT (I a )=α、TRANSPARENT(I b )=1-α, the triangle patch is covered on the original texture and resampled to form a new texture frame. These texture frames are a dynamic texture that can be used for natural transition of panoramic scenes as a whole.

5. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 4 is characterized in that: Get control points with the same name: extract pairs of points with the same name through feature point extraction operators and matching algorithms, and conduct manual review. Open two panoramas with a customized editor for control points with the same name, and add some pairs of control points with the same name through the mouse and keyboard. In the editor, the points with the same name in the two panoramas are connected. This editor provides basic operations for loading, saving, adding, deleting, and editing control points with the same name. When manually adding pairs of control points with the same name, the visual focus of the user when browsing the panorama should be considered.

6. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 4 is characterized in that: Triangulation seamless transition: Move the control points with the same name to the (a) The point set {P i (a) }Triangulation, a triangular face Face transitions to another triangular face. When a triangle △ABC transitions to position △A'B'C', at time α, the three vertex coordinates P1, P2 and P3 of Face are linear interpolations of A and A', B and B', C and C' respectively. At the same time, the color transition of the internal points is achieved through Interpolation is obtained, where is the pixel value with the centroid coordinates (i, j) on △ABC, is the pixel value with the centroid coordinates (i, j) on △A'B'C', for {T j Each triangle in} generates a triangular patch and records the texture at each α moment, recorded as Generated from I (a) To I (b) A series of texture frames Create a dynamic texture Playing dynamic textures on the panoramic sphere can achieve the effect of natural scene transition.

7. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 1 is characterized in that: Solution B for smooth switching transition between 360-degree browsing scenes: Real-time panoramic sphere: Use the same-name control point pairs to build matching triangles, and perform linear transitions of position and color based on the same-name points to form each frame of dynamic texture. These texture frames are then applied to the panoramic sphere to achieve natural panoramic scene transitions; The real-time panoramic sphere solution avoids the process of regenerating textures and remapping by moving the nodes associated with the panoramic sphere; Based on the dynamic texture method, the control point pairs with the same name are generated, and {P i (a) } and {P i (b) }, based on the dynamic texture method to generate triangulated networks, generate {T j (a) }, for each triangle T j (a) ∈{T j (a) }, find the nodes on the panoramic sphere within this triangle, assuming there are K nodes, set as {V k (a,j) }, where k∈{1,…,K}, for each node V k (a,j) , find its j (a) = Coordinates of the barycentric coordinate system in △ABC And find I (b) The corresponding triangle T j (b) =△A'B'C's same centroid coordinates Cartesian coordinates of the point V k (b,j) , determine a node V on the panoramic sphere k (a,j) The deformation target position V k (b,j) In the dynamic texture, the position of △ABC gradually transitions to △A'B'C'. In the real-time panoramic ball, V k (a,j) The position gradually transitions to V k (b,j) ; At the panorama sphere node V k (a,j) The geometric position gradually transitions to V k (b,j) When the texture is deformed, another texture is generated for the original panoramic sphere. This texture uses the panoramic VRI (b) , the texture coordinates remain unchanged, the node V k (a,j) There are two texture coordinates (u (a，j，k) , (a，j，k) and(u (b，j，k) , (b，j，k) ) According to V k (a,j) The target position V k (b,j) Calculate the new texture coordinates and assign them to the second texture coordinates, that is lonlat2uv converts the spherical longitude and latitude coordinates into texture UV coordinates. Similarly, when moving the nodes, the transparency of the two sets of textures of the panoramic sphere needs to be changed accordingly.

8. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 7 is characterized in that: Improvement 1: Introducing triangular patches in three-dimensional space: Based on the dynamic panoramic texture generation process, triangular patches are introduced in three-dimensional space: for each triangle T j (a) ∈{T j (a) }, Generate triangular face Face j , and apply I on it according to the triangle position (a) and I (b) Two panoramic VRs are used as texture maps, and then the triangular face is mapped according to the control points of the same name. j The position of T j (a) Transition to T j (b) The color interpolation ratio of the two textures is gradually changed accordingly to achieve a natural transition effect; the position transition of the triangle facets in the dynamic panoramic VR solution adopts LERP linear interpolation, and the position transition of the triangle facets in the dynamic panoramic solution in three-dimensional space adopts SLERP spherical interpolation; Improvement 2: Use finely enhanced triangles: further subdivide the triangles and perform more detailed texture mapping to reduce distortion; for each edge of each triangle, let n be the number of nodes on the edge. When n=2, there is no need to interpolate other nodes; when n>2, it is necessary to interpolate some nodes on the edges and inside of the triangles to form triangles.

9. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 1 is characterized in that: Full scene data processing flow: To extract the bounding box model from the panoramic VR, it is necessary to find the coordinates of the eight corners in the panoramic VR. The steps to generate the bounding box model are as follows: 1) Manually determine the coordinates of the eight corner points p i , i∈{1,…,8}; 2) Determine the ground and ceiling based on the relative comparison between the panoramic VR zenith direction and the camera height in the room; 3) The eight corner points p i The ground and ceiling projected into the three-dimensional space correspond to P i ,i∈{1,…,8}; 4) According to P i Generate six planes: front, back, left, right, top, and bottom, and project the panoramic VR onto these six planes to form a bounding box model; The site information is organized in JSON text, which records the information of each panoramic site in a scene, including: 1) ID; 2) Name; 3) Introduction; 4) Panoramic VR path; 5) Panoramic posture information (north direction and zenith direction); 6) Other roaming points that can be jumped to; The data processing flow of the indoor panoramic roaming system with 360-degree seamless scene transition includes: 1) using PTGui to stitch the original images into a panoramic VR with an aspect ratio of 2:1 using PlateCarree projection; 2) manually locating the corner points in the panoramic VR, generating a bounding box model, and making a roaming environment model; 3) manually determining the control point pairs with the same name between the roamable sites and writing them into the configuration file.

10. The 360-degree indoor dynamic VR panoramic roaming browsing seamless transition method according to claim 1 is characterized in that: Panoramic VR layered slicing ensures code reusability: generate a new panoramic VR, and use the CanvasRenderer of Three.js to map the Canvas element on the HTML page as a texture to the panoramic sphere. After obtaining the new slice, update the virtual panoramic VR on the Canvas element, and then globally render and redraw to achieve the effect of updating a part of the spherical surface on the panoramic sphere. The specific steps are: 1) Write a script to layer and slice a high-definition panoramic VR according to certain rules to obtain a series of sub-images; 2) Write the corresponding virtual panoramic VR adaptation code according to the above rules, which is responsible for the drawing and updating of a Canvas element; 3) Bind the user's field of view changes to the adapter, and update the virtual panoramic VR in time according to the changes in the field of view.

Images