A method for generating a VR panoramic image

Abstract

The present application provides a method for generating a VR panoramic image, which relates to the field of VR technology and includes: obtaining 6 RGB-D images at the same moment and the same shooting position, wherein the shooting angle between two adjacent RGB-D images is 60°; performing elastic grid modeling on two RGB-D images with adjacent shooting angles, aligning the overlapping regions, and splicing and fusing the overlapping regions to determine a panoramic image; generating a VR panoramic image based on the panoramic image corresponding to each shooting position. This solution introduces depth information at a low cost, combines 2D image information, and makes full use of existing integrated processing solutions (such as SIFT-3D, FLANN, RANSAC, etc. implemented after extending other libraries in OpenCV) to extract and match basic feature information, improves the splicing method of VR panoramic images, realizes the splicing and fusion of panoramic images, and improves the generation quality of VR panoramic images.

Description

Technical Field

[0001] This application relates to the field of VR technology. Specifically, it relates to a method for generating VR panoramic images. Background Art

[0002] The core of VR technology lies in the generation of VR panoramas. Currently, the VR panorama generation technology based on multi-view images has become the core means for constructing immersive scenes. The existing technology mainly realizes panoramic image generation through the following two types of solutions:

[0003] Multi-view registration technology based on laser point cloud: Obtain the 3D point cloud of the scene through a lidar, align the multi-view data by combining the ICP algorithm, and then map the texture to generate a panoramic image. However, this solution relies on high-precision lidar and synchronous control devices, making it difficult to popularize and apply. When the point cloud density is insufficient, the texture mapping in the far-distance area is blurred, and specular reflection materials cannot be processed.

[0004] RGB image stitching technology based on multi-camera array: Collect images through multiple surrounding RGB cameras (such as a six-eye camera array), and complete stitching using feature point matching (such as SIFT, ORB) and cylindrical / spherical projection. However, this technology currently only relies on 2D image information and lacks depth perception, resulting in non-rigid deformation (such as building edge distortion) in the stitching overlapping area that cannot be effectively corrected, and serious geometric distortion.

[0005] Therefore, how to improve the generation quality of current VR panoramic images at a low cost is a problem that needs to be solved in this field. Summary of the Invention

[0006] The purpose of the embodiments of this application is to provide a method for generating VR panoramic images, introduce depth information at a low cost, combine 2D image information, make full use of existing integrated processing solutions (such as modules like SIFT-3D, FLANN, RANSAC implemented after extending other libraries in OpenCV) to extract and match basic feature information, improve the stitching method of VR panoramic images, realize the stitching and fusion of panoramic images, and improve the generation quality of VR panoramic images.

[0007] To achieve the above purpose, the embodiments of this application are implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides a method for generating a VR panoramic image, including: obtaining six RGB-D images at the same moment and at the same shooting position, wherein the shooting angles of two adjacent RGB-D images are separated by 60°; performing elastic mesh modeling on two RGB-D images with adjacent shooting angles, aligning the overlapping regions, and splicing and fusing the overlapping regions to determine a panoramic image; generating a VR panoramic image based on the panoramic image corresponding to each shooting position. In a first aspect, an embodiment of the present application provides a method for generating a VR panoramic image, including: obtaining six RGB-D images at the same moment and at the same shooting position, wherein the shooting angles of two adjacent RGB-D images are separated by 60°; performing elastic mesh modeling on two RGB-D images with adjacent shooting angles, aligning the overlapping regions, and splicing and fusing the overlapping regions to determine a panoramic image; generating a VR panoramic image based on the panoramic image corresponding to each shooting position.

[0009] In combination with the first aspect, in the first possible implementation manner of the first aspect, performing elastic mesh modeling on two RGB-D images with adjacent shooting angles and aligning the overlapping regions includes: respectively extracting features from two RGB-D images with adjacent shooting angles to determine the feature points of each RGB-D image, and matching the feature points of the two RGB-D images to determine the matching feature point pairs; generating an adaptive double-layer elastic mesh for the two RGB-D images based on the feature point pairs, and projecting the grid vertices contained in each of the two RGB-D images onto a unified spherical coordinate system, wherein the adaptive elastic mesh includes a primary mesh and a secondary mesh, and the secondary mesh is located within the unit mesh region of the primary mesh; constructing a joint optimization energy function and using nonlinear optimization to solve it to align the overlapping regions of the two RGB-D images.

[0010] In combination with the first possible implementation manner of the first aspect, in the second possible implementation manner of the first aspect, the SIFT-3D algorithm is used to detect the feature points of two RGB-D images with adjacent shooting angles respectively to determine the feature points of each RGB-D image, and then the FLANN+RANSAC combination scheme is used for feature point matching to determine the matching feature point pairs in the two RGB-D images with adjacent shooting angles.

[0011] In combination with the first possible implementation manner of the first aspect, in the third possible implementation manner of the first aspect, generating an adaptive double-layer elastic mesh for the two RGB-D images based on the feature point pairs includes: generating a primary mesh in each of the two RGB-D images respectively; determining the target unit meshes from the primary meshes based on the number of feature point pairs covered in each primary mesh, and generating secondary meshes in each target unit mesh.

[0012] Combined with the third possible implementation of the first aspect, in the fourth possible implementation of the first aspect, a joint optimization energy function is constructed and solved by non-linear optimization to align the overlapping regions of two RGB-D images, including:

[0013] Construct a joint optimization energy function for the grid vertices in two RGB-D images: , where is the joint optimization energy function, , , and are the weights of the color alignment term, depth constraint term, deformation smoothing term, and boundary preservation term respectively, , , and are the color alignment term, depth constraint term, deformation smoothing term, and boundary preservation term respectively; use the LM algorithm to iteratively solve for the vertex displacement:

[0014] , where and are the optimization variable vectors after the -th iteration and the optimization variable vectors after the -th iteration respectively, and are the Jacobian matrix and its transpose matrix respectively, is the weight diagonal matrix, is the damping factor of the LM algorithm, means taking of the diagonal elements, is the residual vector; the iteration convergence condition of the LM algorithm is:

[0015] , where is the energy value after the -th iteration, is the energy value after the -th iteration, means taking the larger value of and 1, means the coordinate change amount of the -th grid vertex in the -th iteration, is the current iteration number, is the set maximum iteration number.

[0016] Combined with the fourth possible implementation of the first aspect, in the fifth possible implementation of the first aspect, the color alignment term satisfies: , where is the set of all grid vertices in the overlapping region, containing N grid vertices, is the coordinate of the th grid vertex, is the structural similarity metric operator, is the grid vertex of the first RGB-D image in two RGB-D images with adjacent shooting perspectives at the position of the color value, is the grid vertex of the second RGB-D image in two RGB-D images with adjacent shooting perspectives after geometric transformation at the projected position of the color value, the dynamic adjustment parameter satisfies: , where, is the function, represents the image gradient magnitude.

[0017] Combined with the fourth possible implementation manner of the first aspect, in the sixth possible implementation manner of the first aspect, the depth constraint term satisfies: , where, is the set of all grid vertices in the overlapping region, containing N grid vertices, is the confidence weight of the th grid vertex, is the grid vertex of the first RGB-D image in two RGB-D images with adjacent shooting perspectives at the position of the depth value, is the grid vertex of the second RGB-D image in two RGB-D images with adjacent shooting perspectives after geometric transformation at the projected position of the depth value, the confidence weight satisfies: , where, is to calculate the signal-to-noise ratio, is the distance from the grid vertex to the coordinate of the nearest boundary point.

[0018] Combined with the fourth possible implementation manner of the first aspect, in the seventh possible implementation manner of the first aspect, the deformation smoothing term satisfies: , where, represents the set of grid edges in the overlapping region, represents the grid edge length deformation amount, is the balance parameter, is a constant, taking 0.5, The set of mesh surfaces representing the overlapping region represents a mesh surface of the deformation Jacobian matrix is the Jacobian determinant, representing the area change rate of the mesh surface , and the length deformation amount satisfies: , where and are the coordinates of two mesh vertices constituting the mesh edge , and is a fixed value during mesh initialization.

[0019] Combined with the fourth possible implementation manner of the first aspect, in the eighth possible implementation manner of the first aspect, the boundary retention term satisfies: , where represents the coordinates of the deformed vertex of the th mesh vertex , is the initial coordinate of the th mesh vertex, is the regional dynamic weight, satisfying: , where represents the set of true overlapping mesh vertices in the overlapping region. A true overlapping mesh vertex means that the depth difference between the mesh vertex of the first RGB-D image in the overlapping region and the corresponding mesh vertex of the second RGB-D image in the overlapping region in two adjacent captured viewpoints does not exceed the threshold represents the set of transitional zone mesh vertices in the overlapping region. A transitional zone mesh vertex means that the depth difference between the mesh vertex of the first RGB-D image in the overlapping region and the corresponding mesh vertex of the second RGB-D image in the overlapping region in two adjacent captured viewpoints exceeds the threshold, but the interval between it and the nearest true overlapping mesh vertex does not exceed mesh vertices represents the set of non-overlapping mesh vertices in the overlapping region. A non-overlapping mesh vertex means that the depth difference between the mesh vertex of the first RGB-D image in the overlapping region and the corresponding mesh vertex of the second RGB-D image in the overlapping region in two adjacent captured viewpoints exceeds the threshold, but the interval between it and the nearest true overlapping mesh vertex exceeds mesh vertices.

[0020] ​​​Combined with the first aspect, in the ninth possible implementation manner of the first aspect, generating a VR panoramic image based on the panoramic images corresponding to each shooting point includes: converting the panoramic image of each shooting point into a cube map, performing adaptive bi-paraboloid projection correction on the close-range area, and then mapping the cube map to the spherical space in the global coordinate system based on the spatial pose parameters of the shooting point, where the depth of the close-range area does not exceed 5 meters; generating key points for each shooting point, constructing the spatial topological relationship of the shooting points based on the spatial coordinates of the shooting points, establishing an accessibility relationship graph with the key points as nodes, and defining the traffic logic for adjacent areas; performing post-processing optimization on the panoramic image and outputting the VR panoramic image, where the post-processing optimization includes dynamic parallax correction and motion blur compensation.

[0021] Beneficial effects: This solution obtains 6 RGB-D images at the same moment and the same shooting point (the RGB-D images of adjacent viewpoints are spaced 60° apart), performs elastic grid modeling on two adjacent RGB-D images of shooting viewpoints, aligns the overlapping areas, and splices and fuses the overlapping areas to determine the panoramic image, and then generates a VR panoramic image based on the panoramic images corresponding to each shooting point. This solution can introduce depth information into the generation of VR panoramic images at a relatively low cost, and fully utilize the existing integrated processing solutions for the extraction and matching of basic feature information (this solution uses SIFT-3D implemented by extending other libraries in OpenCV for feature point extraction, and uses a combination of FLANN and RANSAC compatible with SIFT-3D for the matching of feature point pairs), which can be achieved with only a small amount of code, greatly reducing the development workload. For the splicing of adjacent-view RGB-D images, which is the core part, by designing a double-layer elastic grid scheme, the density of feature points is used to determine whether to arrange secondary elastic grids within the primary grid cells to achieve high-quality splicing.

[0022] In the process of splicing process modeling, by constructing a joint optimization energy function, on the basis of the color alignment term and the deformation smoothing term, a depth constraint term is introduced, and the boundary preservation term is improved, thereby effectively reducing the mis-matching under different viewpoints and avoiding perspective distortion. And the weight of the color alignment term , the weight of the depth constraint term , the weight of the deformation smoothing term and the weight of the boundary preservation term can adjust the weight parameters according to different scenarios (such as indoor, outdoor, lighting, etc.) (such as increasing the weight of the color alignment term under strong light, increasing the weight of the depth constraint term in a weak texture scene, increasing the weight of the deformation smoothing term in a vast area, increasing the weight of the depth constraint term and the weight of the boundary preservation term in a scene involving complex structures ), to achieve dynamic adaptation to different scenarios. In the boundary retention term, by designing and dividing the real coincidence area, the transition zone area, and the pseudo-coincidence area, high-precision deformation adjustment can be maintained, the amount of deformation can be smoothed, and mis-matches (which are actually pseudo-coincidences, basically indistinguishable in 2D, but can be recognized by this solution) can be effectively recognized and corresponding optimization processing can be carried out, thereby achieving high-precision stitching and fusion. When using the LM algorithm for iterative solution, by combining gradient descent and the Gauss-Newton method, it can well adapt to the non-convexity of the joint optimization energy function and achieve robust and rapid convergence.

[0023] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically gives preferred embodiments and, in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0025] Figure 1 It is a flowchart of the method for generating a VR panoramic image provided by an embodiment of the present application.

[0026] Figure 2 Figure A is a schematic diagram of generating an adaptive double-layer elastic grid for one RGB-D image in two adjacent RGB-D images.

[0027] Figure 2 Figure B is a schematic diagram of generating an adaptive double-layer elastic grid for the other RGB-D image in two adjacent RGB-D images. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application.

[0029] Please refer to Figure 1 , Figure 1 It is a flowchart of the method for generating a VR panoramic image provided by an embodiment of the present application. In this embodiment, the method for generating a VR panoramic image may include step S10, step S20, and step S30.

[0030] For a scene that needs to construct an AR panoramic image, usually multiple shooting positions are required to present the scene as completely as possible. Therefore, it is necessary to plan and decompose the scene, determine multiple photography positions, and capture images and obtain depth data at each photography position as the basis for generating a VR panoramic image.

[0031] Accordingly, for each shooting point, step S10 can be executed.

[0032] Step S10: Obtain 6 RGB-D images at the same moment and the same shooting point, where the shooting angles of two adjacent RGB-D images are separated by 60°.

[0033] In this embodiment, for each shooting point, 6 RGB-D images at the same moment and the same shooting point need to be obtained, and the shooting angles of two adjacent RGB-D images are separated by 60°. For example, 6 RGB images can be captured by a six-eye camera, and at the same time, 6 three-dimensional scanners can perform depth scanning in a layout with an interval of 60° (depending on the field of view angle of different scanners, if the field of view angle is sufficient to cover 180°, 3 scanners arranged at an interval of 120° can be selected for depth scanning) to obtain depth information, and the depth information is registered with the corresponding RGB image and mapped into the RGB image to obtain 6 RGB-D images. Of course, six RGB-D cameras arranged at an interval of 60° can also be used for shooting, but the depth image information captured by the current RGB-D cameras is not sufficient, so in this embodiment, a combination of a six-eye camera and a three-dimensional scanner is used to obtain 6 RGB-D images. For example, the RGB-D image under the view angle of camera 1 contains things in the range of 0 to 120°, the RGB-D image under the view angle of camera 2 contains things in the range of 60 to 180°, the RGB-D image under the view angle of camera 3 contains things in the range of 120 to 240°, the RGB-D image under the view angle of camera 4 contains things in the range of 180 to 300°, the RGB-D image under the view angle of camera 5 contains things in the range of 240 to 360°, and the RGB-D image under the view angle of camera 6 contains things in the range of 300 to 60° (i.e., 300 to 420°).

[0034] After obtaining 6 RGB-D images, step S20 can be executed.

[0035] Step S20: Perform elastic mesh modeling on two RGB-D images with adjacent shooting angles, align the overlapping regions, and splice and fuse the overlapping regions to determine the panoramic image.

[0036] In this embodiment, the most crucial part and the core improvement part of this method is the image splicing and fusion to form the panoramic image corresponding to the shooting point. This process requires splicing and fusing two RGB-D images with adjacent shooting angles to achieve end-to-end connection, so as to obtain the panoramic image spliced and fused in sequence according to the view angles. The splicing process of every two adjacent RGB-D images with shooting angles is similar. Therefore, in this embodiment, the process of splicing and fusing two RGB-D images with adjacent shooting angles is taken as an example for introduction.

[0037] For two RGB-D images with adjacent shooting perspectives, feature extraction needs to be performed separately to determine the feature points of each RGB-D image, and the feature points of the two RGB-D images are matched to determine the matching feature point pairs. After testing, based on the existing OpenCV, there are suitable functions to complete this task (other libraries need to be extended to implement it). Therefore, this embodiment adopts this method, saving a lot of development time.

[0038] Exemplarily, this embodiment uses the SIFT-3D algorithm to detect the feature points of two RGB-D images with adjacent shooting perspectives respectively to determine the feature points of each RGB-D image, and then uses the FLANN+RANSAC combination scheme for feature point matching to determine the matching feature point pairs in the two RGB-D images with adjacent shooting perspectives.

[0039] After determining the matching feature point pairs in the two RGB-D images, an adaptive double-layer elastic grid can be further generated for the two RGB-D images based on the feature point pairs, and the grid vertices contained in each of the two RGB-D images are projected onto a unified spherical coordinate system. Among them, the adaptive elastic grid includes a primary grid and a secondary grid, and the secondary grid is located within the cell grid area of the primary grid.

[0040] Exemplarily, the primary grids can be generated in the two RGB-D images respectively.

[0041] In order to keep the primary grids generated in the two RGB-D images consistent in the overlapping area, it can be calculated in advance according to the camera parameters (mainly the field of view angle of the camera, taking 120° as an example) and the size of the RGB-D image. For example, taking the field of view angle of the camera as 120° as an example, assuming the image size is 1920×1080, then 640px on the left side of this RGB-D image is used for splicing with the previous RGB-D image (that is, the RGB-D image at the adjacent shooting perspective on its left), 640px on the right side of this RGB-D image is used for splicing with the subsequent RGB-D image (that is, the RGB-D image at the adjacent shooting perspective on its right), and the middle 640px presents the content of the current shooting perspective. Then, the primary grid can be designed as a 30*3*0 grid, and the size of each primary grid is 64×36.

[0042] As the primary grid, since the size of each RGB-D image is the same, the size of the primary grid is also the same. Subsequently, the number of feature point pairs within each primary grid in the first RGB-D image and the number of feature point pairs within each primary grid in the second RGB-D image can be counted. In the overlapping region (i.e., the 640×1080 on the right side of the first RGB-D image and the 640×1080 on the left side of the second RGB-D image), based on the number of feature point pairs covered in each primary grid obtained from the statistics, the target unit grids can be determined from the primary grids (for example, when the number of feature point pairs exceeds 8), and secondary grids can be generated in each target unit grid (for the secondary grids, they can be divided into 4*4, and the size of each secondary grid is 16×9). Here, it is necessary to synchronize the target unit grids in the overlapping region of the first RGB-D image with the target unit grids in the overlapping region of the second RGB-D image (understood as taking the union) to ensure the consistency of the secondary grids. Of course, the primary grid division and secondary grid division methods given here are only exemplary. If higher-precision stitching is pursued, higher-density primary grid and secondary grid division methods can be further set, which are not limited here.

[0043] In this way, the generation of the double-layer adaptive elastic grid is completed, as Figure 2 A and Figure 2 B shown. Subsequently, the grid vertices contained in the two RGB-D images can be projected onto a unified spherical coordinate system, and the method of mapping to the unified spherical coordinate (i.e., the conversion method between the pixel coordinate system, camera coordinate system, world coordinate system, and spherical coordinate system) will not be elaborated here.

[0044] Based on this, a joint optimization energy function can be constructed and solved using nonlinear optimization to align the overlapping regions of the two RGB-D images.

[0045] Exemplarily, a joint optimization energy function regarding the grid vertices in the two RGB-D images can be constructed:

[0046] , (1),

[0047] where is the joint optimization energy function, , , and are the weights of the color alignment term, depth constraint term, deformation smoothing term, and boundary preservation term respectively, , , and are the color alignment term, depth constraint term, deformation smoothing term, and boundary preservation term respectively.

[0048] Color alignment item weights , depth constraint weight , deformation smoothing weight and boundary-preserving term weights , can adjust the weight parameters according to different scenes (such as indoor, outdoor, lighting, etc.) (such as increasing the weight of color alignment items under strong light) , weak texture scene increases the weight of depth constraint , increase the weight of deformation smoothing term in wide areas , Increase the weight of depth constraints in scenes involving complex structures and boundary-preserving term weights ), to achieve dynamic adaptation to different scenarios.

[0049] Color alignment items satisfy:

[0050] , (2)

[0051] in, is the set of all mesh vertices in the overlapping area, including N mesh vertices (taking 30*30 primary mesh division and 4*4 secondary mesh division as an example, but the specific value needs to be determined according to the number of target unit meshes). For the The coordinates of the mesh vertices, is a structural similarity measurement operator (the calculation method is public and the operator is not adjusted in this embodiment), To dynamically adjust parameters, The mesh vertices of the first RGB-D image of two RGB-D images with adjacent shooting angles The color value of the location, The mesh vertices of the second RGB-D image of two RGB-D images with adjacent shooting angles After geometric transformation The color value of the projection position after the dynamic adjustment of parameters satisfy:

[0052] , (3)

[0053] in, for function, Represents the image gradient amplitude (here we need to calculate the first RGB-D image and the second RGB-D image in the The mean of the image gradient magnitudes at the grid vertices), where the parameter value of 0.7 is a relatively appropriate parameter set according to the current implementation project. In other projects, this parameter value can vary and can be adjusted according to the specific project scenario.

[0054] Depth constraint term Satisfy:

[0055] , (4)

[0056] Among them, Is the set of all grid vertices in the overlapping area, containing N grid vertices, Is the Confidence weight of the th grid vertex, Is the grid vertex of the first RGB-D image in two RGB-D images with adjacent shooting perspectives Depth value at the location, Is the grid vertex of the second RGB-D image in two RGB-D images with adjacent shooting perspectives After geometric transformation Depth value at the projected position.

[0057] And the confidence weight Satisfy:

[0058] , (5)

[0059] Among them, Is the coordinate of the boundary point closest to the grid vertex , For calculating the signal-to-noise ratio:

[0060] , (6)

[0061] Among them, Is the median, Is the median absolute deviation, Is the minimum number, take . And the attenuation factor:

[0062] , (7)

[0063] The attenuation factor is a Gaussian-type spatial attenuation factor used to dynamically adjust the weight of the depth constraint according to the distance from the vertex to the boundary.

[0064] In this embodiment, the deformation smoothing term Satisfy.

[0065] , (8)

[0066] Among them, represents the set of grid edges in the overlapping region, represents the grid edge The length deformation amount, is the balance parameter, which is a constant and takes 0.5, represents the set of grid faces in the overlapping region, represents the grid face The deformation Jacobian matrix, is the Jacobian determinant, representing the area change rate of the grid face It is obtained by calculating the ratio of the area of the deformed grid to the area of the grid before deformation. Greater than 1 indicates area expansion, equal to 1 indicates no area change, and less than 1 indicates area contraction.

[0067] The length deformation amount Satisfies:

[0068] , (9)

[0069] Among them, and are the coordinates of two grid vertices that make up the grid edge , is a fixed value during grid initialization (for example, the size of the primary grid is 64×36, and the size of the secondary grid is 16×9).

[0070] And the boundary retention term Satisfies:

[0071] , (10)

[0072] Among them, represents the deformed vertex coordinates of the th grid vertex , is the initial coordinate of the th grid vertex, is the regional dynamic weight, satisfying:

[0073] , (11)

[0074] Among them, represents the set of true overlapping grid vertices in the overlapping region. True overlapping grid vertices mean that the depth difference between the grid vertices of the first RGB-D image in the overlapping region and the corresponding grid vertices of the second RGB-D image in the overlapping region of two adjacent captured viewpoints does not exceed the threshold, The set of transitional zone grid vertices representing the overlapping region, where the transitional zone grid vertices indicate that the depth difference between the grid vertices of the first RGB-D image in the overlapping region and the corresponding grid vertices of the second RGB-D image in the overlapping region of two RGB-D images with adjacent shooting perspectives exceeds the threshold, but the interval between it and the nearest true overlapping grid vertex does not exceed grid vertices (in this embodiment, 2 is taken as an example), The set of non-overlapping grid vertices representing the overlapping region, where the non-overlapping grid vertices indicate that the depth difference between the grid vertices of the first RGB-D image in the overlapping region and the corresponding grid vertices of the second RGB-D image in the overlapping region of two RGB-D images with adjacent shooting perspectives exceeds the threshold, but the interval between it and the nearest true overlapping grid vertex exceeds grid vertices.

[0075] After constructing the joint optimization energy function, iterative solution can be carried out. In this embodiment, the LM algorithm is used to iteratively solve the vertex displacement:

[0076] , (12)

[0077] Among them, and are the optimized variable vectors after the -th iteration and the optimized variable vectors after the -th iteration respectively, and are the Jacobian matrix and its transpose matrix respectively, is the weight diagonal matrix, is the damping factor of the LM algorithm, represents taking of the diagonal elements, is the residual vector.

[0078] The iterative convergence condition of the LM algorithm is:

[0079] , (13)

[0080] Among them, is the energy value after the -th iteration, is the energy value after the -th iteration, represents taking and the larger value of 1, represents the coordinate change amount of the -th grid vertex in the -th iteration, is the current iteration number, is the set maximum iteration number (for example, 100).

[0081] After the algorithm optimization and solution, the final optimized solution result can be output, that is, the position coordinate change amount of the optimized grid vertices, denoted as , and further denoted as . By mapping the part of the first RGB-D image in the overlapping area to the optimized elastic grid, and mapping the part of the second RGB-D image in the overlapping area to the optimized elastic grid, it prepares for the stitching and fusion of the first RGB-D image and the second RGB-D image in the overlapping area.

[0082] To achieve the stitching and fusion of the first RGB-D image and the second RGB-D image in the overlapping area, a panoramic image is determined. In this embodiment, after mapping the parts of the first RGB-D image and the second RGB-D image in the overlapping area to the optimized elastic grid, the following method is used to achieve the image fusion in the overlapping area:

[0083] , (14)

[0084] , (15)

[0085] Among them, represents the fused pixel value of the th pixel in the overlapping area, is the fusion weight of the th pixel in the overlapping area, is the pixel value of the th pixel of the first RGB-D image in the overlapping area after mapping, is the pixel value of the th pixel of the second RGB-D image in the overlapping area after mapping, is the gradient magnitude of the th pixel of the first RGB-D image in the overlapping area after mapping, is the gradient magnitude of the th pixel of the second RGB-D image in the overlapping area after mapping, is the regional dynamic weight of the th pixel in the overlapping area (judged according to the calculation method of formula (11), not limited to grid vertices, but extended to each pixel in the overlapping area), is the confidence weight of the elastic grid where the th pixel in the overlapping area is located, satisfying:

[0086] , (16)

[0087] Among them, , , , are respectively the confidence weights of the first grid vertex, the second grid vertex, the third grid vertex, and the fourth grid vertex of the elastic grid where the th pixel point in the overlapping area is located. , , , are respectively the distances between the th pixel point in the overlapping area and the first grid vertex, the second grid vertex, the third grid vertex, and the fourth grid vertex of the elastic grid where it is located.

[0088] Accordingly, the stitching and fusion of two RGB-D images at the same shooting position in the overlapping area can be completed to generate a panoramic image of the shooting position. The fusion stage well considers the important parameters in the stitching and alignment stage, introduces key parameters as weights and performs fusion in different cases, which can improve the fusion quality of the image as much as possible and effectively reduce the stitching and fusion traces.

[0089] After determining the panoramic image of each shooting position, step S30 can be executed.

[0090] Step S30: Generate a VR panoramic image based on the panoramic image corresponding to each shooting position.

[0091] In this embodiment, the panoramic image of each shooting position can be converted into a cube map, and the near-view area is corrected by adaptive bi-parabolic projection. Then, based on the spatial pose parameters of the shooting position, the cube map is mapped to the spherical space in the global coordinate system, where the depth of the near-view area does not exceed 5 meters.

[0092] Then, key points are generated for each shooting position, and based on the spatial coordinates of the shooting position, the spatial topological relationship of the shooting position is constructed, a reachability relationship graph with key points as nodes is established, and the traffic logic of adjacent areas is defined. After that, post-processing optimization (such as dynamic parallax correction and motion blur compensation, etc.) can be performed on the panoramic image, and finally, a VR panoramic image is output.

[0093] In summary, the embodiment of the present application provides a method for generating a VR panoramic image. In this solution, six RGB-D images at the same moment and the same shooting position (the RGB-D images of adjacent viewpoints are spaced 60° apart) are obtained. By performing elastic mesh modeling on two RGB-D images of adjacent shooting viewpoints, aligning the overlapping regions, and splicing and fusing the overlapping regions, a panoramic image is determined. Then, based on the panoramic images corresponding to each shooting position, a VR panoramic image is generated. This solution can introduce depth information into the generation of VR panoramic images at a relatively low cost, and make full use of the existing integrated processing solutions to extract and match basic feature information (in this solution, SIFT-3D implemented after extending other libraries with OpenCV is used for feature point extraction, and the combination of FLANN and RANSAC compatible with SIFT-3D is used for matching feature point pairs). It can be implemented with only a small amount of code, greatly reducing the development workload. For the splicing of RGB-D images of adjacent viewpoints, which is the core part, by designing a double-layer elastic mesh scheme and using the feature point density, it is determined whether to arrange a secondary elastic mesh in the primary grid unit to achieve high-quality splicing.

[0094] In the splicing process modeling, by constructing a joint optimization energy function, on the basis of the color alignment term and the deformation smoothing term, a depth constraint term is introduced, and the boundary preservation term is improved, thereby effectively reducing mis-matches under different viewpoints and avoiding perspective distortion. And the weight of the color alignment term , the weight of the depth constraint term , the weight of the deformation smoothing term and the weight of the boundary preservation term can adjust the weight parameters according to different scenarios (such as indoor, outdoor, lighting, etc.) (such as increasing the weight of the color alignment term under strong light, increasing the weight of the depth constraint term in a weak texture scene, increasing the weight of the deformation smoothing term in a vast area, increasing the weight of the depth constraint term and the weight of the boundary preservation term in a scene involving complex structures), to achieve dynamic adaptation to different scenarios. And in the boundary preservation term, the real overlapping region, the transition zone region and the pseudo-overlapping region are designed and divided, which can maintain high-precision deformation adjustment, smooth the deformation amount, and can effectively identify and perform corresponding optimization processing on mis-matches (actually pseudo-overlaps, which are basically indistinguishable in 2D, but can be recognized in this solution), so as to achieve high-precision splicing and fusion and effectively reduce the abruptness of the splicing edge. When using the LM algorithm for iterative solution, combining the gradient descent method and the Gauss-Newton method can well adapt to the non-convexity of the joint optimization energy function and achieve robust and fast convergence.

[0095] At present, this solution has been successfully applied in many projects (of course, not all rely on this solution, and there is also the VR panoramic construction technology based on drones), such as: digital culture and tourism (such as the former residence of Fan Changjiang), digital museums (such as the Fan Changjiang Exhibition Hall), digital land investment promotion (Neijiang Smart Investment Promotion, this project mainly relies on the VR panoramic construction technology based on drone aerial photography), etc., which has good application value and commercial potential.

[0096] In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0097] The above are only examples of this application and are not used to limit the protection scope of this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims

1. A method for generating a VR panoramic image, characterized in that, Including: Obtain 6 RGB-D images at the same moment and the same shooting position, where the shooting angle interval between two adjacent RGB-D images is 60°; Perform elastic grid modeling on two RGB-D images with adjacent shooting angles, align the overlapping regions, and splice and fuse the overlapping regions to determine a panoramic image; Generate a VR panoramic image based on the panoramic image corresponding to each shooting position; Perform elastic grid modeling on two RGB-D images with adjacent shooting angles to align the overlapping regions, including: Respectively perform feature extraction on two RGB-D images with adjacent shooting angles to determine the feature points of each RGB-D image, and match the feature points of the two RGB-D images to determine the matching feature point pairs; Based on the feature point pairs, generate an adaptive double-layer elastic grid for the two RGB-D images, and project the grid vertices contained in each of the two RGB-D images onto a unified spherical coordinate system, where the adaptive elastic grid includes a primary grid and a secondary grid, and the secondary grid is located within the unit grid area of the primary grid; Construct a joint optimization energy function and use nonlinear optimization to solve it to align the overlapping regions of the two RGB-D images; Construct a joint optimization energy function and use nonlinear optimization to solve it to align the overlapping regions of the two RGB-D images, including: Construct a joint optimization energy function for the grid vertices in the two RGB-D images: ， Among them, is the jointly optimized energy function, , , and are the weights of the color alignment term, the depth constraint term, the deformation smoothing term, and the boundary preservation term respectively, , , and are the color alignment term, the depth constraint term, the deformation smoothing term, and the boundary preservation term respectively; Use the LM algorithm to iteratively solve the vertex displacement: ， wherein, and are respectively the optimized variable vectors after the -th iteration and the optimized variable vectors after the -th iteration, and are respectively the Jacobian matrix and its transpose matrix, is the weight diagonal matrix, is the damping factor of the LM algorithm, denotes taking the diagonal elements, is the residual vector; The iterative convergence condition of the LM algorithm is: ， Among them, is the energy value after the -th iteration, is the energy value after the -th iteration, means taking the larger value of and 1, represents the change in the coordinates of the -th grid vertex in the -th iteration, is the current iteration number, is the set maximum number of iterations; Color alignment item Satisfy: ， Among them, is the set of all grid vertices in the overlapping area, including N grid vertices, is the coordinates of the th grid vertex, is the structural similarity measurement operator, is the dynamically adjusted parameter, is the color value at the position of the grid vertex of the first RGB-D image in two RGB-D images with adjacent shooting perspectives, is the grid vertex of the second RGB-D image in two RGB-D images with adjacent shooting perspectives after geometric transformation at the projected position, and the dynamically adjusted parameter satisfies: ， Among them, is a function, representing the magnitude of the image gradient.

2. The method for generating a VR panoramic image according to claim 1, wherein Use the SIFT-3D algorithm to detect the feature points of two RGB-D images with adjacent shooting angles respectively to determine the feature points of each RGB-D image, and then use the FLANN+RANSAC combination scheme for feature point matching to determine the matching feature point pairs in the two RGB-D images with adjacent shooting angles.

3. The method for generating a VR panoramic image according to claim 1, wherein Based on the feature point pairs, generate an adaptive double-layer elastic grid for the two RGB-D images, including: Generate primary grids in the two RGB-D images respectively; Based on the number of feature point pairs covered in each primary grid, determine the target unit grids from the primary grids, and generate secondary grids in each target unit grid.

4. The method for generating a VR panoramic image according to claim 1, wherein Depth constraint term Satisfy: ， Among them, is the set of all grid vertices in the overlapping region, containing N grid vertices, is the confidence weight of the th grid vertex, is the depth value at the position of the grid vertex of the first RGB-D image in two RGB-D images with adjacent shooting perspectives, and is the depth value at the projected position of the grid vertex of the second RGB-D image in two RGB-D images with adjacent shooting perspectives after geometric transformation, and the confidence weight satisfies: ， Among them, to calculate the signal-to-noise ratio, is the coordinate of the boundary point closest to the grid vertex.

5. The method for generating a VR panoramic image according to claim 1, wherein Deformation smoothing term Satisfy: ， Among them, represents the set of grid edges in the overlapping region, represents the grid edge of the length deformation variable, is the balance parameter, which is a constant and takes 0.5, represents the set of grid faces in the overlapping region, represents the grid face of the deformation Jacobian matrix, is the Jacobian determinant, representing the area change rate of the grid face , and the length deformation variable satisfies: ， Among them, and are the coordinates of two grid vertices that form the grid edge , and is a fixed value during grid initialization.

6. The method for generating a VR panoramic image according to claim 1, wherein, Boundary-preserving term Satisfy: ， Among them, represents the th grid vertex coordinates of the deformed vertex, is the initial coordinate of the th grid vertex, is the regional dynamic weight, satisfying: ， Among them, represents the set of true coincidence grid vertices in the coincidence region. A true coincidence grid vertex means that the depth difference between the grid vertex of the first RGB-D image in the coincidence region and the corresponding grid vertex of the second RGB-D image in the coincidence region of two adjacent captured viewpoints does not exceed the threshold. represents the set of grid vertices in the transition zone of the coincidence region. A grid vertex in the transition zone means that the depth difference between the grid vertex of the first RGB-D image in the coincidence region and the corresponding grid vertex of the second RGB-D image in the coincidence region of two adjacent captured viewpoints exceeds the threshold, but the interval between it and the nearest true coincidence grid vertex does not exceed grid vertices. represents the set of non-coincidence grid vertices in the coincidence region. A non-coincidence grid vertex means that the depth difference between the grid vertex of the first RGB-D image in the coincidence region and the corresponding grid vertex of the second RGB-D image in the coincidence region of two adjacent captured viewpoints exceeds the threshold, but the interval between it and the nearest true coincidence grid vertex exceeds grid vertices.

7. The method for generating a VR panoramic image according to claim 1, wherein Generate a VR panoramic image based on the panoramic image corresponding to each shooting position, including: Convert the panoramic image of each shooting position into a cube map, perform adaptive bi-parabolic projection correction on the near-view area, and then map the cube map to the spherical space under the global coordinate system based on the spatial pose parameters of the shooting position, where the depth of the near-view area does not exceed 5 meters; Generate key points for each shooting position, construct the spatial topological relationship of the shooting positions based on the spatial coordinates of the shooting positions, establish an accessibility relationship graph with the key points as nodes, and define the traffic logic for adjacent areas; Perform post-processing optimization on the panoramic image and output the VR panoramic image, where the post-processing optimization includes dynamic parallax correction and motion blur compensation.

Images