5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII

Abstract

The invention discloses a 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII, and the method comprises the steps: firstly carrying out the frame extraction processing of data shot when an unmanned plane flies around a signal tower, and obtaining an original image set; obtaining a signal tower sparse point cloud and a camera pose through a motion structure recovery SfM algorithm, and carrying out 3DGS initialization operation; the method is characterized in that in the process of iterative optimization of a three-dimensional Gaussian point cloud, a GSII network is constructed, structural similarity comparison is carried out on an original image of the same visual angle and a deblurred rendering image, a loss function is sensed by using a high receptive field and back propagation is carried out, meanwhile, parameter updating is carried out on a 3D Gaussian ball attribute and FFC residual error repair network, and the quality of a new visual angle rendering image is improved; and finally, designing a central pixel time sequence splicing module, and converting any number of single-view-angle rendering images into a panoramic image. According to the method, the phenomena of artifacts and blurring which are easy to occur under a high-altitude view angle are solved, and the signal tower high-altitude panorama is finally obtained by synthesizing a plurality of rendering images of a single view angle.

Description

Technical Field

[0001] The present invention relates to a method for synthesizing high-altitude panoramic images of 5G signal towers, specifically a method for three-dimensional rendering and high-altitude panoramic images of 5G signal towers based on GSII, and belongs to the field of computer vision technology. Background Art

[0002] With the rapid development of my country's 5G network, the total number of 5G base station towers will reach over four million by the end of 2024. This massive number presents enormous challenges in the maintenance and inspection of base station towers. Currently, inspections primarily rely on manual labor, a method with numerous limitations and risks. To address these issues, some regions are experimenting with the use of drones for tower inspections. Drone inspections not only improve efficiency and reduce safety risks, but also provide clear image data. However, direct observation of towers from drones is not only limited in perspective but also subject to background interference. To obtain high-quality views of towers, enabling inspectors to analyze and process data on the ground, 3D reconstruction and image rendering technologies are required.

[0003] The 3D Gaussian Splatting (3DGS) algorithm is commonly used for three-dimensional reconstruction tasks in various scenes. This technology achieves high-quality real-time rendering of a new perspective by representing the point cloud as a 3D Gaussian function and using differentiable fast rasterization for rendering and optimization. It can provide accurate and efficient reconstruction results while maintaining the continuity and details of the scene. 3DGS can effectively retain the excellent properties of the continuous volume radiation field while avoiding unnecessary calculations in blank space, thereby improving reconstruction efficiency. However, when faced with large scenes from a high-altitude perspective, the 3DGS rendering results have artifacts and blurring, which will affect the observation of high-altitude precision components of signal towers. Moreover, the signal tower is huge, and only a single perspective is rendered, and the information obtained is limited.

[0004] 5G signal towers have many precision components, such as antennas. During inspections, meticulous observation of these components at high altitudes is crucial. However, antennas are highly similar, and the differences within a category are not significant. Using drones equipped with target detection algorithms for direct detection would make it impossible to distinguish between different targets within the same category, and repeated or missed images could easily occur after a full orbit. While existing image stitching algorithms can achieve the goal of ensuring all antennas are coplanar, they impose certain requirements on the input image, making them difficult to apply in practice. Both traditional methods based on scale-invariant feature transformation and deep learning-based methods have high requirements for input image resolution, viewing angle, and overlap, making them unsuitable for high-altitude signal tower scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a GSII-based 3D rendering and high-altitude panoramic image synthesis method for 5G signal towers. By constructing a GSII network for 3D rendering, the artifacts and blurring that are prone to occur at high altitude perspectives are solved. At the same time, a central pixel timing splicing module is used to synthesize multiple single-perspective rendered images to obtain a high-altitude panoramic image of the signal tower.

[0006] To achieve the above objectives, the present invention provides a GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method, comprising the following steps:

[0007] S1: A drone is used to fly around the signal tower and record the video. The resulting video is evenly framed to obtain the original image set, denoted as I, which is specifically defined as:

[0008] I={i1,i2,···,i n}

[0009] Where: the numerical subscript represents the image number;

[0010] S2: Use the SfM algorithm to perform preliminary 3D reconstruction on the original image set to obtain a sparse point cloud and camera pose matrix. The sparse point cloud is initialized with 3DGS, where each point is set as the center point of a 3D Gaussian sphere, and each Gaussian sphere is assigned attributes such as color, shape, and transparency. The set of all Gaussian spheres is defined as a 3D Gaussian point cloud.

[0011] S3: The three-dimensional Gaussian point cloud G can be projected onto a two-dimensional plane through fast differentiable rasterization to obtain the preliminary rendering result I R The viewing angle is determined by the camera pose matrix obtained in the preliminary 3D reconstruction. Each iteration will get the preliminary rendering result I R Input the FFC residual repair module to generate a deblurred rendering image I Deblur , and compare the pixels with the original image of the corresponding perspective, use the structural similarity loss function and the high receptive field perception loss function to backpropagate and update the network parameters to optimize the properties of each Gaussian ball in the three-dimensional Gaussian point cloud G;

[0012] S4: After training, the FFC residual restoration module can render and repair the signal tower from any perspective to obtain ultra-clear rendering images from continuous perspectives. At this time, the three-dimensional Gaussian point cloud G has been optimized into a dense point cloud. The ultra-clear rendering images from continuous perspectives are input into the central pixel temporal stitching module to obtain a high-altitude panoramic image of the signal tower.

[0013] Step S2 of the present invention is specifically as follows:

[0014] S21: Based on the original image set I, a preliminary 3D reconstruction can be performed using the SfM algorithm to obtain sparse 3D point cloud data P SPThe camera pose matrix M corresponding to each image;

[0015] S22: sparse 3D point cloud data P SP Each point in is initialized as the center point of a three-dimensional Gaussian sphere, and a three-dimensional Gaussian point cloud G is obtained, G={g1,g2,…,g n}, where: g n represents the nth Gaussian sphere;

[0016] S23: Each Gaussian sphere not only has the center point coordinates, but is also given a covariance matrix Σ, opacity α and spherical harmonics Attribute, θ is the polar angle, is the azimuth;

[0017]

[0018] Where: R is the rotation transformation matrix;

[0019] S is the scale transformation matrix;

[0020] σ is the probability density of light being blocked by the current point;

[0021] δ is the distance from the current point passed by the light to the next point continuing along that direction;

[0022] is the spherical harmonic basis function;

[0023] c l,m The color coefficient is obtained and updated through model iteration.

[0024] Step S3 of the present invention is specifically as follows:

[0025] S31: Project the 3D Gaussian point cloud G onto a 2D plane using a fast differentiable rasterization method, and obtain the initial rendering result I R The number of is consistent with the number in the original image set I, and the viewing angle is determined by the camera pose matrix M obtained during the initial 3D reconstruction, that is, the rendering viewing angle is the same as the original image;

[0026] S32: Then the preliminary rendering result I R Input FFC residual restoration module, one channel uses traditional convolution to extract local information, and the other channel uses fast Fourier transform to extract global context information, and finally cross-fusion is performed and connected with the channel level to obtain the deblurred rendering image I Deblur ;

[0027] S33: In each iteration, deblur the rendered image I DeblurThe pixel comparison is performed in pairs with the original image of the corresponding perspective in the original image set I, and a structural similarity loss function is constructed based on this. The attribute parameters of each Gaussian sphere in the three-dimensional Gaussian point cloud G are optimized and updated through iteration through the stochastic gradient descent method, thereby improving the accuracy of the three-dimensional Gaussian point cloud of the signal tower and deblurring the rendered image I. Deblur And the preliminary rendering results I R Through the high receptive field perception loss function, the parameters of the FFC residual repair module are updated to improve the results in the next round of iteration.

[0028] Step S4 of the present invention is specifically as follows:

[0029] S41: After several rounds of iterative optimization, the three-dimensional Gaussian point cloud G is transformed into a dense point cloud P DS , and then the high-altitude antenna ultra-clear rendering image set I is obtained through the rasterization method and the trained FFC residual repair module A , I A ={A1,A2,···,A n The viewing angle of each image can be set arbitrarily, but for the purpose of subsequent time-sequential splicing of the central pixels, the present invention sets the viewing angle between images to change uniformly, and n images are connected end to end to circle the signal tower.

[0030] S42: Collection of high-altitude antenna high-definition rendering images I A The resolution of n images in the image is the same, with width w and height h. If each image is regarded as a matrix formed by the arrangement of pixels, then A n It can be expressed as:

[0031]

[0032] A n With A n-1 Difference in perspective therefore and The details of the signal towers observed also vary. Represents image A n No. Column (center column) pixels, after splicing the center pixels of n rendered images in time sequence, a panoramic image O can be obtained:

[0033]

[0034] Compared with the existing technology, the present invention addresses the problem that the basic 3DGS model in the intelligent inspection task of 5G signal towers is prone to artifacts and blurring when performing large-scale three-dimensional rendering, and the information contained in the single-perspective rendered image is limited. A method for synthesizing signal tower three-dimensional rendering and high-altitude panoramic images based on GS II is provided. First, the data captured by the drone flying around the signal tower is subjected to frame extraction processing to obtain an image sequence; then, the sparse point cloud of the signal tower and the camera pose are obtained through the motion structure recovery (Structure from Motion, SfM) algorithm, and a 3DGS initialization operation is performed; focusing on the process of iterative optimization of the three-dimensional Gaussian point cloud, the present invention constructs a GSII (Gaussian Splatting Image Inpainting) network, compares the structural similarity between the original image and the deblurred rendered image of the same perspective, uses a high receptive field perception loss function and backpropagation, and simultaneously updates the parameters of the 3D Gaussian sphere attributes and the FFC residual repair module to improve the quality of the new perspective rendered image; finally, a central pixel temporal splicing module is designed to convert any number of single-perspective rendered images into a panoramic image. The present invention solves the problem of artifacts and blurring that are prone to occur at high altitude perspectives by constructing a GSII network for three-dimensional rendering. At the same time, it uses a central pixel temporal splicing module to synthesize multiple single-perspective rendering images to obtain a high-altitude panoramic view of the signal tower. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the overall framework diagram of the method of the present invention;

[0036] Figure 2 This is the effect of the preliminary 3D reconstruction of SfM;

[0037] Figure 3 This is a visualization of the 3D Gaussian point cloud after iterative optimization;

[0038] Figure 4 is the input original image at a certain perspective;

[0039] Figure 5 For Figure 4 Preliminary rendering results from the same viewing angle;

[0040] Figure 6 This is the optimized rendering;

[0041] Figure 7 Schematic diagram of the time-sequential stitching of center pixels for rendered images;

[0042] Figure 8 This is a spliced panoramic view of the signal tower from above;

[0043] Figure 9 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below with reference to the accompanying drawings.

[0045] Important components of the signal tower, such as the antenna, are located in a high-altitude scene. If the 3DGS method is used directly for three-dimensional reconstruction and rendering, artifacts and blurring will occur. Moreover, only local information of the signal tower can be obtained by rendering from a single perspective. In order to solve this problem, it is necessary to generate a panoramic image so that all components on the tower can be directly observed. However, the existing image stitching method has high requirements for input, especially in terms of image overlap rate, and is not suitable for camera revolution scenes (around a certain target) with drastic background changes. In order to effectively solve these problems, the present invention discloses a 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII;

[0046] like Figure 1 and Figure 9 As shown, a GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method includes the following steps:

[0047] Step S1: Collect original image data and perform preliminary sparse reconstruction;

[0048] like Figure 1 As shown in the overall framework diagram, during the image data acquisition process, the drone's flight path is set to circle the tower and record it. The obtained video is evenly framed to obtain the original image set, which is recorded as I;

[0049] I={i1,i2,···,i n}

[0050] Where: the numerical subscript represents the image number;

[0051] The frame extraction process of the present invention is a prior art and will not be described in detail.

[0052] S2: Use the SfM algorithm to perform preliminary 3D reconstruction on the original image set to obtain a sparse point cloud and camera pose matrix. The sparse point cloud is initialized with 3DGS, where each point is set as the center point of a 3D Gaussian sphere, and each Gaussian sphere is assigned attributes such as color, shape, and transparency. The set of all Gaussian spheres is defined as a 3D Gaussian point cloud.

[0053] S21: Input original image set I={i1,i2,…,i n}, the SfM algorithm (Schonberger JL, Frahm JM. Structure-from-Motion Revisited [C] / / IEEE Conference on Computer Vision & Pattern Recognition. IEEE, 2016: 4104-4113.) is used for preliminary 3D reconstruction, and the following is obtained: Figure 2 The sparse 3D point cloud data P shown SP The camera pose matrix M corresponding to each image in the original image set I, the camera pose matrix of the nth image is recorded as M n ;

[0054]

[0055] Where: R WC is the rotation matrix of the camera relative to the world coordinate system;

[0056] T represents the translation vector;

[0057] S22: sparse 3D point cloud data P SP Perform 3D GS initialization, initialize each point to the center point of a three-dimensional Gaussian sphere, and obtain the initial three-dimensional Gaussian point cloud G, G={g1,g2,…,g n}, where: g n represents the nth Gaussian ball. The mathematical form of any Gaussian ball g can be expressed as:

[0058]

[0059] Specifically, its shape is described by the covariance matrix Σ, which is a positive symmetric matrix and an optimization parameter during scene learning. It is decomposed into a rotation matrix R and a scaling matrix S, that is,

[0060] Σ=RSS T R T

[0061] In addition to shape attributes, each Gaussian sphere also has color attributes, which are determined by opacity α and spherical harmonics Joint decision-making;

[0062] Opacity α is specifically defined as:

[0063] α=1-e -σδ ;

[0064] Where: σ represents the probability density of light being blocked by the current point;

[0065] δ represents the distance from the current point passed by the ray to the next point continuing along that direction;

[0066] Spherical harmonics Specifically defined as:

[0067]

[0068] Where: θ is the polar angle;

[0069] is the azimuth;

[0070] is the spherical harmonic basis function;

[0071] c l,m The color coefficient is obtained and updated through model iteration.

[0072] Therefore, any Gaussian ellipsoid in the initial three-dimensional Gaussian point cloud G has the following properties:

[0073]

[0074] S3: GSII network training optimization;

[0075] The shape and color attributes of the Gaussian sphere mentioned above are based on the sparse 3D point cloud data P obtained by the initial 3D reconstruction of SfM. SP , multiple iterative optimizations are needed to improve the accuracy;

[0076] Specifically, it uses a fast differentiable tile-based rasterizer technology. First, it divides the two-dimensional plane into small regions to reduce the amount of calculation. Second, it uses the sigmoid function to soften the pixel coverage judgment and calculates the impact on the pixel in a probabilistic form. At the same time, it handles depth conflicts through weighted blending, retaining the gradient of interpolation attributes (such as color and shape). Finally, the independent results of each region are merged into a complete image.

[0077] When the points of the three-dimensional Gaussian point cloud G are projected onto the two-dimensional pixel plane, the color C of each pixel is determined by the alpha blending method, which is specifically defined as:

[0078]

[0079] Where: c i represents the spherical harmonic function of point i;

[0080] a i Indicates the opacity of point i;

[0081] Then the preliminary rendering result I RInput FFC residual restoration module, specifically through two parallel branch channels, on the one hand, using traditional convolution to extract local information, on the other hand, using fast Fourier transform to extract global information, and finally cross-fusion and channel level connection to obtain the deblurred rendering image I Deblur ;

[0082] In each round of iteration, any original image i in the original image set I j There are corresponding preliminary rendering results I R , the two cameras have the same pose, such as Figure 3 and Figure 4 As shown, a pixel-by-pixel comparison is then performed, and the structural similarity loss function is used for calculation. The results are back-propagated through the stochastic gradient descent method to optimize the three-dimensional Gaussian point cloud G.

[0083] Deblurred rendered image I Deblur Then compare with the preliminary rendering results I R By using the high receptive field perception loss function, the parameters of the FFC residual restoration module are updated to improve the quality of the final output image, such as Figure 5 As shown;

[0084] S4: Center Pixel Sequential Stitching Module: After training, the FFC residual repair module can repair the signal tower rendering image from any perspective to obtain ultra-clear rendering images from continuous perspectives. The ultra-clear rendering images from continuous perspectives are then input into the center pixel sequential stitching module to obtain a high-altitude panoramic image of the signal tower.

[0085] After several iterations of optimization, the three-dimensional Gaussian point cloud G is transformed into a dense point cloud P DS ,like Figure 3 As shown, compared with the sparse 3D point cloud data P obtained from the SfM algorithm SP , which is closer to the shape, texture and color of real signal towers;

[0086] For dense point cloud P DS Perform rendering and restoration at any angle to obtain a collection of high-definition rendering images of high-altitude antennas. A , expressed as:

[0087] I A ={A1,A2,···,A n}

[0088] Among them: A n Indicates the nth rendered image;

[0089] The viewing angle of each rendered image can be freely set according to the specific situation of different signal towers. However, in order to perform the subsequent central pixel sequential splicing, the present invention sets the viewing angle to change evenly. The n images are connected end to end, and the viewing angle moves around the signal tower for one circle, that is, the viewing angle difference of adjacent images is

[0090] After experiments, considering the quality of the panoramic image generated after pixel stitching and the efficiency of the algorithm, the present invention sets n=360, that is, the parallax between two adjacent images is 1°;

[0091] High-altitude antenna ultra-clear rendering image collection I A The n images have the same resolution, width w, height h;

[0092] If each image is regarded as a matrix formed by the arrangement of pixels, then A n It can be expressed as:

[0093]

[0094] Because the n images are evenly distributed around the perimeter, A n With A n-1 Difference in perspective The middle column of pixels between the two can be recorded as and The details of the signal tower represented by these two columns of pixels are also different.

[0095] Arrange and stitch the middle column of pixels of all rendered images in the order of the circular motion of the viewing angle, such as Figure 7 As shown, a panoramic image O showing all antennas can be obtained, such as Figure 8 As shown, it is defined as:

[0096]

[0097] This paper proposes a GSII-based 3D rendering and high-altitude panoramic image synthesis method for 5G signal towers to better facilitate intelligent signal tower inspections. In large scenes from a high-altitude perspective, direct 3D reconstruction and rendering using 3DGS results in artifacts and blurring, and it is difficult to capture the global information of the signal tower using only a single-viewpoint rendered image. Therefore, a GSII network is specifically designed. During the iterative optimization of the 3D Gaussian point cloud, an FFC residual repair module is introduced to simultaneously improve Gaussian sphere accuracy and rendered image quality. Furthermore, a center pixel temporal splicing module is designed at the network end to synthesize rendered images from a circumference of the tower, producing a high-quality panoramic image with all tower components coplanar, addressing the information limitations of a single viewpoint. Observing signal towers using this panoramic image not only eliminates the safety risks of manual tower climbing but also increases automation, making the 5G signal tower inspection process more convenient and efficient. This method can completely replace manual tower inspections, improving the accuracy of the results and making 5G signal tower inspections safer, more convenient, and more efficient.

Claims

1. A 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII, characterized in that: The following steps are involved: S1: A drone is used to fly around the signal tower and record the video. The resulting video is evenly framed to obtain the original image set, denoted as I, which is specifically defined as: I={i1,i2,···,i n } Where: the numerical subscript represents the image number; S2: Use the SfM algorithm to perform preliminary 3D reconstruction on the original image set to obtain a sparse point cloud and camera pose matrix. The sparse point cloud is initialized with 3DGS, where each point is set as the center point of a 3D Gaussian sphere, and each Gaussian sphere is assigned color, shape, and transparency attributes. The set of all Gaussian spheres is defined as a 3D Gaussian point cloud G. S3: Project the 3D Gaussian point cloud G onto a 2D plane through fast differentiable rasterization to obtain the preliminary rendering result I R The viewing angle is determined by the camera pose matrix obtained in the preliminary 3D reconstruction. Each iteration will render the preliminary rendering result I R Input the FFC residual repair module to generate a deblurred rendering image I Deblur , and compare the pixels with the original image of the corresponding perspective, use the structural similarity loss function and the high receptive field perception loss function to backpropagate and update the network parameters to optimize the properties of each Gaussian ball in the three-dimensional Gaussian point cloud G; S4: After training, the FFC residual restoration module can render and repair the signal tower from any perspective to obtain ultra-clear rendering images from continuous perspectives. At this time, the three-dimensional Gaussian point cloud G has been optimized into a dense point cloud. The ultra-clear rendering images from continuous perspectives are input into the central pixel temporal stitching module to obtain a high-altitude panoramic image of the signal tower.

2. The method for synthesizing 5G signal tower 3D rendering and high-altitude panoramic images based on GSII according to claim 1, characterized in that: The step S2 is specifically as follows: S21: Based on the original image set I, preliminary 3D reconstruction is performed using the SfM algorithm to obtain sparse 3D point cloud data P SP The camera pose matrix M corresponding to each image; S22: sparse 3D point cloud data P SP Each point in is initialized as the center point of a three-dimensional Gaussian sphere, and a three-dimensional Gaussian point cloud G is obtained, G={g1,g2,…,g n }, where: g n represents the nth Gaussian sphere; S23: Each Gaussian sphere not only has the center point coordinates, but is also given a covariance matrix Σ, opacity α and spherical harmonics Attribute, θ is the polar angle, is the azimuth; Where: R is the rotation transformation matrix; S is the scale transformation matrix; σ is the probability density of light being blocked by the current point; δ is the distance from the current point passed by the light to the next point continuing along that direction; is the spherical harmonic basis function; c l,m is the color coefficient.

3. The GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method according to claim 2 is characterized in that: The step S3 is specifically as follows: S31: Project the 3D Gaussian point cloud G onto a 2D plane using a fast differentiable rasterization method, and obtain the initial rendering result I R The number of is consistent with the number in the original image set I, and the viewing angle is determined by the camera pose matrix M obtained during the initial 3D reconstruction, that is, the rendering viewing angle is the same as the original image; S32: Then the preliminary rendering result I R Input FFC residual restoration module, one channel uses traditional convolution to extract local information, and the other channel uses fast Fourier transform to extract global context information, and finally cross-fusion is performed and connected with the channel level to obtain the deblurred rendering image I Deblur ; S33: In each iteration, deblur the rendered image I Deblur The pixel comparison is performed in pairs with the original image of the corresponding perspective in the original image set I, and a structural similarity loss function is constructed based on this. The attribute parameters of each Gaussian sphere in the three-dimensional Gaussian point cloud G are optimized and updated through iteration through the stochastic gradient descent method, thereby improving the accuracy of the three-dimensional Gaussian point cloud of the signal tower and deblurring the rendered image I. Deblur And the preliminary rendering results I R Through the high receptive field perception loss function, the parameters of the FFC residual repair module are updated to improve the results in the next round of iteration.

4. The method for 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis based on GSII according to claim 1 is characterized in that: The step S4 is specifically as follows: S41: After several rounds of iterative optimization, the three-dimensional Gaussian point cloud G is transformed into a dense point cloud P DS , and then the high-altitude antenna ultra-clear rendering image set I is obtained through the rasterization method and the trained FFC residual repair module A , I A ={A1,A2,···,A n The viewing angle of each image can be customized, but for the subsequent time-series stitching of the central pixels, the viewing angles between images are set to change evenly, and n images are connected end to end to circle the signal tower. S42: Collection of high-altitude antenna high-definition rendering images I A The resolution of n images in the image is the same, with width w and height h. If each image is regarded as a matrix formed by the arrangement of pixels, then A n Expressed as: A n With A n-1 Difference in perspective therefore and The details of the signal towers observed also vary. Represents image A n No. Column pixels, after splicing the center pixels of n rendered images in time sequence, a panoramic image O can be obtained:

Images