A 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII

Abstract

A kind of 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII, first, the data of unmanned aerial vehicle flying around signal tower and shooting are frame extraction processing, and the original image set is obtained;Then, the sparse point cloud of signal tower and camera pose are obtained by SfM algorithm, and 3D GS initialization operation is carried out;Focus on the process of three-dimensional Gaussian point cloud iterative optimization, GSII network is constructed, the structural similarity of the same view original image and deblurring rendered image is compared, high receptive field perception loss function is used and back propagation is carried out, at the same time, the parameters of 3D Gaussian sphere attribute and FFC residual repair network are updated, and the quality of new view rendered image is improved;Finally, the center pixel time sequence splicing module is designed, and any number of single view rendered image is converted into panoramic image.The present application solves the artifacts and blur phenomenon easily appeared under high-altitude view, synthesizes multiple single view rendered images, and finally obtains signal tower high-altitude panoramic image.

Description

TECHNICAL FIELD

[0001] The application relates to a 5G signal tower high-altitude panoramic image synthesis method, in particular to a 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII, and belongs to the technical field of computer vision. BACKGROUND

[0002] With the rapid development of China's 5G network, by the end of 2024, the total number of 5G base station signal towers has reached more than 4 million. Behind this huge number is a huge challenge to the maintenance and repair of base station signal towers. At present, its inspection mainly relies on manual methods, which has many limitations and risks. In order to solve these problems, some regions have begun to try to use drones to inspect signal towers. Drone inspection not only can improve the inspection efficiency and reduce the safety risk, but also can provide clear image data. However, direct observation of signal towers by drones will not only be limited in perspective, but also be disturbed by the background. In order to obtain high-quality views of signal towers and let inspectors analyze and process data on the ground, three-dimensional reconstruction and image rendering technology needs to be used.

[0003] 3D Gaussian Splatting (3DGS) algorithm is commonly used for three-dimensional reconstruction tasks in various scenes. This technology represents point clouds as 3D Gaussian functions and uses differentiable fast rasterization for rendering and optimization, achieving high-quality real-time rendering of new perspectives. It can provide accurate and efficient reconstruction results while maintaining the continuity and details of the scene. 3DGS can effectively preserve the excellent properties of continuous volume radiation field while avoiding unnecessary calculations in empty space, improving reconstruction efficiency. However, in the face of high-altitude views of large scenes, 3DGS rendering results have artifacts and blurring, which will affect the observation of high-precision components of signal towers at high altitudes, and the volume of signal towers is large. Only single-view rendering will obtain limited information.

[0004] There are many precision components such as antennas on 5G signal towers, and detailed observation of various components at high altitudes is an important task during inspection. However, antennas have high similarity and the intra-class gap is not obvious. If a drone equipped with a target detection algorithm is used for direct detection, it will not be able to distinguish different targets in the same category, and after one round, it may appear repeated or missed. Although the existing image stitching algorithm can achieve the purpose of all antennas being coplanar, it has certain conditions for input images and is difficult to apply in practice. Whether it is a traditional method based on scale-invariant feature transform or a method based on deep learning, it has high requirements for the resolution, shooting angle, and overlap rate of input images, and cannot be applied to signal tower high-altitude scenes. SUMMARY

[0005] The purpose of the present application is to provide a GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method, which solves the artifacts and blurring phenomenon that easily occurs under high-altitude perspective by constructing a GSII network for three-dimensional rendering, and uses a center pixel time sequence splicing module to synthesize multiple single perspective rendering images, and then obtains a high-altitude panoramic image of the signal tower.

[0006] In order to achieve the above purpose, the present application provides a GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method, comprising the following steps:

[0007] S1: An unmanned aerial vehicle flies around a signal tower and records, and the obtained video is uniformly frame-sampled to obtain an original image set, denoted as I, and specifically defined as:

[0008] I={i1,i2,···,i n}

[0009] In the formula: the numerical subscript represents the image number;

[0010] S2: The SfM algorithm is used to preliminarily three-dimensionally reconstruct the original image set to obtain a sparse point cloud and a camera pose matrix, and the sparse point cloud is 3D GS initialized, wherein each point is set as a center point of a three-dimensional Gaussian ball, and each Gaussian ball is assigned with attributes such as color, shape and transparency, and the set of all Gaussian balls is defined as a three-dimensional Gaussian point cloud;

[0011] S3: The three-dimensional Gaussian point cloud G can be projected to a two-dimensional plane through fast differentiable rasterization to obtain a preliminary rendering result I R , and the perspective is determined by the camera pose matrix obtained in the preliminary three-dimensional reconstruction, and each iteration will obtain a preliminary rendering result I R The input FFC residual repair module generates a deblurring rendering image I Deblur , and compares the pixels with the corresponding perspective original image, and uses the structure similarity loss function and the high receptive field perception loss function to update the network parameters in the reverse direction, and optimizes the attributes of each Gaussian ball in the three-dimensional Gaussian point cloud G;

[0012] S4: The FFC residual repair module after training can render and repair the signal tower at any perspective to obtain a super-definition rendering image of continuous perspective, and at this time the three-dimensional Gaussian point cloud G has been optimized into a dense point cloud, and the super-definition rendering image of continuous perspective is input into the center pixel time sequence splicing module to obtain a high-altitude panoramic image of the signal tower.

[0013] The step S2 of the present application is specifically:

[0014] S21: Based on the original image set I, the SfM algorithm can be used for preliminary three-dimensional reconstruction to obtain sparse three-dimensional point cloud data P SPThe camera pose matrix M corresponding to each image;

[0015] S22: Transfer sparse 3D point cloud data P SP Each point in the array is initialized as the center point of a three-dimensional Gaussian sphere, resulting in a three-dimensional Gaussian point cloud G, where G = {g1, g2, ..., g...} n}, where: g n Represents the nth Gaussian sphere;

[0016] S23: Each Gaussian sphere not only has the coordinates of its center point, but is also assigned a covariance matrix Σ, an opacity α, and a spherical harmonic function. Attribute, θ is the polar angle. It is the azimuth angle;

[0017]

[0018] In the formula: R is the rotation transformation matrix;

[0019] S is the scaling transformation matrix;

[0020] σ is the probability density of the light ray being blocked by the current point;

[0021] δ is the distance from the current point through which the ray passes to the next point continuing along that direction;

[0022] These are spherical harmonic basis functions;

[0023] c l,m The color coefficients are obtained and updated through model iteration.

[0024] Step S3 of the present invention specifically includes:

[0025] S31: The preliminary rendering result I is obtained by projecting the 3D Gaussian point cloud G onto a 2D plane using the fast differentiable rasterization method. R The number of images is the same as that in the original image set I, and the viewpoint is determined by the camera pose matrix M obtained during the initial 3D reconstruction, that is, the rendering viewpoint is the same as that of the original image.

[0026] S32: Next, the initial rendering result I R The input to the FFC residual repair module uses traditional convolution to extract local information in one channel and fast Fourier transform to extract global context information in the other channel. Finally, these are cross-fused and connected at the channel level to obtain the deblurred rendered image I. Deblur ;

[0027] S33: In each iteration, the deblurred rendering image I DeblurThe image is compared pixel-wise with the corresponding viewpoint original image in the original image set I, and a structural similarity loss function is constructed accordingly. This is then iteratively applied using stochastic gradient descent to optimize and update the attribute parameters of each Gaussian sphere in the 3D Gaussian point cloud G, thereby improving the accuracy of the signal tower's 3D Gaussian point cloud and deblurring the rendered image I. Deblur Compared with the initial rendering result I R By using a high receptive field perception loss function, the parameters of the FFC residual repair module are updated to improve the results in the next iteration.

[0028] Step S4 of the present invention specifically includes:

[0029] S41: After several rounds of iterative optimization, the 3D Gaussian point cloud G is transformed into a dense point cloud P. DS Then, by using rasterization and the trained FFC residual repair module, a set of high-altitude antenna ultra-high-definition rendered images I is obtained. A I A ={A1,A2,···,A n In this invention, the viewing angle of each image can be set arbitrarily, but in order to stitch the center pixels sequentially, the viewing angle between the images is set to change uniformly, and the n images are connected end to end to circle the signal tower once.

[0030] S42: High-Altitude Antenna Ultra-High Definition Rendering Image Collection I A Given n images with the same resolution, width w, and height h, if each image is considered as a matrix formed by the arrangement of pixels, then A n It can be represented as:

[0031]

[0032] A n With A n-1 Difference in perspective therefore and The observed details of the signal towers also differed. Image A n The The panoramic image O is obtained by sequentially stitching together the center pixels of n rendered images.

[0033]

[0034] Compared with existing technologies, this invention addresses the problems in 5G signal tower intelligent inspection tasks, where basic 3DGS models are prone to artifacts and blurring during large-scene 3D rendering, and the information contained in single-view rendered images is limited. It provides a signal tower 3D rendering and high-altitude panoramic image synthesis method based on GS II. First, frame extraction is performed on data captured by a drone flying around the signal tower to obtain an image sequence. Then, the sparse point cloud of the signal tower and camera pose are obtained through the Structure from Motion (SfM) algorithm, followed by 3DGS initialization. Crucially, during the iterative optimization of the 3D Gaussian point cloud, this invention constructs a GSII (Gaussian Splatting Image Inpainting) network. This network compares the structural similarity of the original image and the deblurred rendered image from the same viewpoint, uses a high receptive field perceptual loss function and backpropagation, and updates the parameters of the 3D Gaussian sphere properties and FFC residual repair module to improve the quality of the new viewpoint rendered image. Finally, a center pixel temporal stitching module is designed to transform any number of single-view rendered images into a panoramic image. This invention solves the problems of artifacts and blurring that easily occur from high-altitude perspectives by constructing a GSII network for 3D rendering. At the same time, it uses a central pixel temporal stitching module to synthesize multiple rendered images from a single perspective, thereby obtaining a panoramic view of the signal tower from high altitude. Attached Figure Description

[0035] Figure 1 This is a general framework diagram of the method of the present invention;

[0036] Figure 2 This is a preliminary 3D reconstruction result of SfM;

[0037] Figure 3 This is a visualization of the 3D Gaussian point cloud after iterative optimization.

[0038] Figure 4 The original image is the input image from a specific viewpoint;

[0039] Figure 5 To and Figure 4 Preliminary rendering results from the same perspective;

[0040] Figure 6 This is the optimized rendered image;

[0041] Figure 7 A schematic diagram illustrating the temporal stitching of center pixels for rendering an image;

[0042] Figure 8 This is a composite panoramic view of the signal tower from above.

[0043] Figure 9 This is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0044] The application will be further described below with reference to the drawings.

[0045] Important components such as antennas of signal towers are in high-altitude scenes, if the method of 3DGS is directly used for three-dimensional reconstruction and rendering, artifacts and blur phenomena will appear, and only through single-view rendering, only partial information of the signal tower can be obtained. In order to solve this problem, panoramic images need to be generated to directly observe all components on the tower, but the existing image stitching method has high requirements for input, especially in the overlapping rate of images, and is not suitable for camera revolution scenes (circulating around a target) with large background changes. In order to effectively solve these problems, the application discloses a 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII;

[0046] As shown in Figure 1 and Figure 9 , a 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method based on GSII, comprising the following steps:

[0047] Step S1: collect original image data and perform preliminary sparse reconstruction;

[0048] As shown in the overall framework diagram of Figure 1 , in the process of image data collection, the flight path of the unmanned aerial vehicle is set to circulate around the tower and record, and the obtained video is uniformly frame-extracted to obtain an original image set, denoted as I;

[0049] I={i1,i2,···,i n}

[0050] In the formula: the numerical subscript represents the image serial number;

[0051] The frame extraction processing of the application is prior art, which will not be described in detail;

[0052] S2: using SfM algorithm to perform preliminary three-dimensional reconstruction on the original image set, obtaining sparse point cloud and camera pose matrix, and performing 3DGS initialization on the sparse point cloud, each point in which is set as the center point of a three-dimensional Gaussian ball, and each Gaussian ball is assigned with attributes such as color, shape, and transparency, and the set of all Gaussian balls is defined as a three-dimensional Gaussian point cloud;

[0053] S21: input the original image set I={i1,i2,…,i nThe initial 3D reconstruction was performed using the SfM algorithm (Schonberger JL, Frahm JM. Structure-from-Motion Revisited [C] / / IEEE Conference on Computer Vision & Pattern Recognition. IEEE, 2016: 4104-4113.), resulting in the following... Figure 2 The sparse 3D point cloud data P shown SP The camera pose matrix M corresponding to each image in the original image set I, and the camera pose matrix of the nth image is denoted as Mn. n ;

[0054]

[0055] In the formula: R WC This is the rotation matrix of the camera relative to the world coordinate system;

[0056] T represents the translation vector;

[0057] S22: Transfer sparse 3D point cloud data P SP Perform 3D GS initialization, initializing each point as the center point of a 3D Gaussian sphere, to obtain the initial 3D Gaussian point cloud G, G = {g1, g2, ..., g...} n}, where: g n Let g represent the nth Gaussian sphere. For any Gaussian sphere g, the mathematical form can be expressed as:

[0058]

[0059] Specifically, its shape is described by the covariance matrix Σ, which is a positive definite symmetric matrix and represents the optimization parameters during scene learning. Σ is decomposed into a rotation matrix R and a scaling matrix S, i.e.

[0060] Σ=RSS T R T

[0061] Each Gaussian sphere, in addition to its shape attribute, also has a color attribute, determined by its opacity α and spherical harmonic function. Joint decision;

[0062] Opacity α is specifically defined as:

[0063] α = 1 - e -σδ ;

[0064] Where: σ represents the probability density of the light ray being blocked by the current point;

[0065] δ represents the distance from the current point through which the ray passes to the next point continuing along that direction;

[0066] spherical harmonics Specifically defined as:

[0067]

[0068] Where: θ is the polar angle;

[0069] It is the azimuth angle;

[0070] These are spherical harmonic basis functions;

[0071] c l,m The color coefficients are obtained and updated through model iteration.

[0072] Therefore, for any Gaussian ellipsoid in the initial three-dimensional Gaussian point cloud G, the following property is present:

[0073]

[0074] S3: GSII network training optimization;

[0075] The shape and color attributes of the Gaussian spheres mentioned above are based on the sparse 3D point cloud data P obtained from the preliminary 3D reconstruction using SfM. SP Multiple iterations and optimizations are still needed to improve accuracy.

[0076] Specifically, the Tile-based Rasterizer technique was used. First, the two-dimensional plane was divided into small regions to reduce the amount of computation. Second, the Sigmoid function was used to soften the pixel coverage determination and calculate the impact on the pixels in a probabilistic form. At the same time, depth conflicts were handled by weighted mixing to preserve the gradient of interpolation attributes (such as color and shape). Finally, the independent results of each region were merged into a complete image.

[0077] When projecting the points of a 3D Gaussian point cloud G onto a 2D pixel plane, the color C of each pixel is determined by the alphablending method, specifically defined as:

[0078]

[0079] Where: c i Represents the spherical harmonic function at point i;

[0080] a i Indicates the opacity of point i;

[0081] Next, the initial rendering result I RThe input to the FFC residual repair module specifically involves two parallel branch channels. One channel uses traditional convolution to extract local information, while the other uses Fast Fourier Transform to extract global information. Finally, these are cross-fused and connected to the channel level to obtain the deblurred rendered image I. Deblur ;

[0082] In each iteration, any original image i in the original image set I... j Each has a corresponding preliminary rendering result I R Both cameras are in the same pose, such as Figure 3 and Figure 4 As shown, a pixel-by-pixel comparison is then performed, and the structural similarity loss function is used for calculation. The results are then backpropagated through stochastic gradient descent to optimize the 3D Gaussian point cloud G.

[0083] Deblurring and rendering image I Deblur Then compare with the initial rendering result I R By using a high receptive field perceptual loss function, the parameters of the FFC residual inpainting module are updated to improve the quality of the final output image, such as... Figure 5 As shown;

[0084] S4: Center pixel temporal stitching module: After training, the FFC residual repair module can repair the signal tower rendering image from any angle to obtain a continuous view ultra-high-definition rendering image. Then, the continuous view ultra-high-definition rendering image is input into the center pixel temporal stitching module to obtain a high-altitude panoramic image of the signal tower.

[0085] After several iterations of optimization, the 3D Gaussian point cloud G is transformed into a dense point cloud P. DS ,like Figure 3 As shown, compared to the sparse 3D point cloud data P obtained from the SfM algorithm, SP Its shape, texture, and color are closer to those of real-world signal towers;

[0086] For dense point cloud P DS By performing rendering and restoration from any viewpoint, a set of ultra-high-definition rendered images of high-altitude antennas, I, can be obtained. A , represented as:

[0087] I A ={A1,A2,···,A n}

[0088] Among them: A n This represents the nth rendered image;

[0089] The viewpoint of each rendered image can be freely set according to the specific situation of different signal towers. However, in order to perform subsequent time-sequential stitching of the center pixels, this invention sets the viewpoint to change uniformly. The n images are joined end-to-end, and the viewpoint travels around the signal tower once. That is, the viewpoint difference between adjacent images is...

[0090] Through experiments, considering both the quality of the panoramic image generated after pixel stitching and the algorithm efficiency, this invention sets n=360, that is, the parallax between two adjacent images is 1°;

[0091] High-altitude antenna ultra-high-definition rendering image collection I A There are n images with the same resolution, width w, and height h;

[0092] If each image is considered as a matrix formed by the arrangement of pixels, then A n It can be represented as:

[0093]

[0094] Because the n images are evenly distributed around the perimeter, A n With A n-1 Difference in perspective The middle column of pixels between the two can be denoted as... and The details of the signal tower represented by these two columns of pixels also differ.

[0095] Arrange and stitch together the middle column of pixels of all rendered images according to the order of the circular motion of the viewpoint, as follows: Figure 7 As shown, a panoramic image O displaying all antennas can be obtained, such as... Figure 8 As shown, it is defined as:

[0096]

[0097] This invention proposes a 3D rendering and high-altitude panoramic image synthesis method for 5G signal towers based on Gaussian GS II (GS II) to better accomplish intelligent signal tower inspection tasks. In large-scale scenes from a high altitude perspective, direct 3D reconstruction rendering using 3DGS results in artifacts and blurring, and rendering images from only a single viewpoint makes it difficult to grasp the global information of the signal tower. Therefore, a GSII network is specifically designed. During the iterative optimization of the 3D Gaussian point cloud, an FFC residual repair module is introduced to improve the accuracy of the Gaussian sphere and the quality of the rendered image. Furthermore, a central pixel temporal stitching module is designed at the network end to synthesize rendered images from a circumferential viewpoint around the tower, obtaining a high-quality panoramic image of all tower components coplanar, solving the problem of limited information from a single viewpoint. Observing the signal tower on this panoramic image not only eliminates the safety hazards of manual tower climbing but also improves automation, making the 5G signal tower inspection process more convenient and efficient. This invention can completely replace manual tower climbing for signal tower inspection, improving the accuracy of results and making 5G signal tower inspection safer, more convenient, and more efficient.

Claims

1. A GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method, characterized by, The method comprises the following steps: S1: A drone flies around the signal tower for a round and records, and the obtained video is uniformly frame-sampled to obtain an original image set, denoted as , which is specifically defined as: In the formula, the numerical subscript represents the image sequence number; S2: using SfM algorithm to perform preliminary three-dimensional reconstruction on the original image set to obtain sparse point cloud and camera pose matrix, performing 3DGS initialization on the sparse point cloud, wherein each point is set as a center point of a three-dimensional Gaussian ball, and each Gaussian ball is assigned color, shape and transparency attributes, and the set of all Gaussian balls is defined as a three-dimensional Gaussian point cloud ; S3: Transforming 3D Gaussian point clouds using fast differentiable rasterization Projecting onto a two-dimensional plane yields preliminary rendering results. The viewpoint is determined by the camera pose matrix obtained in the initial 3D reconstruction, and each iteration renders the initial rendering results. Input the FFC residual repair module to generate a deblurred rendered image. The network parameters are then compared pixel-wise with the original image from the corresponding viewpoint. A structural similarity loss function and a high receptive field perception loss function are used for backpropagation to update the network parameters, thereby optimizing the 3D Gaussian point cloud. The properties of each Gaussian sphere; S4: the FFC residual repair module after training can render and repair the signal tower at any viewing angle to obtain a super-definition rendering image of continuous viewing angle, and at this time the three-dimensional Gaussian point cloud has been optimized into a dense point cloud, and the super-definition rendering image of continuous viewing angle is input into a center pixel time sequence splicing module to obtain a high-altitude panoramic image of the signal tower; The step S3 is specifically: S31: Project the three-dimensional Gaussian point cloud to a two-dimensional plane by a fast microlithographic method, and obtain a preliminary rendering result G S32: The number of the preliminary rendering result is consistent with the number of the original image set S33: The view angle is determined by the camera pose matrix obtained during the preliminary three-dimensional reconstruction, that is, the rendering view angle is the same as the original image M ​​ S32: Then the preliminary rendering result is refined The input FFC residual repair module extracts local information using traditional convolution in one channel, extracts global context information using fast Fourier transform in another channel, and finally performs cross fusion and channel level connection to obtain a deblurring rendered image ; S33: In each round of iteration, deblur the rendered image With the original image set Pixel comparison is made between the corresponding view original images in each pair, and a structural similarity loss function is constructed therefrom. Through random gradient descent method, the three-dimensional Gaussian point cloud is optimized and updated The attribute parameters of each Gaussian ball in the original image set, thereby improving the accuracy of the signal tower three-dimensional Gaussian point cloud and deblurring the rendered image With the preliminary rendering result Through the high receptive field perception loss function, the FFC residual repair module parameters are updated to improve the results in the next round of iteration.

2. The GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method according to claim 1, characterized in that, The step S2 is specifically: S21: based on the original image set , a preliminary three-dimensional reconstruction is performed by an SfM algorithm to obtain sparse three-dimensional point cloud data A camera pose matrix corresponding to each image ; S22: Sparse 3D point cloud data Each point in the cloud is initialized as the center point of a three-dimensional Gaussian sphere, resulting in a three-dimensional Gaussian point cloud. , In the formula: Indicates the first One Gaussian ball; S23: Each Gaussian sphere not only possesses a center point coordinate, but is also endowed with a covariance matrix opacity and spherical harmonics attributes, is the polar angle, is the azimuthal angle; ； In the formula: R is a rotation transformation matrix; S is a scale transform matrix; is the probability density that a ray is obstructed by the current point; is the distance from the current point, through which the light ray passes, to the next point in the direction of continuation; is a spherical harmonic basis function; C is the color coefficient.

3. The GSII-based 5G signal tower three-dimensional rendering and high-altitude panoramic image synthesis method of claim 1, wherein The step S4 is specifically: S41: After several rounds of iterative optimization, the three-dimensional Gaussian point cloud is converted into a dense point cloud , and then a high-altitude antenna super-definition rendering image set is obtained through rasterization and the trained FFC residual repair module , , wherein the viewing angle of each image can be customized, but in order to splice the center pixel timing later, the viewing angles between images are set to change uniformly, and the first and last images are connected to form a circle around the signal tower. S42: high-altitude antenna super-definition rendering image set In The resolution of each image is the same, the width is , and the height is If each image is regarded as a matrix formed by pixel points, then is represented as: ； with a viewing angle difference therefore with the observed signal tower details also differ , representing an image of the first column of pixels, the panorama image can be obtained after the center pixel time sequence splicing of the : 。

Images