A method and system for super-resolution of spherical images

Abstract

This invention provides a method and system for super-resolution of spherical signals, comprising: constructing a parameterized spherical icosahedral triangular mesh; projecting an equidistant rectangular projection image (ERP) from two-dimensional space onto the icosahedral triangular mesh to obtain a spherical mesh signal; passing the spherical mesh signal through a first feature extractor to obtain a first multi-scale feature; passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature; expanding the second multi-scale feature based on an adaptive filter learning of Chebyshev polynomials and spherical pixel recombination to obtain spherical mesh expanded features; reconstructing the upsampled spherical mesh signal; and inversely projecting the upsampled spherical mesh signal onto the ERP to obtain a super-resolution spherical image. This invention enables high-quality representation learning and detail characterization of low-resolution spherical image inputs, especially in ultra-high magnification image super-resolution tasks, generating super-resolution spherical images with more complete details and less distortion.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a method and system for super-resolution of spherical images. Background Technology

[0002] With the rise of 360-degree panoramic cameras, spherical signal processing has become a common concern in academia and industry. Considering the breadth of the field of view, spherical signals often require extremely high resolution to maintain detailed signal features. However, limited by the performance of panoramic camera equipment and data transmission costs, industry often cannot directly obtain or process high-resolution images, and therefore needs to convert low-resolution images into high-resolution images through signal reconstruction.

[0003] The search revealed:

[0004] The early paper "Learning spherical convolution for fast features from 360imagery" extended convolutional neural networks for traditional images, using convolutional kernels of different sizes to perform convolution operations on ERP images based on the different degrees of distortion at different latitudes. "SphereNet: Learning spherical representations for detection and classification in omnidirectional images" adopted a similar idea, introducing equal distortion to the convolutional kernels to adapt to convolution operations on ERP images at different latitudes. Deng et al., in "Lau-net: Latitude adaptive upscaling network for omnidirectional image super-resolution," divided the ERP image into several parts according to latitude. Following the common knowledge that information is abundant near the equator and scarce near the poles, they applied different upsampling operations to the sub-images at different latitudes, and obtained a super-resolution ERP image by fusing the upsampling results from different latitudes. In his paper "Analysis of Sparse Coding Models for Image-Based Classification," Nishiyama introduced an additional distortion matrix to measure the degree of distortion at each location in the ERP image, in addition to using the ERP image as input. He proved that concatenating the input image with the distortion matrix as input can improve the performance of the ERP image super-resolution model. However, directly upsampling the ERP image in the two-dimensional plane cannot completely eliminate the influence of distortion in the ERP image, which is not conducive to better defining the neighborhood of a pixel and capturing its neighborhood features.

[0005] In subsequent work, super-resolution on spheres is mainly achieved by upsampling images onto a sphere using spherical meshes. Wu et al., in "SMSIR: Spherical measure based spherical image representation," incorporated triangles from a spherical triangular mesh generated by a regular octahedron into a recursive index, deriving a spherical mesh lifting wavelet architecture under this index coordinate, and resampling the spherical signal through the learning of the spherical signal's implicit function. In "Gauge equivariant convolutional networks and the icosahedral CNN," Cohen et al. unfolded an icosahedral spherical mesh and extended convolutional kernels for 2D image processing onto the mesh, proposing a canonical equivariant network on a sphere. Yoon et al., in "Spheresr: 360deg image super-resolution with arbitrary projection via continuous spherical image representation," further constructed directional convolutional kernels based on unfolding an icosahedral spherical mesh and designed an implicit function learning method suitable for spherical super-resolution. However, due to the complex structure of the icosahedral spherical mesh and the presence of discontinuous edges in the unfolded diagram, it is difficult to construct index coordinates. Therefore, how to efficiently process spherical icosahedral triangular mesh signals has become an urgent problem to be solved. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for super-resolution of spherical images, along with a corresponding computer terminal and computer-readable storage medium.

[0007] According to a first aspect of the present invention, a spherical image super-resolution method is provided, comprising:

[0008] A parameterized spherical icosahedral triangular mesh is constructed, and the equidistant rectangular projection image ERP is projected forward from the two-dimensional space onto the icosahedral triangular mesh to obtain the spherical mesh signal.

[0009] The spherical mesh signal is passed through a first feature extractor to obtain a first multi-scale feature;

[0010] The first multi-scale feature is passed through the second feature extractor to obtain the second multi-scale feature;

[0011] The second multi-scale feature is extended by adaptive filter learning based on Chebyshev polynomial and spherical pixel recombination to obtain spherical grid extended features and reconstruct the upsampled spherical grid signal.

[0012] The upsampled spherical grid signal is back-projected using ERP to obtain a super-resolution spherical image.

[0013] Preferably, the construction of the parameterized spherical icosahedral triangular mesh includes:

[0014] Calculate the spherical coordinates of all grid points in a spherical icosahedral triangular mesh;

[0015] Based on the calculated spherical coordinates of all grid points, a spherical regular icosahedral triangular mesh and grid point index are recursively generated, and the neighbor index of the grid points is updated.

[0016] By using the spherical icosahedral triangular mesh point search method, the correspondence between the spherical mesh and the spherical coordinates is established.

[0017] Preferably, the calculation of the spherical coordinates of all grid points in the spherical icosahedral triangular mesh includes:

[0018] Depending on the latitude, the i-order spherical grid divides the sphere into 3×2 i-1 +1 different layers, respectively dividing layers 0 to 2 i-1 , 2nd i-1 ~2×2 i-1 and the 2nd × 2 i-1 ~3×2 i-1 Layers are defined as the upper, middle, and lower parts of a spherical mesh;

[0019] Let N(j) be the number of grid points in the j-th layer.

[0020]

[0021] In this spherical icosahedral mesh, the grid points in each layer are evenly distributed along this latitude, and the latitudinal spacing of the upper, middle, and lower parts of the mesh is consistent. Based on the properties of the icosahedron, the second... i-1 Layer and 2 i The latitudes of the layers are arctan(1 / 2) and -arctan(1 / 2), respectively;

[0022] Based on latitude information, the spherical coordinates of the k-th grid point in the j-th layer are obtained as follows:

[0023]

[0024] Preferably, the step of recursively generating a spherical icosahedral triangular mesh and grid point indices based on the calculated spherical coordinates of all grid points, and updating the neighbor indices of the grid points, includes:

[0025] Recursively generate a spherical icosahedral triangular mesh, specifically as follows:

[0026] When a low-order spherical mesh is upsampled to generate a high-order spherical mesh, the grid points and their indices of the low-order spherical mesh can be embedded in the high-order spherical mesh.

[0027] For an i-th order regular icosahedral spherical triangular mesh, the total mesh size is 10×4. i-1 +2 grid points;

[0028] When generating an (i+1)th order triangular mesh from an i-th order triangular mesh, an additional 3×10×4 mesh is required. i-1 For each grid point in the i-th order triangular mesh, except for the two poles of the spherical mesh, grid points are downsampled to the upper right, right side, and lower right, generating a 3×10×4 grid. i-1 One downsampling grid point;

[0029] The grid point indices of the spherical icosahedral triangular mesh are generated recursively, specifically as follows:

[0030] For a first-order spherical regular icosahedral triangular mesh, the north and south pole grid points of the spherical mesh are indexed as 0 and 1 respectively. Divided by the 0-degree meridian, the indices of the five grid points in the Northern Hemisphere are 2-6 from west to east, and the indices of the five grid points in the Southern Hemisphere are 7-11 from west to east.

[0031] For a grid point with index idx in an i-th order triangular mesh (excluding the poles), the indices of the downsampled grid points in the upper right, right side, and lower right of the grid point generated from the i-th order triangular mesh are idx+10×4 respectively. i idx+2×10×4 i idx+3×10×4 i ;

[0032] Update the neighbor index of the grid point, specifically:

[0033] The icosahedral spherical triangular mesh stores the adjacent point index of each grid point in the order of "top right, right side, bottom right, top left, left side, bottom left".

[0034] For the 12 vertices of a regular icosahedral mesh, for the poles numbered 0 and 1, the spherical mesh uses the grid point's own index as the index of the 6th adjacent point; for grid points with indices 2-6, the upper left and upper right are the same grid point index; for grid points with indices 7-11, the lower right and lower left are the same grid point index.

[0035] For an i-th order triangular mesh, the neighbor index information of the grid points is denoted as n. c = (c1, c2, ..., c6), define the index generation function f. i (idx a idx b )=f(idx a idx b ,n a ,n b ,i), where idx a idx b These are the indices of grid points a and b, respectively, and n a ,n b are the indices of the adjacent points of grid points a and b, respectively, and i is the order of the grid.

[0036] Define p1 and p2 as idx a In n b The minimum value of the index and idx b In n a The minimum value of the subscript in the middle; if a is a pole, then p1 = p2, idx a =idx b If b is a pole, then p2 = p1, idx b =idx a ;at this time,

[0037]

[0038] When generating the grid point index of an i-th order spherical triangular mesh based on an (i-1)-th order spherical triangular mesh and its grid point index, for a grid point c in the (i-1)-th order spherical triangular mesh and its adjacent grid point index information n... c = (c1, c2, ..., c6), where the adjacency index information of c in the i-th order spherical triangular mesh is:

[0039] n′ c =(f i (idx c ,c1),f i (idx c ,c2),…,f i (idx c ,c6))

[0040] The adjacent point indices of the grid point c in the (i-1)th order spherical triangular mesh are obtained by sampling to the upper right, right side, and lower right of the generated grid point, respectively, and are n'. up ,n' mid ,n' low ;

[0041] If grid point c is derived from grid points indices 2-6 in a first-order spherical grid through stepwise downsampling, then

[0042] n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c4,c5),f i (idx c ,c5),idx c )

[0043] n′ mid =(f i (c1,c2),c2,f i (c2,c3),f i (idx c ,c1),idx c ,f i (idx c ,c3))

[0044] n′ low =(f i (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c6),f i (c3,c6))

[0045] If grid point c is derived from grid points indices 7-11 in a first-order spherical grid through successive downsampling, then

[0046]

[0047] Preferably, the method of establishing the correspondence between the spherical grid and the spherical coordinates through the spherical icosahedral triangular grid point search method includes:

[0048] To determine an approximate grid point v on an arbitrary spherical surface with coordinates (φ', θ'), specifically:

[0049] Based on the fact that all grid points except the poles in the high-order spherical regular icosahedral triangular mesh are derived from the grid points with index 2-11 in the first-order triangular mesh through recursive upsampling, the grid points with index 2-11 are named "generated points".

[0050] For any spherical coordinate (φ', θ'), its latitude information is obtained. By comparing the latitude information with that of different layers of the triangular mesh, the layer j closest to this coordinate is determined.

[0051] The nearest grid point v = Cor(k) to (φ', θ') is determined by the distance between φ' and the j-th layer grid point.v ,j v );

[0052] Based on the neighbor information n of the approximate grid point v v To perform a traversal and determine its index, specifically:

[0053] Let the point where v is generated be denoted as v0 = Cor(k0,j0), then we have

[0054] |k v -k0|+|j v -j0|<2 i-1 ;

[0055] Let the index of v0 be idx0, k v = kmod(N(j) / 5), where N(j) is the number of grid points in the j-th layer; based on the above relationship, calculate the index idx0 of the generated point v0, and v is equivalent to the displacement shift of v0.

[0056]

[0057] For a grid point (0) located at the top of an i-th order grid... <j<2 i-1 Let's define the displacement vectors for the upper right, right side, and lower right respectively: (0,-1), (1,0), (1,1); for the second... i-1 For the grid points of the first layer, define the displacement vectors for the upper right, right side, and lower right respectively: (0,-1), (1,0), (0,1); for the grid points of the remaining layers (2... i-1 <j≤3×2 i-1 Define the displacement vectors for the upper right, right side and lower right respectively: (1,-1), (1,0), (0,1);

[0058] Based on the defined displacement, the displacement shift of the approximate grid point v to its "generation point" v0 is inversely calculated;

[0059] Let the displacements generated during upsampling of the spherical grid points be denoted as choice.

[0060]

[0061] For grid point v and displacement shift, starting from grid unit length unit = 1, find a suitable displacement choice[m] from the choice such that shift-choice[m]×unit is divisible by 2×unit;

[0062] Update shift = shift - choice[m] × unit, unit = 2 × unit, and record a. i =m∈{0,1,2,3});

[0063] Repeat this step until unit = 2 i-1 At this point, shift = (0,0);

[0064] Based on the generated point index idx0 and the records a2, a3, ..., a i Calculate Cor(k) v ,j v The index idx v

[0065]

[0066] Based on Cor(k) v ,j v The index idx v Determine the triangular mesh surface containing spherical coordinates (φ', θ').

[0067] Preferably, the step of projecting the ERP image from two-dimensional space onto the parameterized sphere to obtain the spherical mesh signal specifically involves:

[0068] By utilizing the transformation relationship between arbitrary spherical coordinates (φ, θ) and two-dimensional coordinates (h, w), the ERP image is projected from two-dimensional space onto the parameterized sphere to obtain the spherical grid point coordinates corresponding to the two-dimensional coordinates (h, w).

[0069]

[0070] Where H and W are the height and width of the two-dimensional ERP image, respectively;

[0071] For a spherical grid point with coordinates (φ0, θ0) and its corresponding two-dimensional coordinates (h0, w0), let the coordinates of the four closest two-dimensional image pixel centers to (h0, w0) be (h1, w1), (h2, w2), (h3, w3), and (h4, w4), located at the upper left, upper right, lower left, and lower right of (h0, w0) respectively, and their corresponding pixel values ​​be val1, val2, val3, and val4. Then the pixel value of this spherical grid point is...

[0072]

[0073] Where S1 = (h4-h0)×(w4-w0), S2 = (h3-h0)×(w0-w3), S3 = (h0-h2)×(w2-w0), and S4 = (h0-h1)×(w0-w1); the spherical grid pixel value val will be used for subsequent feature extraction and spherical image reconstruction of the spherical image signal;

[0074] Based on the obtained spherical grid point coordinates corresponding to the two-dimensional coordinates (h,w), the triangular grid surface of the spherical grid where the spherical grid point coordinates corresponding to the two-dimensional coordinates (h,w) are located is obtained through the correspondence between the spherical grid and the spherical coordinates.

[0075] Preferably, the first feature extractor includes an anisotropic spherical scattering network and a channel attention mechanism;

[0076] The step of passing the spherical grid signal through a first feature extractor to obtain the first multi-scale feature specifically involves:

[0077] Multi-scale spherical wavelets in anisotropic spherical scattering networks are used to extract multi-directional, multi-scale features of spherical signals based on different azimuth and rotation angles. Specifically:

[0078] The basis functions of the directional spherical wavelet are related to both longitude and latitude, and the wavelet coefficients... Calculated as

[0079]

[0080] Where f(ω) is the input spherical signal, Convolution in the spherical direction For wavelet functions on a sphere, ρ=(α,β,γ)∈SO(3) rotation group, It is a rotation group.

[0081] Anisotropic spherical wavelets exhibit localization properties in scale, location, and direction. The wavelet scale... Represents Ψ j Angular locality; by combining the function f and wavelet Ψ j Expanding them into spherical harmonic space yields their respective spherical harmonic coefficients. and Transformation writing

[0082]

[0083] Based on the above transformation, the scaling coefficient W Φ (ω) is calculated as

[0084]

[0085] Where ⊙ represents the axisymmetric convolution on a sphere, and the rotation operator is written as... but The scaling function is an axisymmetric function.

[0086] Spherical harmonic space computational scaling factor writing

[0087]

[0088] Constructing a scattering network based on the above anisotropic spherical wavelets has rotational invariance:

[0089] For the input spherical grid signal The first-step scattering propagation of the scattering convolutional network based on anisotropic spherical wavelets is represented by ;

[0090] For an azimuth angle with a bandwidth of N, the wavelet coefficients at any azimuth angle γ are written as a linear combination of N basic wavelet coefficients

[0091]

[0092] where the rotation angle γ g ∈[0,2π), is an interpolation function;

[0093] Using the modulus of the wavelet coefficients as the signal on the rotation group L 2 (SO(3)), is composed of linear combination;

[0094] Denote the scattering propagation U[j]f(α,β,γ g ) at the azimuth angle (α,β) as U[j]f(γ g ). For a propagation path p = ((j1,g1),…,(j m ,g m )) (j i >j i+1 , i = {1,…,m}) with a decreasing frequency of length m, the corresponding scattering propagation is

[0095] <00006(09>The scattering coefficient output by path p is calculated as <00006(11>

[0098] where 0 ≤ J0 < J, J is the maximum wavelet scale, is an axisymmetric scaling function;

[0099] Using to take the convolution ensures rotational invariance within the range determined by J0; the scattering transform iteratively decomposes the signal into various directions and scales, and collects the scattering path coefficients of different lengths to obtain the scattering features of the spherical grid signal;

[0100] H sct = Concat({S[p]f|p ∈ ∪Ω i , 0 ≤ i ≤ M - 1})

[0101] Where M is the number of layers in the anisotropic spherical scattering network, Ω i Let i be the set of all frequency-decreasing propagation paths of length i.

[0102] Input X = [X] RGB three-color spherical signal (R) ,X (G) ,X (B) Anisotropic scattering characteristics were obtained respectively. in

[0103]

[0104] The multi-azimuth, multi-scale information, i.e., scattering features, output from the anisotropic spherical scattering network is aggregated through a channel attention mechanism, specifically:

[0105] for x = R, G, B, and channel coefficients are obtained by performing max pooling and mean pooling at the node dimension, respectively. a max ,a mean The channel attention weight 'a' is obtained by using a two-layer shared-parameter perceptron and then summing the results through a non-linear activation layer. channel :

[0106] a channel =σ(MLP(a) max )+MLP(a mean ))

[0107] Based on channel attention weight a channel scattering characteristics Compression to single-channel features

[0108]

[0109] Compressed scattering characteristics The signal is fused with the original signal to obtain the first multi-scale feature, which is then used as the input to the second feature extractor.

[0110] Preferably, the second feature extractor includes K densely connected feature extraction units, each feature extraction unit including a graph scattering attention mechanism and a dense convolutional network;

[0111] The step of obtaining second multi-scale features by passing the first multi-scale features through a second feature extractor includes:

[0112] The input to the first feature extractor unit is the first multi-scale feature. Output d1-dimensional point features in, For the real number field;

[0113] The input of the k-th feature extraction unit is the output of the (k - 1)-th feature extraction unit Through cross-layer connection, X k-1 is concatenated with the output feature X of the dense convolutional network of the k-th feature extraction unit k,out to obtain the output feature of the k-th feature extraction unit where k = 2, …, K

[0114] Preferably, the step of obtaining the second multi-scale feature by passing the first multi-scale feature through the second feature extractor includes:

[0115] For the k-th feature extraction unit, the input feature X k-1 is compressed to a specified dimension through a convolutional layer to obtain a compressed feature

[0116] The obtained compressed feature is passed through a graph scattering attention mechanism, and the features output by different scattering paths in the scattering network are aggregated by weight learning to obtain an adaptive multi-scale feature representation H containing the low-pass and band-pass features of the compressed feature gsan ;

[0117] The adaptive multi-scale feature representation H of the input feature gsan aggregates all lattice point features within l-hop through a three-layer dense graph convolutional network to output the depth feature of this feature extraction unit

[0118] The output features X of the first k feature extraction units are concatenated k = Concat(X 1,out , …, X k,out ) as the input of the (k + 1)-th feature extraction unit (1 ≤ k < K) or as the final output of the feature extractor (k = K).

[0119] Preferably, the step of performing feature expansion on the second multi-scale feature based on the adaptive filter learning of Chebyshev polynomials and spherical pixel recombination to obtain a spherical grid expansion feature and reconstruct an upsampled spherical grid signal includes:

[0120] Adaptive filter learning based on Chebyshev polynomials is used to perform feature expansion on the output feature X of the second feature extractor K Specifically:

[0121] ?When upsampling from an i-th order spherical grid signal to an (i + 1)-th order spherical grid signal, the spherical signal representation needs to be expanded to 4 times the original, and this feature expansion uses adaptive filter learning based on Chebyshev polynomials

[0122]

[0123] T i (λ) is a Chebyshev polynomial, H is the highest-order constraint of the Chebyshev polynomial, and θ i and Ξ i All parameters are learnable; X is trained using four different adaptive filters F1(λ), ​​F2(λ), F3(λ), and F4(λ). K Filtering, through cross-layer connections, yields extended spherical signal characteristics.

[0124]

[0125] Where P = 1 / 2(I + AD) -1 Let Θ be the lazy random walk matrix of the spherical grid. i These are learnable parameters;

[0126] Spherical mesh features generated based on different filters The extended features are reorganized into spherical icosahedral mesh pixels to generate a higher-order icosahedral spherical mesh and its corresponding spherical mesh features, specifically as follows:

[0127] Extending the signal characteristics from the i-th order spherical grid to the (i+1)-th order spherical grid, the grid points are divided into four groups according to the (i+1)-th order spherical grid point index, and then... The correspondence is as follows: the (i+1) order spherical triangular mesh index and the spherical triangular mesh grid point features correspond to each other, in the order of 0 to 10×4^i+1. 10×4 i +2~20×4 i +1 corresponds to 20×4 i +2~30×4 i +1 corresponds to 30×4 i +2~40×4 i +1 corresponds to

[0128] Will By fusing the (i+1)th order spherical mesh, pixel recombination of the spherical triangular mesh is achieved, resulting in the extended feature X of the spherical mesh. out ;

[0129] X based on high-order spherical mesh feature representation out Upsampled spherical signal pixel reconstruction Z = FC(σ(FC(X)) is achieved through a two-layer fully connected network. out ))).

[0130] Preferably, the step of obtaining a super-resolution spherical image by inverse projection of the upsampled spherical grid signal using ERP includes:

[0131] For any two-dimensional coordinates (h, w), the following relationship exists between them and the spherical coordinates (φ, θ):

[0132]

[0133] Using spherical coordinates, we can obtain the triangular face containing the spherical coordinates and the spherical coordinates of the three vertices.

[0134] Based on spherical coordinates, vertex coordinates of triangular faces, and pixel values, the pixel value at (h,w) in the two-dimensional pixel space is obtained through triangular face interpolation.

[0135] By utilizing the relationship between two-dimensional coordinates and spherical coordinates, the Z-axis of the upsampled spherical signal pixels is reconstructed and projected onto the two-dimensional pixel plane from both coordinate and pixel value perspectives.

[0136] According to a second aspect of the present invention, a spherical image super-resolution system is provided, comprising:

[0137] ERP projection module: Constructs a parameterized spherical icosahedral triangular mesh, projects the equidistant rectangular projection image ERP forward from two-dimensional space onto the icosahedral triangular mesh, and obtains the spherical mesh signal;

[0138] First feature extraction module: The spherical grid signal is passed through a first feature extractor to obtain first multi-scale features;

[0139] Second feature extraction module: The first multi-scale feature is processed by the second feature extractor to obtain the second multi-scale feature;

[0140] Feature extension module: The second multi-scale feature is extended by adaptive filter learning based on Chebyshev polynomial and spherical pixel recombination to obtain spherical grid extended features and reconstruct the upsampled spherical grid signal;

[0141] ERP Image Reconstruction Module: The upsampled spherical grid signal is back-projected using ERP to obtain a super-resolution spherical image.

[0142] According to a third aspect of the present invention, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described herein, or to run the system described herein.

[0143] According to a fourth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described herein, or to run the system described herein.

[0144] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with existing spherical image super-resolution technology:

[0145] The spherical image super-resolution method and system provided in this invention can perform high-quality representation learning and detail characterization on low-resolution spherical image inputs, especially in ultra-high magnification image super-resolution tasks, generating super-resolution spherical images with more complete details and less distortion.

[0146] The spherical image super-resolution method and system provided in this invention adopts a parameterizable spherical icosahedral triangular mesh, which solves the problem of node encoding difficulties caused by the discontinuity of the unfolded graph of the spherical icosahedral triangular mesh, improves the efficiency of spherical convolution, and significantly reduces the computational complexity of the interpolation algorithm for projecting spherical mesh signals into two-dimensional pixel space.

[0147] The spherical image super-resolution method and system provided in this invention employs ERP projection (from two-dimensional pixel space to spherical grid space) based on bilinear interpolation of pixel coordinates and ERP inverse projection (from spherical grid space to two-dimensional pixel space) based on grid triangular facet grid point coordinate interpolation to realize the mutual conversion of spherical signals in two-dimensional pixel space and spherical grid space, effectively reducing the noise introduced by ERP projection and ERP inverse projection, and improving the quality of super-resolution images.

[0148] The spherical image super-resolution method and system provided in this invention employ an anisotropic spherical scattering spherical signal preprocessing method to extract multi-azimuth and multi-scale features of the spherical signal. These features are then aggregated using a channel attention mechanism and stitched with the original spherical image to obtain the preprocessed spherical signal features. This spherical signal preprocessing method expands the feature channels of the input spherical image while also possessing rotation invariance to angles, enhancing the overall expressive power of the model.

[0149] The spherical image super-resolution method and system provided in this invention employ a feature extractor unit based on a graph scattering attention mechanism and a dense convolutional network to extract multi-scale depth features of the spherical signal. Compared with traditional graph convolutional networks, the graph scattering attention mechanism can take into account both low-frequency and mid-to-high-frequency information in the input spherical grid image, avoiding the feature oversmoothing problem that may occur in multi-layer graph convolutional networks. At the same time, it adaptively aggregates features through the attention mechanism, enhancing expressive power while discarding redundant features.

[0150] The spherical image super-resolution method and system provided in this invention employ multiple densely connected feature extractor units as feature extractors, which can enhance the flow of features between different feature extractor units, increase the diversity of hidden layer features, and enhance the expressive power of the feature extractors.

[0151] The spherical image super-resolution method and system provided in this invention employ a method based on spherical icosahedral triangular mesh pixel recombination to aggregate spherical signal extended features. Compared to traditional spherical feature upsampling, the spherical feature extension operation based on spherical mesh pixel recombination is more convenient, has a clearer geometric meaning, and ensures better feature extension effect while improving feature extension efficiency.

[0152] The spherical image super-resolution method and system provided in this embodiment were tested on multiple ERP image datasets with different scenes and information distributions. Compared with existing methods (see embodiments), they all achieved good results, and achieved certain performance improvements in the ultra-high magnification ERP image super-resolution task. Attached Figure Description

[0153] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0154] Figure 1 This is a flowchart illustrating the process of a spherical image super-resolution method according to an embodiment of the present invention.

[0155] Figure 2 This is a schematic diagram illustrating the projection of an ERP image from a two-dimensional pixel space to a spherical grid space in a preferred embodiment of the present invention.

[0156] Figure 3 This is a schematic diagram of the operation of the feature preprocessing feature extractor (first feature extractor) in a preferred embodiment of the present invention.

[0157] Figure 4 This is a schematic diagram of the operation of the spherical signal depth feature extractor (second feature extractor) in a preferred embodiment of the present invention.

[0158] Figure 5 This is a schematic diagram illustrating the operation of spherical feature expansion and signal reconstruction in a preferred embodiment of the present invention.

[0159] Figure 6 This is a schematic diagram illustrating the projection of a spherical image from a spherical grid space to a two-dimensional pixel space in a preferred embodiment of the present invention.

[0160] Figure 7 This is a schematic diagram of the module composition of a spherical signal super-resolution system based on a spherical regular icosahedral triangular mesh in one embodiment of the present invention. Detailed Implementation

[0161] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0162] One embodiment of the present invention provides a spherical image super-resolution method, which can perform high-quality representation learning and detail characterization on low-resolution spherical image inputs, especially in ultra-high magnification image super-resolution tasks, generating super-resolution spherical images with more complete details and less distortion.

[0163] like Figure 1 As shown, the spherical image super-resolution method provided in this embodiment may include the following steps:

[0164] S100, construct a parameterized spherical regular icosahedral triangular mesh, project the equidistant rectangular projection image (ERP image) forward from the two-dimensional space onto the parameterized sphere to obtain the spherical mesh signal;

[0165] S200, the spherical grid signal is passed through the first feature extractor to obtain the first multi-scale feature;

[0166] S300, the first multi-scale feature is passed through the second feature extractor to obtain the second multi-scale feature;

[0167] S400, the second multi-scale feature is extended by adaptive filter learning based on Chebyshev polynomial and spherical pixel recombination to obtain spherical grid extended features and reconstruct the upsampled spherical grid signal;

[0168] S500 obtains a super-resolution spherical image by inversely projecting the upsampled spherical grid signal into a two-dimensional pixel space via ERP.

[0169] The above embodiments parameterize the icosahedral spherical mesh, greatly improving the processing efficiency of the spherical mesh signal. Simultaneously, by employing multi-scale representation learning to consider both local and global information of the spherical signal, accurate representation of the spherical signal is ensured, better characterizing signal details. This invention achieves superior results across multiple metrics in ultra-high magnification spherical image super-resolution tasks, demonstrating significant advantages for panoramic camera image processing.

[0170] The discontinuity of the unfolded icosahedral triangular mesh leads to difficulties in node encoding, resulting in low efficiency of spherical convolution. Therefore, in a preferred embodiment of the present invention, a preferred spherical parameterization method is provided to further project the ERP image from two-dimensional space onto the parameterized sphere to obtain a spherical mesh signal. Specifically, as shown... Figure 2 As shown, S100 can adopt the following steps:

[0171] S101, calculates the spherical coordinates of all grid points in the spherical regular icosahedral triangular mesh;

[0172] S102, based on the spherical coordinates of all grid points obtained, recursively generate a spherical regular icosahedral triangular mesh and grid point index, and update the neighbor index of the grid points;

[0173] S103, based on the spherical icosahedral triangular mesh and grid point index of S102, and update the neighbor index of the grid points, adopt the spherical icosahedral triangular mesh grid point search algorithm to establish the correspondence between the spherical mesh and the spherical coordinates;

[0174] S104, the ERP image is projected from the two-dimensional space onto the parameterized sphere to obtain the spherical coordinates of the ERP image. Based on the correspondence between the spherical grid and the spherical coordinates obtained in S103, the spherical grid signal of the ERP image is obtained.

[0175] In a preferred embodiment, S101, the spherical coordinates of all grid points in the spherical icosahedral triangular mesh are calculated, and the specific process is as follows:

[0176] Depending on the latitude, the i-order spherical grid divides the sphere into 3×2 i-1 +1 different layer.

[0177] Separately, the 0th to 2nd i-1 , 2nd i-1 ~2×2 i-1 and the 2nd × 2 i-1 ~3×2 i-1 Layers are defined as the upper, middle, and lower parts of a spherical mesh.

[0178] Let N(j) be the number of grid points in the j-th layer.

[0179]

[0180] In this spherical icosahedral grid, the grid points in each layer are evenly distributed along this latitude, and the latitudinal spacing of the upper, middle, and lower parts of the grid is consistent. Based on the properties of the icosahedron, it is easy to obtain the second... i-1 Layer and 2 i The latitudes of the layers are arctan(1 / 2) and -arctan(1 / 2), respectively.

[0181] The spherical coordinates of the k-th grid point in the j-th layer are obtained as follows:

[0182]

[0183] In a preferred embodiment, S102, based on the calculated spherical coordinates of all grid points, a spherical icosahedral triangular mesh and grid point index are recursively generated, and the neighbor index of the grid points is updated. The specific process is as follows:

[0184] S102.1 uses the calculated spherical coordinates of all grid points as a basis to recursively generate a spherical regular icosahedral triangular mesh.

[0185] When a high-order spherical mesh is generated by upsampling a low-order spherical mesh, the grid points and their indices of the low-order spherical mesh can be embedded in the high-order spherical mesh.

[0186] For an i-th order regular icosahedral spherical triangular mesh, the total mesh size is 10×4. i-1 +2 grid points. When generating an (i+1)th order triangular mesh from an i-th order triangular mesh, an additional 3×10×4 grid points are required. i-1 Each grid point in the i-th order triangular mesh is downsampled to the upper right, right side, and lower right, except for the two poles of the spherical mesh, generating a 3×10×4 grid. i-1 Each downsampling grid point.

[0187] S102.2 Recursively generate the lattice index of a spherical regular icosahedron.

[0188] For a first-order spherical icosahedral triangular mesh, the North and South pole grid points of the spherical mesh are indexed as 0 and 1, respectively. Divided by the 0-degree meridian, the indices of the five grid points in the Northern Hemisphere are 2-6 from west to east, and the indices of the five grid points in the Southern Hemisphere are 7-11 from west to east.

[0189] For a grid point with index idx in an i-th order triangular mesh (excluding the poles), the indices of the downsampled grid points in the upper right, right side, and lower right of the grid point generated from the i-th order triangular mesh are idx+10×4 respectively. i idx+2×10×4 i idx+3×10×4 i .

[0190] S102.3 Updates the indices of all adjacent points of a spherical grid of any order.

[0191] In this embodiment, the adjacent point index of each grid point is stored in the order of "top right, right side, bottom right, top left, left side, bottom left".

[0192] For the 12 vertices of a regular icosahedral mesh, since each vertex has only 5 adjacent points, the adjacent point indices need to be adjusted appropriately. For the vertices numbered 0 and 1, the spherical mesh uses the grid point's own index as the index of the 6th adjacent point; for grid points with indices 2-6, the upper left and upper right are the same grid point index; for grid points with indices 7-11, the lower right and lower left are the same grid point index.

[0193] Let n be the adjacent point index information of the grid point. c = (c1, c2, ..., c6), define the index generation function f. i (idx a idx b )=f(idx a idx b ,n a ,n b ,i), where idx a idx b These are the indices of grid points a and b, respectively, and n a ,n b Let p1 and p2 be the adjacent indices of grid points a and b, respectively, and i be the order of the grid. Define p1 and p2 as idx, respectively. a In n b The minimum value of the index and idx b In n a The minimum value of the subscript in the middle. If a is a pole, then p1 = p2, idx a =idx b If b is a pole, then p2 = p1, idx b =idx a .at this time,

[0194]

[0195] When generating the grid point index of an i-th order spherical triangular mesh based on an (i-1)-th order spherical triangular mesh and its grid point index, for a grid point c in the (i-1)-th order spherical triangular mesh and its adjacent grid point index information n... c = (c1, c2, ..., c6), where the adjacency index information of c in the i-th order spherical triangular mesh is:

[0196] n′ c =(f i (idx c ,c1),f i (idx c ,c2),…,f i (idx c ,c6))

[0197] The adjacent point indices of the grid point c in the (i-1)th order spherical triangular mesh are obtained by sampling to the upper right, right side, and lower right of the generated grid point, respectively, and are n'. up ,n' mid ,n' low If grid point c is derived from grid points indices 2-6 in a first-order spherical grid through successive downsampling, then

[0198] n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c4,c5),f i (idx c ,c5),idx c )

[0199] n′ mid =(f i (c1,c2),c2,f i (c2,c3),f i (idx c ,c1),idx c ,f i (idx c ,c3))

[0200] n′ low =(f i (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c6),f i (c3,c6))

[0201] If grid point c is derived from grid points indices 7-11 in a first-order spherical grid through successive downsampling, then

[0202] n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c1,c4),f i (idx c ,c4),idx c )

[0203] n′ mid =(f i (c1,c2),c2,f i (c2,c3),fi (idx c ,c1),idx c ,f i (idx c ,c3))

[0204] n′ low =(fi(idx) c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c5),f i (c5,c6))

[0205] In a preferred embodiment, in S103, based on the spherical icosahedral triangular mesh and grid point index from S102, and updating the neighbor index of the grid points, a spherical icosahedral triangular mesh grid point search algorithm is used to establish the correspondence between the spherical mesh and spherical coordinates (i.e., for an i-order spherical triangular mesh with N grid points, given any spherical coordinates, determine the triangular mesh surface containing those spherical coordinates). The specific process is as follows:

[0206] S103.1 Determine the approximate grid point v for any spherical coordinates (φ', θ').

[0207] Since all grid points in the higher-order spherical icosahedral triangular mesh, except for the poles, are derived from grid points with indices 2-11 in the first-order triangular mesh through recursive downsampling, the grid points with indices 2-11 are named "generated points". For any spherical coordinates (φ', θ'), the closest layer j is determined by the latitude of different layers of the triangular mesh, and one of the grid points close to (φ', θ') is determined by the distance between φ' and the grid points in the j-th layer, v = Cor(k v ,j v ).

[0208] S103.2 Based on the neighbor information n of the approximate grid point v v Perform a traversal to determine the area containing spherical coordinates (φ). ' The triangular mesh faces of ,θ').

[0209] Let the point where v is generated be denoted as v0 = Cor(k0,j0), then we have

[0210] |k v -k0|+|j v -j0|<2 i-1

[0211] Let the index of v0 be idx0, k v= kmod(N(j) / 5), based on the above relationship, we can calculate the index idx0 of the generated point v0 and the displacement shift of v0.

[0212]

[0213] For a grid point (0) located at the top of an i-th order grid... <j<2 i-1 Let's define the displacement vectors for the upper right, right side, and lower right respectively: (0,-1), (1,0), (1,1); for the second... i-1 For the grid points of the first layer, define the displacement vectors for the upper right, right side, and lower right respectively: (0,-1), (1,0), (0,1); for the grid points of the remaining layers (2... i-1 <j≤3×2 i-1 Let the displacement vectors for the upper right, right side, and lower right be defined as (1,-1), (1,0), and (0,1), respectively. Based on these defined displacements, the displacement shift of the approximate grid point v relative to its "generation point" v0 is used for inverse calculation. Let all displacements generated during the upsampling of the spherical grid points be denoted as choice.

[0214]

[0215] For a grid point v and a displacement shift, starting from unit = 1, find a suitable displacement choice[m] from the choices such that shift - choice[m] × unit is divisible by 2 × unit. Update shift = shift - choice[m] × unit, unit = 2 × unit, and record ai = m ∈ {0, 1, 2, 3}. Repeat this step until unit = 2. i-1 At this point, shift = (0,0). Based on the generated point index idx0 and the records a2, a3, ..., a i Cor(k) can be calculated v ,j v The index idx v

[0216]

[0217] In the above embodiments, a parameterizable spherical icosahedral triangular mesh is used to generate grid point indices and neighborhood information in a recursive manner. At the same time, a recursive search method is used to find the approximate grid point index and the spherical mesh triangular face where any spherical coordinates are located. This solves the problem of node encoding difficulties caused by the discontinuity of the unfolded graph of the spherical icosahedral triangular mesh, which can improve the efficiency of spherical convolution and significantly reduce the computational complexity of projecting the spherical mesh signal into the two-dimensional pixel space.

[0218] In a preferred embodiment, step S104 is performed to project the ERP image from a two-dimensional space onto the parameterized sphere to obtain the spherical coordinates of the ERP image. Based on the correspondence between the spherical grid and the spherical coordinates obtained in step S103, the spherical grid signal of the ERP image is obtained. The specific process is as follows:

[0219] Given any spherical coordinates (φ, θ), the following transformation relationship exists between them and two-dimensional coordinates (h, w):

[0220]

[0221] Where H and W are the height and width of the two-dimensional ERP image, respectively.

[0222] Then, pixel value for each grid point on the spherical surface is determined during projection using bilinear interpolation in pixel space.

[0223] For a spherical grid point with coordinates (φ0, θ0) and its corresponding two-dimensional coordinates (h0, w0), let the coordinates of the four closest two-dimensional image pixel centers to (h0, w0) be (h1, w1), (h2, w2), (h3, w3), and (h4, w4), located at the upper left, upper right, lower left, and lower right of (h0, w0) respectively, and their corresponding pixel values ​​be val1, val2, val3, and val4. Then the pixel value of this spherical grid point is...

[0224]

[0225] Where S1 = (h4-h0)×(w4-w0), S2 = (h3-h0)×(w0-w3), S3 = (h0-h2)×(w2-w0), and S4 = (h0-h1)×(w0-w1). The spherical grid pixel value val will be used for subsequent feature extraction and spherical image reconstruction of the spherical image signal.

[0226] By implementing step S103 in the above embodiment, the spherical coordinates (φ, θ) corresponding to the two-dimensional coordinates (h, w) and the triangular face of the spherical mesh they occupy can be determined. This embodiment, by projecting based on the spherical mesh signal, effectively reduces the computational complexity of the interpolation algorithm.

[0227] After the 2D ERP image is projected from the 2D space onto the parameterized sphere to obtain the spherical mesh signal, further feature extraction is required. In a preferred embodiment of the present invention, the first feature extractor includes a scattering network based on anisotropic spherical wavelet and a channel attention mechanism. S200 can employ the following steps, such as... Figure 3 As shown:

[0228] S21, input the spherical image X = [X (R) ,X (G) ,X(B) Extract the multi-azimuth and multi-scale information of the image through the anisotropic spherical scattering network. Let f denote the signal of a certain channel of the spherical image, and use the anisotropic spherical wavelet to calculate the wavelet coefficients of the image at different azimuth angles.

[0229]

[0230] where is the spherical direction convolution, is the wavelet function on the sphere, ρ = (α, β, γ) ∈ SO(3) rotation group, and the wavelet scale represents the angular locality of Ψ j The scale coefficient W Φ (ω) of this transformation can be calculated as

[0231]

[0232] where ⊙ is the convolution symmetric about the axis on the sphere, and the rotation operator is written as then For the input signal The scattering propagation of the first step of the scattering convolution network based on the anisotropic spherical wavelet is represented by Using the modulus of the wavelet coefficient as the signal on the rotation group L 2 (SO(3)), it can be linearly combined by The scattering propagation U[j]f(α, β, γ g ) at the azimuth angle (α, β) is abbreviated as U[j]f(γ g ), for the propagation path p = ((j1, g1), …, (j m , g m )) (j i > j i+1 , i = {1, …, m}) with decreasing frequency of length m, the corresponding scattering propagation is

[0233]

[0234] The scattering coefficient output by the path p is calculated as

[0235]

[0236] where 0 ≤ J0 < J, J is the maximum wavelet scale, ​​​​​​sct =Concat({s[p]f|p∈∪Ω) i ,0≤i≤M-1})

[0238] Where M is the number of layers in the anisotropic spherical scattering network, Ω i Let X be the set of all frequency-decreasing propagation paths of length i. For an RGB spherical signal input X = [X...] (R) ,X (G) ,X (B) Anisotropic scattering characteristics were obtained respectively. in

[0239] Similarly.

[0240] S22 employs a channel attention mechanism to extract features from the anisotropic spherical scattering network output. Each feature is compressed into a single channel. For example, channel coefficients are obtained by performing max pooling and mean pooling at the node level. a max ,a mean The channel attention weight 'a' is obtained by using a two-layer shared-parameter perceptron and then summing the results through a non-linear activation layer. channel :

[0241] a channel =σ(MLP(a) max )+MLP(a mean ))

[0242] Based on channel attention weight a channel scattering characteristics Compression to single-channel features

[0243]

[0244] S23, by concatenating the scattering features aggregated in S22 with the signals of the corresponding channels of the original image, the spherical signal preprocessing feature X0 is obtained.

[0245]

[0246] The above embodiments employ an anisotropic spherical scattering spherical signal preprocessing method to extract multi-azimuth and multi-scale features of the spherical signal. These features are then aggregated using a channel attention mechanism and stitched together with the original spherical image to form the preprocessed spherical signal features. This spherical signal preprocessing method expands the feature channels of the input spherical image while also possessing rotational invariance to angles, enhancing the overall expressive power of the model.

[0247] Furthermore, in a preferred embodiment of the present invention, the second feature extractor is constructed by densely connecting feature extraction units consisting of K graph scattering attention mechanisms and dense convolutional networks, where K is a positive integer. Wherein:

[0248] The input to the first feature extractor unit is the fused feature of anisotropic spherical scattering features and the original spherical signal. (When the input spherical image is an RGB three-channel image, d) in =6), output spherical mesh features in, For the real number field;

[0249] The input of the k-th feature extraction unit is the output of the (k-1)-th feature extraction unit. By using cross-layer connections, the output features of the dense convolutional network of the k-th feature extraction unit are combined with the input features. By concatenating the features, the output features of the k-th feature extraction unit are obtained. Where k = 2, ..., K.

[0250] Furthermore, in a preferred embodiment of S300, the preprocessed features of the spherical mesh are input into a feature extractor based on a graph scattering attention mechanism and a dense convolutional network, such as... Figure 4 As shown, it includes:

[0251] S31, for the k-th feature extractor unit, the input feature X is... k-1 Compressed features are obtained by compressing them to a specified dimension using a one-dimensional convolutional layer.

[0252] S32, the obtained compressed features are processed through a graph scattering attention mechanism (GSAN), and the features output from different scattering paths in the scattering network are aggregated through weight learning to obtain an adaptive multi-scale feature representation H that includes compressed low-pass and band-pass features. gsan ;

[0253] S33, Calculate the adaptive multi-scale feature representation H of the input features. gsan The feature difference between it and all grid points within its l-hop is concatenated with its own feature;

[0254] S34, based on a three-layer dense graph convolutional network, learns the concatenated features of S33 and outputs the deep features of this feature extractor unit.

[0255] S35, the output features X of the first k feature extractor units concatenated k =Concat(X) 1,out ,…,X k,out)As the input of the (k + 1)-th feature extractor unit (1 ≤ k < K) or as the final output of the feature extractor (k = K).

[0256] In a preferred embodiment of S32, the compressed spherical grid signal X cmp passes through an M-layer graph scattering network based on diffusion wavelets and a series of graph convolutional networks. An attention mechanism is used to aggregate the output features of different scattering paths in the scattering network and the low-pass features obtained in the graph convolutional network, to obtain an adaptive multi-scale feature representation H of the spherical grid signal gsan , including:

[0257] S321, based on the lazy random walk matrix P = 1 / 2(I + AD -1 ), the feature extraction part of the graph scattering attention mechanism includes the band-pass features based on the scattering network and the low-pass features based on the graph convolutional network. The graph scattering network uses diffusion wavelets to extract the multi-scale features of the input signal

[0258] Ψ0 = I - P, For the scattering path and the input signal X cmp , the scattering transfer feature is

[0259]

[0260] The scattering transfer features output by different paths are respectively taken modulo to obtain the graph scattering features

[0261] S322, the graph convolution part uses a multi-order normalized looped adjacency matrix to extract the low-pass features X of different scale receptive fields gcn,i = W i X cmp ;

[0262] S323, denote the number of feature channels output by the graph scattering network and the number of feature channels output by the graph convolutional network as C gst and C gcn , based on the output features X sct,i (1 ≤ i ≤ C sct ) and X gcn,i (1 ≤ i ≤ C gcn ), an attention mechanism is used to fuse the features. Denote the shared attention parameter as a, then the pointwise attention weights of different feature channels are

[0263] e gcn,i = [X cmp ] ||X gcn,i a, e sct,i = [Xcmp ||X sct,i ]a

[0264] Where || represents feature concatenation. Based on e gcn,i and e sct,i Calculate the normalized attention weight α gcn,i ,α sct,i

[0265]

[0266] S324 aggregates multi-channel features through attention weights to obtain an adaptive multi-scale feature representation H. gsan

[0267]

[0268] Where ⊙ is the Hadamard product of vectors, σ(·)=LeakyRelu(·);

[0269] In a preferred embodiment of S33 and S34, the spherical grid signal output by the attention mechanism of the graph scattering network is adaptively represented as a multi-scale feature representation H. gsan A three-layer dense graph convolutional network aggregates all grid features within the l-hop, outputting the deep feature of this feature extractor unit. include:

[0270] S331 uses a residual splicing method to aggregate point by point. l-jump inner neighborhood features

[0271]

[0272] splice all get This represents the number of grid points within the distance grid point vl-hop. Features are concatenated point-by-point to obtain the output feature vector of the dense convolutional network.

[0273] S341, H dense,in Hidden features are learned through a three-layer dense convolutional network. H... dense,in Through 2D convolution and nonlinear activation layers, with H dense,in Concatenation as the first hidden input

[0274]

[0275] H dense,hd1 Through a single 2D convolutional layer and a non-linear activation layer, with H dense,hd1 The data is concatenated and used as input for the second hidden layer.

[0276]

[0277] H dense,hd2 Through a single 2D convolutional layer and a non-linear activation layer, with H dense,hd2 The features are concatenated and used as the output features of the dense convolutional network.

[0278]

[0279] S342, uses max pooling to obtain the output of the dense convolutional network.

[0280]

[0281] S343, through cross-layer splicing, X k,out The output feature X of the (k-1)th feature extractor unit k-1 The concatenation is used as the output X of the k-th feature extractor unit. k =Concat(X) k-1 ,X k,out =Concat(X) 1,out ,…,X k,out ).of

[0282] In the above embodiment, a feature extractor unit based on graph scattering attention mechanism and dense convolutional network is used to extract multi-scale depth features of spherical signals. Compared with traditional graph convolutional networks, graph scattering attention mechanism can take into account both low-frequency and mid-to-high-frequency information in the input spherical grid image, avoid the feature oversmoothing problem that may occur in multi-layer graph convolutional networks, and adaptively aggregate features through attention mechanism, enhancing expressive power while discarding redundant features.

[0283] In a preferred embodiment of the present invention, in step S400, an adaptive filter based on Chebyshev polynomials is used to train extended spherical features. These extended features are then integrated onto a higher-resolution spherical grid through pixel recombination within the spherical grid. The super-resolution spherical signal is reconstructed from the extended features of the spherical grid. This can be achieved through the following steps: Figure 5 As shown:

[0284] S41, using the output feature X of the feature extractor K As input to the feature expansion part, an adaptive filter based on Chebyshev multinomials is used to learn the feature extractor output feature X. K Perform 4x feature expansion:

[0285]

[0286] Here, T i (λ) is an i-th order Chebyshev polynomial, H is the highest order constraint of the Chebyshev polynomial, and θi and Ξ i All parameters are learnable. Four different adaptive filters F are trained. j (λ)(j=1,2,3,4) for X K Filtering, through cross-layer connections, yields extended spherical signal characteristics.

[0287]

[0288] Where P = 1 / 2(I + AD) -1 Let Θ be the lazy random walk matrix of the spherical grid. i These are learnable parameters;

[0289] S42, Spherical mesh features generated based on different filters The extended features are reorganized using a spherical icosahedral mesh pixel recombination, integrating the low-order spherical mesh extended features onto a higher-resolution spherical triangular mesh. For the (i+1) order spherical mesh, the grid points are divided into four groups according to the spherical mesh grid point index, and... Fill in the information; the correspondence between the index and the grid feature is as follows:

[0290]

[0291] in express The index is 2 to 10×4. i +1 grid feature;

[0292] S43, X based on high-order spherical mesh feature representation out (Output of the spherical icosahedral pixel reconstruction layer) Upsampled spherical signal pixel reconstruction is achieved through two fully connected network layers.

[0293] Z = FC(σ(FC(X) out ))).

[0294] The above embodiments employ a method based on spherical icosahedral triangular mesh pixel recombination to aggregate spherical signal extended features. Compared to traditional spherical feature upsampling, spherical feature extension based on spherical mesh pixel recombination is more convenient, has a clearer geometric meaning, and ensures better feature extension effect while improving feature extension efficiency.

[0295] In a preferred embodiment of the present invention, in S500, a super-resolution spherical mesh image is projected onto a two-dimensional pixel space using triangular geometric interpolation, and a super-resolution ERP image Z is output. out The following steps can be taken, such as Figure 6 As shown:

[0296] S51, For any two-dimensional pixel coordinate (h, w), calculate the corresponding spherical coordinates (φ, θ) using the ERP projection relationship.

[0297]

[0298] S52, determine the spherical coordinates of an approximate grid point v with spherical coordinates (φ, θ) using the grid point coordinates of the spherical grid. Based on a recursive spherical grid point search algorithm, determine the index of the approximate grid point v.

[0299] Determine one of the grid points closest to (φ,θ) using the coordinates of the grid points in a spherical icosahedral triangular mesh: v = Cor(k v ,j v In a first-order spherical mesh, the grid points indexed 2-11 are called "generating points." For any grid point in an arbitrary higher-order spherical mesh, excluding the North and South Poles, this grid point is derived from a "generating point" through multiple recursive upsampling operations. Let the generating point of v be denoted as v0 = Cor(k0,j0), then...

[0300] |k v -k0|+|j v -j0|<2 i-1

[0301] Let the index of v0 be idx0, k v = kmod(N(j) / 5), based on the above relationship, we can calculate the index idx0 of the generated point v0 and the displacement shift of v0.

[0302]

[0303] For a grid point (0) located at the top of an i-th order grid... <j<2 i-1 Let's define the displacement vectors for the upper right, right side, and lower right respectively: (0,-1), (1,0), (1,1); for the second... i-1 For the grid points of the first layer, define the displacement vectors for the upper right, right side, and lower right respectively: (0,-1), (1,0), (0,1); for the grid points of the remaining layers (2... i-1 <j≤3×2 i-1 Let the displacement vectors for the upper right, right side, and lower right be defined as (1,-1), (1,0), and (0,1), respectively. Based on these defined displacements, the displacement shift of the approximate grid point v relative to its "generation point" v0 is used for inverse calculation. Let all displacements generated during the upsampling of the spherical grid points be denoted as choice.

[0304]

[0305] For a grid point v and a displacement shift, starting from unit = 1, find a suitable displacement choice[m] from the choices such that shift - choice[m] × unit is divisible by 2 × unit. Update shift = shift - choice[m] × unit, unit = 2 × unit, and record ai = m ∈ {0, 1, 2, 3}. Repeat this step until unit = 2. i-1 At this point, shift = (0,0). Based on the generated point index idx0 and the records a2, a3, ..., a i Cor(k) can be calculated v ,j v The index idx v

[0306]

[0307] S53, based on the index of approximate grid point v and its adjacent grid point information n v Determine the triangular face containing (φ,θ), and the indices and spherical coordinates of the three vertices of that triangular face.

[0308] Based on the index of the approximate grid point (φ',θ') of (φ,θ), the six neighbors of (φ',θ') are determined through the neighbor node index information of the approximate grid point (φ',θ'), denoted as v1, v2, ..., v6. The triangular faces Δv1vv2, Δv2vv3, Δv3vv6, Δv5vv6, Δv4vv5, and Δv1vv4 associated with the approximate grid point v are traversed to determine the triangular mesh face containing the spherical coordinates (φ,θ).

[0309] S54, based on the spherical coordinates (φ,θ) and the spherical coordinates (φ1,θ1), (φ2,θ2), (φ3,θ3) of the three vertices of the triangle face, uses triangular face geometric interpolation to obtain the pixel value at (h,w) in the two-dimensional pixel space.

[0310] Let the three vertices of the triangle containing the spherical coordinates (φ, θ) be denoted as points A, B, and C, respectively, with corresponding spherical coordinates of (φ1, θ1), (φ2, θ2), and (φ3, θ3), and pixel values ​​of val. A ,val B ,val C Let (φ,θ) be denoted as point O, then the pixel value at (h,w) in the two-dimensional pixel space is...

[0311]

[0312] The area of ​​the triangle can be calculated using the cross product of vectors in Cartesian coordinates. Using the above triangular interpolation method, the upsampled spherical grid signal Z is projected onto a two-dimensional pixel plane to obtain the super-resolution ERP image Z. out .

[0313] To achieve better super-resolution, in a preferred embodiment, the spherical signal super-resolution method obtains learnable parameters based on end-to-end training using training data, including:

[0314] The spherical signal super-resolution method uses the output super-resolution signal I pred With truth value I gt The mean squared error (MSE) is used to measure the difference between the output super-resolution image and the ground truth.

[0315]

[0316] Where C is the number of channels in the ERP image, Mean((I pred (c)-I gt (c)) 2 ) is the output super-resolution signal I pred With truth value I gt The mean squared error of pixels in channel c. The model training uses a mean squared error mixture regularization term as the loss function.

[0317]

[0318] Where β is a hyperparameter, and Θ is a learnable parameter in the spherical signal super-resolution deep network. The gradient of parameter Θ is calculated based on the loss function Loss, and the parameter Θ in the spherical image super-resolution model is updated through backpropagation. The parameters of the spherical image super-resolution model are obtained by iteratively optimizing and minimizing the aforementioned loss function. Specifically, the spherical signal super-resolution deep model can be trained using a mean square error defined on a spherical grid or defined in a two-dimensional pixel space. When training using a mean square error defined in a two-dimensional pixel space, I... pred =Z out ,I gt =Z gt For the true value of the high-resolution ERP image, at this time

[0319]

[0320] Among them I pred (h,w,c) and I gt(h,w,c) represent the pixel values ​​of the output super-resolution ERP image and the ground truth at channel c and pixel (h,w), respectively, where H and W are the height and width of the super-resolution ERP image, respectively. When training is performed using a mean squared error defined on a spherical triangular mesh, I... pred =Z,I gt =Proj(Z) gt The high-resolution ground truth image is projected onto a high-order spherical triangular mesh based on the bilinear geometric interpolation method.

[0321]

[0322] Among them I pred (v,c) and I gt (v,c) represent the pixel values ​​of the output upsampled spherical grid signal and the true value at channel c and grid point v, respectively, and N represents the number of grid points of the spherical triangular grid.

[0323] To better understand, the technical solution provided by the above embodiments of the present invention will be further described in detail below with reference to a specific application example.

[0324] In this specific application example, the ERP image super-resolution method based on spherical triangular mesh, spherical scattering, and spherical pixel reconstruction mainly includes the following six steps:

[0325] Step 1: Generate a spherical icosahedral triangular mesh with a grid number matching the pixel count of the ERP image. Using the correspondence between 2D pixel coordinates and spherical coordinates in the ERP projection, project the grid point coordinates (φ, θ) of the spherical mesh onto the 2D pixel space (h, w). Find the four pixel center coordinates (h1, w1), (h2, w2), (h3, w3), and (h4, w4) closest to (h, w) and their corresponding pixel values ​​val1, val2, val3, and val4. Use bilinear interpolation to obtain the pixel value val at the grid point (φ, θ) of the spherical mesh, thus obtaining the spherical image X.

[0326] Step 2: Graph the sphere X = [X] (R) ,X (G) ,X (B) Using an anisotropic spherical scattering network, multi-azimuth and multi-scale information of each channel is learned. The scattering features of each channel are aggregated using a channel attention mechanism, and the scattering features of each channel are compressed to one dimension to obtain the aggregated scattering features. Connect X with By stitching together the images, we obtain the preprocessed features of the spherical image.

[0327] Step 3: The spherical preprocessed feature X0 is processed through a series of densely connected feature extractor units based on graph scattering attention mechanism and dense convolutional network to extract multi-scale hierarchical features of the spherical grid signal. The input of the first feature extractor unit is the fused feature of anisotropic spherical scattering features and the original spherical signal. Output spherical mesh features in, The domain is the real number field. The input of the k-th feature extraction unit is the output of the (k-1)-th feature extraction unit. The dense convolutional network outputs the features of the k-th feature extraction unit. The output features of the (k-1)th feature extractor unit By concatenating the features, the output features of the k-th feature extractor unit are obtained. d k =d in +k(d m +3d hd ), (k=1,…,K).

[0328] Step 4: Extract the depth features X from the feature extractor. K X is processed through four adaptive filters based on Chebyshev polynomials. K Expanded to N = 10 × 4 i-1 +2, where i is the number of grid points in the lower-order spherical mesh. The grid points of the (i+1)th order spherical mesh are divided into 4 groups using index grouping, and then... Fill in the data to form the depth feature X defined on the high-order spherical grid. out Upsampled spherical image reconstruction Z = FC(σ(FC(X)) is achieved through a two-layer fully connected network. out ))).

[0329] Step 5: Based on the coordinate correspondence of the ERP projection, project the two-dimensional pixel coordinates (h,w) of the super-resolution ERP image onto a sphere to obtain spherical coordinates (φ,θ). Find the approximate grid coordinates (φ',θ') of (φ,θ) using the spherical grid coordinates. Obtain the index and adjacent point index information of (φ',θ') using a recursive spherical grid fast search algorithm, and then determine the triangular face containing (φ,θ) and the spherical coordinates of its vertices (φ1,θ1), (φ2,θ2), and (φ3,θ3). Obtain the pixel value at (φ,θ), i.e., the pixel value of (g,w), using a triangular face interpolation algorithm. Repeat this operation to traverse the center coordinates of each pixel in the two-dimensional pixel space to obtain the output super-resolution ERP image Z. out .

[0330] Step Six: When training using the mean squared error defined in two-dimensional pixel space, calculate the output super-resolution ERP image Z. out Compared with the true value Z of high-resolution ERP images gt The differences are calculated, and the regularization term of the trainable parameters in the deep network is also calculated, along with the loss function.

[0331]

[0332] When training is performed using the mean square error defined in the pixel space of a spherical grid, step five needs to be replaced with:

[0333] In step five: the true value Z of the high-resolution ERP image is projected using the same method as in step one. gt Projecting onto a higher-order spherical mesh yields the ground truth value Proj(Z) of the high-resolution spherical image. gt ).

[0334] Computation of the reconstructed super-resolution spherical mesh image Z and the ground truth Proj(Z) gt The differences are calculated, and the regularization term of the trainable parameters in the deep network is also calculated, along with the loss function.

[0335]

[0336] Repeat steps two through six until the network converges. Update the learned parameters Θ for the ERP signal super-resolution task.

[0337] In this specific application case, the comparative evaluation methods for implementation performance are Bicubic, SRCNN, VDSR, LapSRN, MemNet, MSRN, EDSR, D-DBPN, RCAN, EBRN, 360-SS, LAU-Net, and SphereSR. The data for all comparison models are derived from the experimental sections of the papers "LAU-Net: Latitude Adaptive Upscaling Network for Omnidirectional Image Super-resolution" and "SphereSR: 360° Image Super-Resolution with Arbitrary Projection via Continuous Spherical Image Representation". This embodiment of the invention uses the same training set from the ODI-SR dataset as those in the two papers for model training, and tests are performed using the test set from the ODI-SR dataset and the SUN dataset, respectively. The training dataset consists of 1200 samples, with the ground truth being a high-resolution ERP image of 2048×1024. Inputs of 128×64 (×16x), 256×128 (×8x), 512×256 (×4x), and 1024×512 (×2x) are provided based on different super-resolution magnifications. The ODI-SR test set and the SUN dataset each contain 100 samples, with the ground truth also being a high-resolution ERP image of 2048×1024. Inputs of the above different resolutions are provided based on different super-resolution magnifications. Tables 1 and 2 provide the super-resolution results for spherical images at ×8x with 256×128 input and ×16x with 128×64 input, respectively. WS-PSNR is used to measure the difference between the output super-resolution image and the high-resolution ground truth, while WS-SSIR and WS-SSIM are used to measure the difference in overall structure between the output super-resolution image and the high-resolution ground truth.

[0338] Table 1

[0339]

[0340] Table 2

[0341]

[0342]

[0343] Compared to models like SRCNN, 360-SS, and VDSR that process ERP images in two-dimensional pixel space, the method in this embodiment of the invention processes spherical image signals in spherical space. This more clearly defines the neighborhood range of pixels in the spherical image and completely eliminates the influence of distortion during representation learning. Compared to LAU-Net, the method in this embodiment of the invention is based on a spherical triangular mesh, allocating different numbers of grid points at different latitudes, thus avoiding the problem of discontinuous edges when reconstructing and stitching images from different latitudes. Compared to SphereNet, which is also based on a spherical triangular mesh, the method in this embodiment of the invention optimizes the indexing and grid feature storage method of the spherical icosahedral triangular mesh, extending pixel reconstruction from two-dimensional pixel space to a spherical mesh, and employing a simpler convolution kernel, thereby improving the efficiency of spherical mesh signal processing. Meanwhile, by introducing an anisotropic spherical scattering network to extract multi-azimuth and multi-scale features of the spherical grid signal, and further extracting the depth features of the spherical grid signal through graph scattering attention mechanism and dense convolutional network, the method of this embodiment can take into account both low-pass and band-pass features of the spherical grid signal in the representation learning stage, ensuring the diversity of its features, enhancing the expressive power of the model, and having a greater advantage in ultra-high magnification spherical image super-resolution tasks.

[0344] Image super-resolution (ERP) is widely used in virtual modeling, autonomous driving, and other fields. However, due to limitations in hardware and communication costs, industries often cannot directly obtain or process high-resolution images, requiring signal reconstruction to convert low-resolution images into high-resolution ones. The spherical signal super-resolution method based on a spherical icosahedral triangular mesh provided in the above embodiments of this invention can generate relatively smooth and detailed high-resolution spherical signals from low-resolution spherical images, which is of great significance for subsequent mesh modeling and 3D rendering. Therefore, the spherical image super-resolution method based on a spherical triangular mesh provided in the above embodiments of this invention has significant implications for industrial spherical image processing.

[0345] One embodiment of the present invention provides a spherical image super-resolution system based on a spherical regular icosahedral triangular mesh.

[0346] like Figure 7 As shown, the spherical image super-resolution system based on a spherical regular icosahedral triangular mesh provided in this embodiment may include:

[0347] ERP Projection Module: Projects a 2D ERP image onto an i-th order spherical icosahedral triangular mesh using geometric interpolation;

[0348] Feature pre-extraction module: An anisotropic spherical scattering network is used to learn multi-directional and multi-scale features of spherical signals. The spherical scattering features are aggregated through a channel attention mechanism, and the learned spherical signal representation is fused with the original spherical signal.

[0349] Deep feature learning module: Employs multiple graph scattering attention mechanisms (GSAN) and dense convolutional networks to extract multi-scale deep features from the input spherical signal;

[0350] Spherical feature extension module: The module uses an adaptive filter based on Chebyshev polynomials to train and extend the multi-scale depth features of the output spherical signal, and obtains the upsampled spherical signal depth features through pixel recombination of spherical triangular mesh;

[0351] ERP Image Reconstruction Module: Reconstructs the super-resolution spherical signal through a two-layer fully connected network, and projects the spherical signal onto the plane through ERP inverse projection, thus completing the super-resolution ERP image reconstruction.

[0352] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0353] One embodiment of the present invention provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform any of the methods in the above embodiments of the present invention, or to run any of the systems in the above embodiments of the present invention.

[0354] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0355] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0356] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0357] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0358] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0359] The spherical image super-resolution method and system based on an icosahedral spherical triangular mesh provided in the above embodiments of the present invention first projects a two-dimensional ERP image onto an i-order spherical icosahedral triangular mesh through geometric interpolation to obtain a spherical mesh image. Then, an anisotropic spherical scattering network is used to learn multi-directional and multi-scale features of the spherical signal, and the scattering features are aggregated through a channel attention mechanism. The learned spherical signal representation is fused with the original spherical signal to obtain spherical image preprocessing features. Next, multiple graph scattering attention mechanisms and dense convolutional networks are used to extract multi-scale depth features of the input spherical signal. Then, an adaptive filter is used to train and expand the multi-scale depth features of the output spherical signal, and the upsampled spherical signal depth features are obtained through pixel recombination of the spherical triangular mesh. Finally, the reconstructed spherical mesh image is projected onto a two-dimensional pixel space through geometric interpolation to obtain a super-resolution ERP image. Compared with existing methods, this invention solves the problem of node encoding difficulties caused by the discontinuity of the unfolded graph of spherical icosahedral triangular meshes, improves the efficiency of spherical convolution, significantly reduces the computational complexity of interpolation algorithms that project spherical mesh signals into two-dimensional pixel space, takes into account both local and global features of spherical mesh signals during representation learning, and ensures accurate representation of spherical images. By extending pixel recombination to spherical triangular meshes, the geometric meaning of feature upsampling on the sphere becomes more distinct, giving it a greater advantage in ultra-high magnification spherical image super-resolution tasks and practical application value in industrial applications such as virtual modeling and autonomous driving.

[0360] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0361] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0362] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for super-resolution of a spherical image, the method comprising: The method comprises the following steps: Construct a parameterized spherical icosahedron triangular mesh, and project an equidistant rectangular projection image (ERP) forward to obtain a spherical mesh signal by projecting from a two-dimensional space to the icosahedron triangular mesh; The spherical mesh signal is input into a first feature extractor to obtain a first multi-scale feature; The first multi-scale feature is input into a second feature extractor to obtain a second multi-scale feature; The second multi-scale feature is subjected to adaptive filter learning based on a Chebyshev polynomial and spherical pixel recombination to realize feature expansion, so as to obtain a spherical mesh expansion feature and reconstruct an up-sampled spherical mesh signal; The up-sampled spherical mesh signal is subjected to ERP inverse projection to obtain a super-resolution spherical image; The first feature extractor comprises an anisotropic spherical scattering network and a channel attention mechanism; The spherical mesh signal is input into the first feature extractor to obtain the first multi-scale feature, and the method comprises the following steps: Multi-scale spherical wavelets in the anisotropic spherical scattering network are used to extract multi-azimuth multi-scale features of the spherical signal based on different azimuth angles and rotation angles, and the method comprises the following steps: The basis functions of the directional spherical wavelets are simultaneously dependent on longitude and latitude, the wavelet coefficients are calculated as ； wherein is an input spherical signal, is a spherical directional convolution, is a wavelet function on the sphere, is a rotation group, is a rotation group, ω = (θ, φ) ∈ S 2 is a spherical angular position; Anisotropic spherical wavelets have localization in scale, position and direction. The wavelet scale represents the angular localization of the function and the wavelet by expanding them into the spherical harmonic space to obtain their respective spherical harmonic coefficients and , the transform is written as ； is the Wigner D equation; Based on the above transformation, the scale coefficients are computed as ； where is the axisymmetric convolution on the sphere, the spin operator is written as then ; the scaling function is an axisymmetric function, ; The scale coefficient is calculated in the spherical harmonic space as ； The scattering network constructed based on the anisotropic spherical wavelets has rotation invariance: For the input spherical mesh signal , the first step of scattering propagation using an anisotropic spherical wavelet based scattering convolution network is denoted by ; For an azimuthal band limit of the wavelet coefficients at an arbitrary azimuth are written as a linear combination of basic wavelet coefficients ； where the rotation angle , is an interpolation function; Using the modulus of the wavelet coefficients as the signal on the rotation group is linearly combined​ The azimuthal angle under the scattering propagation is simply denoted by , for a propagation path of length under the frequency drop , the corresponding scattering propagation is ； Path The output scattering coefficient is calculated as ； wherein , is the largest wavelet scale, is an axisymmetric scaling function; With convolution guarantees the rotational invariance within the range determined by Scattering transform iteratively decomposes the signal into individual directions and scales, collecting scattering path coefficients of different lengths to get the scattering features of the signal on a spherical grid. ； wherein is the number of layers of an anisotropic spherically scattering network, is a set of all length frequency-decreasing propagation paths; Input of RGB three-color spherical signal Anisotropic scattering characteristics on N-point spherical grid are obtained respectively wherein ； ； ； The multi-azimuth multi-scale information, i.e., the scattering feature, output by the anisotropic spherical scattering network is aggregated by the channel attention mechanism, and the method comprises the following steps: For , x = R, G, B, max-pooling and average-pooling are respectively performed on the node dimension to obtain channel coefficients , Through a double-layer shared parameter perception machine, the channel attention weight is finally obtained through summation and a nonlinear activation layer : ； Based on channel attention weights scatter features compressed to single channel features , ； compressing the scattering features fusing with the original signal to obtain a first multi-scale feature as an input of a second feature extractor.

2. The method of claim 1, wherein, The method for constructing the parameterized spherical icosahedron triangular mesh comprises the following steps: The spherical coordinates of all grid points in the spherical icosahedron triangular mesh are calculated; Based on the calculated spherical coordinates of all grid points, the spherical icosahedron triangular mesh and the grid point index are recursively generated, and the neighbor index of the grid point is updated; The correspondence between the spherical mesh and the spherical coordinates is established by using a spherical icosahedron triangular mesh grid point searching method.

3. The method of claim 2, wherein, The method for calculating the spherical coordinates of all grid points in the spherical icosahedron triangular mesh comprises the following steps: According to the latitude, the spherical surface is divided into different layers, and the first , second , and third , and fourth layers are defined as the upper, middle, and lower parts of the spherical grid. The number of lattice points of the first layer is denoted as The number of lattice points of the first layer is denoted as , ； The grid points in each layer of the spherical icosahedron grid are uniformly distributed on the latitude, and the latitude intervals of the upper, middle and lower parts of the grid are consistent, respectively. According to the property of the icosahedron, the latitudes of the first layer and the second layer are and , respectively. Based on the latitude information, the first layer of the grid point spherical coordinates is 。 4. The method of claim 2, wherein, Based on the calculated spherical coordinates of all grid points, the spherical icosahedron triangular mesh and the grid point index are recursively generated, and the neighbor index of the grid point is updated, and the method comprises the following steps: The spherical icosahedron triangular mesh is recursively generated, and the method comprises the following steps: When a high-order spherical mesh is generated by up-sampling a low-order spherical mesh, the grid points and the index of the low-order spherical mesh can be embedded in the high-order spherical mesh; For a regular icosahedral spherical triangular mesh of order 4, the mesh has a total of grid points; From order triangular mesh generation When generating the order triangular mesh, new grid points need to be added, in addition to the two polar points of the spherical mesh, Each grid point of the order triangular mesh is down-sampled to the right upper, right and right lower, generating down-sampled grid points; The grid point index of the recursively generated spherical icosahedron triangular mesh is generated, and the method comprises the following steps: For the first-order spherical icosahedron triangular mesh, the north pole and the south pole grid points of the spherical mesh are indexed as 0 and 1, respectively, and the 0-degree meridian is used as a division, and the indices of the five grid points in the northern hemisphere from west to east are 2-6, and the indices of the five grid points in the southern hemisphere from west to east are 7-11; For a grid point in a triangular mesh of order except the pole, the indices of the down-sampled grid points to the right upper, right side and right lower of the grid point are respectively. In the down-sampling of a grid point in a triangular mesh of order the indices of the down-sampled grid points to the right upper, right side and right lower of the grid point are The neighbor index of the grid point is updated, and the method comprises the following steps: The neighbor point index of each grid point of the icosahedron spherical triangular mesh is saved in the order of "upper right, right, lower right, upper left, left, lower left"; For the 12 vertices of the icosahedron mesh, for the poles with indices of 0 and 1, the grid point index itself is used as the sixth neighbor point index; for the grid points with indices of 2-6, the upper left and the upper right are the same grid point index; for the grid points with indices of 7-11, the lower right and the lower left are the same grid point index; For a triangular mesh of order k, let the index information of the neighboring points of a grid point be denoted as , define the index generating function , where are the indices of the grid point , respectively, are the indices of the neighboring points of the grid point , respectively, is the order of the mesh. Definition respectively In the minimum of the subscript placed in In the minimum of the subscript placed in; if is a pole, then ; if is a pole, then ; in this case, ； Based on ) order spherical triangular mesh and grid index generation In the process of generating the grid index of the order spherical triangular mesh, for ) order spherical triangular mesh and the index information of its adjacent grids , In the index information of the adjacent grids in the order spherical triangular mesh is ； The grid points in the spherical triangular mesh of the 4th order The grid points in the spherical triangular mesh of the 4th order The grid points in the spherical triangular mesh of the 4th order The grid points in the spherical triangular mesh of the 4th order grid points are derived from the grid points with indices 2-6 in the 1st order spherical grid by successive down-sampling, ； grid points are derived from the grid points with indices 7-11 in the 1st order spherical grid by successive downsampling 。 5. The method of claim 2, wherein, The method for searching through the spherical icosahedral triangular mesh grid points establishes a corresponding relationship between the spherical mesh and the spherical coordinates, and includes: determining approximate grid points of arbitrary spherical coordinates in particular​ All grid points in the high-order spherical icosahedral triangular mesh except the pole points are derived from the grid points with indexes of 2-11 in the 1st-order triangular mesh through recursive up-sampling, and the grid points with indexes of 2-11 are named as "generation points"; For any spherical coordinate , get its latitude information, by comparing the latitude information of different layers of the triangular mesh, determine the layer closest to the coordinate ; By and the layer lattice spacing is determined with the nearest lattice ; According to the approximate grid points of the adjacent point information The traversal is performed, and the index thereof is determined, specifically: Put the generating point of the circle as , have ； Let the index of the , , the number of grid points of the layer be denoted by According to the above relationship, the index of the generated point is calculated corresponding to the displacement of the point ； ； For grid points in the upper part of the grid (G ), the right-up, right-side and right-down displacement vectors are defined as: ; for the grid points in the first layer, the right-up, right-side and right-down displacement vectors are defined as: ; for the grid points in the rest of the layers (G ), the right-up, right-side and right-down displacement vectors are defined as: ; Based on the defined displacement, approximate grid points are determined at their "generation points" of displacement are back-calculated; All the displacements produced when sampling the spherical mesh nodes are noted as , have ； For grid points and displacement From grid unit length Beginning, from Finding a suitable displacement Make able to be Divisible; updating , , and recording ; This step is repeated until At this point ; Based on generating point index And recorded , calculate Index , ； Based on index of the sphere determining a triangle mesh face containing the spherical coordinates of the sphere.

6. The method of claim 2, wherein, The method for projecting the ERP image from a two-dimensional space onto the parameterized sphere to obtain a spherical mesh signal includes: through an arbitrary spherical coordinate and two-dimensional coordinates The ERP image is projected from two-dimensional space onto the parameterized sphere through the conversion relationship between the two-dimensional coordinates and the corresponding spherical grid coordinates: ； wherein and are the height and width of the two-dimensional ERP image, respectively; For spherical grid point coordinates and their corresponding two-dimensional coordinates Record distance The center coordinates of the four most recent two-dimensional image pixels are , respectively located The top left, top right, bottom left, and bottom right of the image have corresponding pixel values ​​of: The pixel value of the spherical grid point is then... ； wherein , , , ; spherical grid pixel values feature extraction for subsequent spherical image signals and spherical image reconstruction; based on the obtained two-dimensional coordinates corresponding spherical grid point coordinates, and a triangular mesh surface of a spherical mesh in which the corresponding spherical grid point coordinates are located, by correspondence between the spherical mesh and the spherical coordinates. corresponding spherical grid point coordinates, and a triangular mesh surface of a spherical mesh in which the corresponding spherical grid point coordinates are located, by correspondence between the spherical mesh and the spherical coordinates.

7. The method of claim 1, wherein, The second feature extractor includes K densely connected feature extraction units, and each feature extraction unit includes a graph scattering attention mechanism and a dense convolutional network. The method for obtaining a second multi-scale feature through a second feature extractor includes: The input of the first feature extractor unit is the first multi-scale features , and the output is a d-dimensional feature where d is an integer, and is the real field. No. The input to the feature extraction unit is the first... The output of each feature extraction unit Through cross-layer connections, With the The dense convolutional network of feature extraction units outputs features. splice together to obtain the first The output features of each feature extractor unit ,in, .

8. The method of claim 7, wherein, The method for obtaining a second multi-scale feature through a second feature extractor includes: For the first feature extractor unit, the input features are compressed to a specified dimension through a convolutional layer to obtain compressed features ; The obtained compressed features are subjected to a graph scattering attention mechanism, and features output by different scattering paths in the scattering network are aggregated by weight learning to obtain adaptive multi-scale feature representation containing compressed feature low-pass and band-pass features ; Adaptive multi-scale feature representation of input features aggregates all the grid features within a hop through a three-layer dense graph convolutional network outputs the feature extractor unit deep features ; before stitching output features of one feature extractor unit as input to a feature extractor unit or as final output of the feature extractor.

9. The method of claim 1, wherein, The method for realizing feature expansion based on adaptive filter learning of Chebyshev polynomials and spherical pixel recombination of the second multi-scale feature to obtain a spherical mesh expansion feature and reconstruct an up-sampled spherical mesh signal includes: Adaptive filter learning based on Chebyshev polynomials is used to learn the output features of the second feature extractor Feature expansion is performed, specifically: From upsampling the 4th order spherical grid signal to When upsampling the 4th order spherical grid signal to the 8th order spherical grid signal, the spherical signal representation needs to be extended to 4 times the original, this representation extension uses an adaptive filter learning based on Chebyshev polynomials ； is a Chebyshev polynomial, is a highest order limit of the Chebyshev polynomial, are all learnable parameters; by training 4 different adaptive filters on filtering, extended spherical signal features are obtained through cross-layer connections , ； wherein is an inert random walk matrix of the spherical grid, is a learnable parameter, A is an adjacency matrix of the spherical grid, and D is a degree matrix of the spherical grid. Spherical mesh features generated based on different filters The extended features are subjected to spherical icosahedral mesh pixel reorganization to generate high-order icosahedral spherical mesh and corresponding spherical mesh features, specifically: From the 2i order spherical grid signal feature is extended to the 2i+1 order spherical grid signal feature, according to the 2i order spherical grid point index, the grid points are divided into four groups, and are filled in, and the corresponding relationship is: the 2i order spherical triangular grid index and the spherical triangular grid point feature correspond to each other, in turn: 0~10×4^i+1 corresponds to ; corresponds to ; corresponds to ; corresponds to ; Will Fused into The order of spherical grid, the pixel reorganization of spherical triangular grid is realized, and the expansion characteristics of spherical grid are obtained ; based on the spherical mesh feature representation up-sampling the spherical signal pixels reconstruction is realized by two layers of fully connected networks .

10. The method of claim 1, wherein, The method for obtaining a super-resolution spherical image through ERP inverse projection of the up-sampled spherical mesh signal includes: For any two-dimensional coordinates , and spherical coordinates The following relationship exists: ； wherein and are the height and width of the two-dimensional ERP image, respectively; Through the spherical coordinates, a triangular face where the spherical coordinates are located and three vertex spherical coordinates can be obtained. Based on the spherical coordinates, the vertex coordinates of the triangular face and the pixel value, the pixel value in the two-dimensional pixel space is obtained by the triangular face interpolation method . Reconstructing the up-sampled spherical signal pixels by the relationship between the two-dimensional coordinates and the spherical coordinates Projecting to the two-dimensional pixel plane from both the coordinates and the pixel values.

11. A spherical image super-resolution system, implementing the spherical image super-resolution method of claim 1, characterized in that, The method includes: An ERP projection module: constructing a parameterized spherical icosahedral triangular mesh, projecting an equidistant rectangular projection image ERP forwardly, projecting from a two-dimensional space onto the icosahedral triangular mesh to obtain a spherical mesh signal; A first feature extraction module: obtaining a first multi-scale feature through a first feature extractor for the spherical mesh signal; A second feature extraction module: obtaining a second multi-scale feature through a second feature extractor for the first multi-scale feature; A feature expansion module: realizing feature expansion based on adaptive filter learning of Chebyshev polynomials and spherical pixel recombination of the second multi-scale feature to obtain a spherical mesh expansion feature and reconstruct an up-sampled spherical mesh signal; An ERP image reconstruction module: obtaining a super-resolution spherical image through ERP inverse projection of the up-sampled spherical mesh signal.

12. A computer terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to execute the method in any one of claims 1-10.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to execute the method in any one of claims 1-10.

Images