Spherical image super-resolution method and system

Abstract

The invention provides a spherical signal super-resolution method and system, and the method comprises the steps: constructing a parameterized spherical regular icosahedron triangular mesh, projecting an equidistant rectangular projection image ERP from a two-dimensional space to the regular icosahedron triangular mesh, and obtaining a spherical mesh signal; enabling the spherical grid signal to pass through a first feature extractor to obtain a first multi-scale feature; enabling the first multi-scale feature to pass through a second feature extractor to obtain a second multi-scale feature; performing feature expansion on the second multi-scale feature based on adaptive filter learning of Chebyshev polynomial and spherical pixel recombination to obtain a spherical grid expansion feature, and reconstructing an up-sampling spherical grid signal; and performing ERP reverse projection on the up-sampling spherical grid signal to obtain a super-resolution spherical image. According to the method, high-quality representation learning and detail description can be carried out on low-resolution spherical image input, and particularly in an image super-resolution task with ultrahigh magnification, a super-resolution spherical image with more complete details and less distortion is generated.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a spherical image super-resolution method and system. Background Art

[0002] With the rise of 360-degree panoramic cameras, spherical signal processing has become a common concern in academia and industry. Considering the breadth of the viewing angle, spherical signals often require ultra-high resolution to maintain the signal's detailed features. However, due to the limitations of panoramic camera equipment performance and data transmission costs, the industry often cannot directly obtain high-resolution images or directly process high-resolution images, and needs to convert low-resolution images into high-resolution images through signal reconstruction.

[0003] After searching, we found:

[0004] In the early paper "Learning spherical convolution for fast features from 360 imagery", the convolutional neural network for traditional images was expanded. Based on the different degrees of distortion of the image at different latitudes, convolution kernels of different sizes were used to perform convolution operations on ERP images. "SphereNet: Learning spherical representations for detection and classification in omnidirectional images" adopted a similar idea, introducing the same distortion to the convolution kernel to adapt to the convolution operation of ERP images at different latitudes. Deng et al. divided the ERP image into several parts according to latitude in "Lau-net: Latitude adaptive upscaling network for omnidirectional image super-resolution". According to the common sense that the amount of information near the equator is large and the amount of information near the poles is small, different upsampling operations are used for sub-images at different latitudes. By fusing the upsampling results of different latitudes, super-resolution ERP images are obtained. In the paper "Analysis sparse coding models for image-based classification", Nishiyama introduced an additional distortion matrix to measure the degree of distortion at each position of the ERP image based on the ERP image input, and proved that the concatenation of the input image and the distortion matrix as input can improve the performance of the ERP image super-resolution model. However, directly upsampling the ERP image on a two-dimensional plane cannot completely eliminate the influence of distortion in the ERP image, which is not conducive to better defining the neighborhood of the pixel point and capturing its neighborhood features.

[0005] In subsequent work, super-resolution on the sphere is mainly achieved by upsampling the spherical grid. In SMSIR: Spherical measure based spherical image representation, Wu et al. compiled the triangles in the spherical triangular grid generated by the regular octahedron into recursive indices, derived the spherical grid lifting wavelet architecture under the index coordinates, and resampled the spherical signal by learning the implicit function of the spherical signal. In Gauge equivariant convolutional networks and the icosahedral CNN, Cohen et al. unfolded the spherical grid of the regular icosahedron, extended the convolution kernel for two-dimensional image processing to the grid, and proposed a gauge equivariant network on the sphere. In Spheresr: 360deg image super-resolution with arbitrary projection via continuous spherical image representation, Yoon et al. further constructed a directional convolution kernel based on the spherical grid of the regular icosahedron, and designed an implicit function learning method suitable for spherical super-resolution. However, due to the complex structure of the spherical icosahedron mesh and the discontinuous edges in the unfolded graph, it is difficult to construct index coordinates. How to efficiently process the spherical icosahedron triangular mesh signal has become an urgent problem to be solved. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a spherical image super-resolution method and system, and also provides a corresponding computer terminal and computer-readable storage medium.

[0007] According to a first aspect of the present invention, a spherical image super-resolution method is provided, comprising:

[0008] Constructing a parameterized spherical icosahedral triangular mesh, forward projecting the equidistant rectangular projection image ERP from the two-dimensional space onto the icosahedral triangular mesh, and obtaining a spherical mesh signal;

[0009] Passing the spherical grid signal through a first feature extractor to obtain a first multi-scale feature;

[0010] Passing the first multi-scale features through a second feature extractor to obtain second multi-scale features;

[0011] The second multi-scale feature is extended by adaptive filter learning based on Chebyshev polynomials and spherical pixel reorganization to obtain spherical grid extension features and reconstruct upsampled spherical grid signals;

[0012] ERP reverse projection is performed on the up-sampled spherical grid signal to obtain a super-resolution spherical image.

[0013] Preferably, the constructing of a parameterized spherical icosahedron triangular mesh comprises:

[0014] Calculate the spherical coordinates of all grid points in the spherical icosahedron triangular mesh;

[0015] Based on the calculated spherical coordinates of all grid points, recursively generate the spherical icosahedral triangular mesh and grid point index, and update the neighbor index of the grid point;

[0016] The corresponding relationship between the spherical grid and the spherical coordinates is established through the spherical icosahedron triangle grid point search method.

[0017] Preferably, the step of calculating the spherical coordinates of all grid points in the spherical icosahedron triangular mesh comprises:

[0018] According to the latitude, the i-order spherical grid divides the sphere into 3×2 i-1 +1 different layer, respectively 0~2 i -1 , 2nd i-1 ~2×2 i-1 and 2×2 i-1 ~3×2 i-1 Layers are defined as the upper, middle, and lower parts of the spherical mesh;

[0019] The number of grid points in the jth layer is recorded as N(j),

[0020]

[0021] Each layer of grid points in the spherical icosahedron grid is evenly distributed at the latitude, and the latitude spacing of the upper, middle and lower parts of the grid is consistent. According to the properties of the icosahedron, the second i-1 Layer and 2 i The latitudes of the layers are arctan(1 / 2) and -arctan(1 / 2);

[0022] Based on the latitude information, the spherical coordinates of the kth grid point in the jth layer are obtained as

[0023]

[0024] Preferably, the recursive generation of a spherical icosahedral triangular mesh and a grid point index based on the calculated spherical coordinates of all grid points, and updating the neighbor index of the grid points, comprises:

[0025] Recursively generate a spherical icosahedron triangular mesh, specifically:

[0026] When a low-order spherical mesh is upsampled to generate a high-order spherical mesh, the low-order spherical mesh grid points and their indexes can be embedded in the high-order spherical mesh;

[0027] For a regular icosahedral spherical triangular mesh of order i, the mesh has a total of 10×4 i-1 +2 grid points;

[0028] When generating an (i+1)-order triangle mesh from an i-order triangle mesh, you need to add 3×10×4 i-1 Each grid point of the i-order triangular grid is downsampled to the upper right, right and lower right, except for the two poles of the spherical grid, to generate 3×10×4 i-1 downsampling grid points;

[0029] Recursively generate the grid point index of the spherical icosahedron triangle mesh, specifically:

[0030] For the 1st-order spherical icosahedral triangular grid, the North Pole and South Pole grid points of the spherical grid are indexed as 0 and 1 respectively. The indexes of the 5 grid points in the northern hemisphere from west to east are 2-6 respectively, and the indexes of the 5 grid points in the southern hemisphere from west to east are 7-11 respectively.

[0031] For the grid points other than the poles with index idx in the i-order triangular mesh, the grid points of the (i+1)-order triangular mesh generated from the i-order triangular mesh are downsampled. The downsampled grid points on the upper right, right, and lower right are idx+10×4 respectively. i ,idx+2×10×4 i ,idx+3×10×4 i ;

[0032] Update the neighbor index of the grid point, specifically:

[0033] The regular icosahedral spherical triangular mesh stores the neighboring point index of each grid point in the order of "upper right, right, lower right, upper left, left, lower left";

[0034] For the 12 vertices of the regular icosahedral grid, for the poles numbered 0 and 1, the spherical grid uses the grid point's own index as the sixth adjacent point index; for grid points with indices 2-6, the upper left and upper right are the same grid point index; for grid points with indices 7-11, the lower right and lower left are the same grid point index;

[0035] For an i-order triangular mesh, the grid point's neighboring point index information is recorded as n c =(c1,c2,…,c6), define the index generation function f i (idx a ,idx b )=f(idx a ,idx b ,n a ,n b ,i), where idx a ,idx b are the indices of grid points a and b respectively, n a ,n b are the neighboring point indices of grid points a and b respectively, and i is the order of the grid;

[0036] Define p1 and p2 as idx respectively a In n b The minimum value of the subscript and idx b In n a The minimum value of the subscript in the ; if a is a pole, then p1=p2,idx a =idx b ; If b is a pole, then p2=p1,idx b =idx a ;at this time,

[0037]

[0038] When generating the grid point index of the i-order spherical triangle mesh based on the (i-1)-order spherical triangle mesh and the grid point index, for the grid point c and its adjacent grid point index information n in the (i-1)-order spherical triangle mesh, c =(c1,c2,…,c6), the index information of the adjacent points of c in the i-order spherical triangular mesh is

[0039] n′ c =(f i (idx c ,c1),f i (idx c ,c2),…,f i (idx c ,c6))

[0040] The adjacent point index information of the grid point c in the (i-1)-order spherical triangular mesh generated by upsampling to the upper right, right, and lower right are n' up ,n' mid ,n' low ;

[0041] If the grid point c is derived from the grid points indexed 2-6 in the 1st-order spherical grid by stepwise downsampling, then

[0042] n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c4,c5),f i (idx c ,c5),idx c )

[0043] n′ mid =(f i (c1,c2),c2,f i (c2,c3),f i (idx c ,c1),idx c ,f i (idx c ,c3))

[0044] n′ low =(f i (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c6),f i (c3,c6))

[0045] If the grid point c is derived from the grid points indexed 7-11 in the 1st-order spherical grid by stepwise downsampling, then

[0046] n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c1,c4),f i (dix c ,c4),idx c )

[0047] n′ mid =(f i (c1,c2),c2,f i (c2,c3),f i (idx c ,c1),idx c ,f i (idx c ,c3)).

[0048] n′ low =(fi (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c5),f i (c5,c6)

[0049] Preferably, the method of establishing the correspondence between the spherical mesh and the spherical coordinates by using a spherical icosahedral triangular mesh grid point search method includes:

[0050] Determine the approximate grid point v of any spherical coordinate (φ',θ'), specifically:

[0051] Based on the fact that all grid points except the poles in the high-order spherical icosahedral triangular mesh are derived from the grid points with indices 2-11 in the 1st-order triangular mesh through recursive upsampling, the grid points with indices 2-11 are named "generated points";

[0052] For any spherical coordinate (φ',θ'), obtain its latitude information, and determine the layer j that is closest to the coordinate by comparing it with the latitude information of different layers of the triangular mesh;

[0053] The grid point v=Cor(k) closest to (φ',θ') is determined by φ' and the distance between the grid points in the jth layer. v ,j v );

[0054] According to the neighboring point information n of the approximate grid point v v Traverse and determine its index, specifically:

[0055] Let v0 = Cor(k0,j0) be the generating point of v, then we have

[0056] |k v -k0|+|j v -j0|<2 i-1 ;

[0057] Let the index of v0 be idx0,k v = kmod(N(j) / 5), N(j) is the number of grid points in the jth layer;

[0058] According to the above relationship, the index idx0 of the generated point v0 is calculated, and v is equivalent to the displacement shift of v0.

[0059]

[0060] For the grid points at the top of the grid in the i-th order grid (0 <j<2 i-1), define the displacement vectors for the upper right, right, and lower right, respectively: (0,-1), (1,0), (1,1); for the second i-1 The grid points of the layer define the displacement vectors for the upper right, right and lower right sides respectively: (0,-1), (1,0), (0,1); for the grid points of the remaining layers (2 i-1 <j≤3×2 i-1 ), define the displacement vectors for the upper right, right side, and lower right sides respectively: (1,-1), (1,0), (0,1);

[0061] According to the defined displacement, reverse calculation is performed based on the displacement shift of the approximate grid point v to its "generating point" v0;

[0062] All displacements generated when sampling the spherical grid points are recorded as choice,

[0063]

[0064] For the grid point v and the displacement shift, starting from the grid unit length unit=1, find a suitable displacement choice[m] from choice so that shift-choice[m]×unit is divisible by 2×unit;

[0065] Update shift = shift-choice[m]×unit, unit = 2×unit, and record a i =m∈{0,1,2,3});

[0066] Repeat this step until unit=2 i-1 , at this time shift = (0,0);

[0067] Based on the generation point index idx0 and the records a2, a3, ..., a i , calculate Cor(k v ,j v ) index idx v

[0068]

[0069] Based on Cor(k v ,j v ) index idx v , determine the triangular mesh surface containing the spherical coordinates (φ',θ').

[0070] Preferably, the projecting of the ERP image from the two-dimensional space onto the parameterized sphere to obtain the spherical grid signal is specifically as follows:

[0071] Through the transformation relationship between arbitrary spherical coordinates (φ, θ) and two-dimensional coordinates (h, w), the ERP image is projected from the two-dimensional space onto the parameterized sphere to obtain the spherical grid coordinates corresponding to the two-dimensional coordinates (h, w):

[0072]

[0073] Where H and W are the height and width of the two-dimensional ERP image, respectively;

[0074] For the spherical grid point coordinates (φ0, θ0) and its corresponding two-dimensional coordinates (h0, w0), the coordinates of the four two-dimensional image pixel centers closest to (h0, w0) are (h1, w1), (h2, w2), (h3, w3), (h4, w4), which are located at the upper left, upper right, lower left and lower right of (h0, w0), respectively. Their corresponding pixel values ​​are val1, val2, val3, val4, respectively. Then the pixel value of the spherical grid point is

[0075]

[0076] Where S1 = (h4-h0) × (w4-w0), S2 = (h3-h0) × (w0-w3), S3 = (h0-h2) × (w2-w0), S4 = (h0-h1) × (w0-w1); the spherical grid pixel value val will be used for the subsequent feature extraction of the spherical image signal and the reconstruction of the spherical image;

[0077] Based on the obtained spherical grid point coordinates corresponding to the two-dimensional coordinates (h, w), the triangular mesh surface of the spherical mesh where the spherical grid point coordinates corresponding to the two-dimensional coordinates (h, w) are located is obtained through the correspondence between the spherical mesh and the spherical coordinates.

[0078] Preferably, the first feature extractor includes an anisotropic spherical scattering network and a channel attention mechanism;

[0079] The step of passing the spherical grid signal through a first feature extractor to obtain a first multi-scale feature is specifically as follows:

[0080] The multi-scale spherical wavelet in the anisotropic spherical scattering network is used to extract the multi-directional and multi-scale features of the spherical signal based on different azimuths and rotation angles. Specifically:

[0081] The basis functions of the directional spherical wavelet are related to both longitude and latitude, and the wavelet coefficients Calculated as

[0082]

[0083] Where f(ω) is the input spherical signal, is the spherical convolution, is the wavelet function on the sphere, ρ=(α,β,γ)∈SO(3) rotation group, is the rotation group.

[0084] Anisotropic spherical wavelets have localized characteristics in scale, position and direction. Indicates Ψ j Angular locality; by combining the function f and the wavelet Ψ j Expand into spherical harmonic space to obtain their respective spherical harmonic coefficients and Transformation Writing

[0085]

[0086] Based on the above transformation, the scale factor W Φ (ω) is calculated as

[0087]

[0088] Where ⊙ is the axisymmetric convolution on the sphere, and the rotation operator is written as but The scaling function is an axisymmetric function.

[0089] Writing the scaling coefficients in spherical harmonic space

[0090]

[0091] The scattering network constructed based on the above anisotropic spherical wavelet has rotation invariance:

[0092] For the input spherical grid signal The first step of scattering propagation in scattering convolutional network based on anisotropic spherical wavelet express;

[0093] For an azimuth with line n, the wavelet coefficient at any azimuth γ is written as a linear combination of n basic wavelet coefficients

[0094]

[0095] Among them, the rotation angle γ g ∈[0,2π), is the interpolation function;

[0096] The modulus of the wavelet coefficients is used as the rotation group L 2 The signal on (SO(3)) is given by Linear combination;

[0097] The scattered propagation U[j]f(α,β,γg ) Briefly denoted as U[j]f(γ g ), for a propagation path p of length m with decreasing frequency p = ((j1,g1),…,(j m ,g m )) (j i >j i+1 , i = {1,…,m}), the corresponding scattering propagation is

[0098]

[0099] The scattering coefficient output by path p is calculated as

[0100]

[0101] where 0 ≤ J0 < J, J is the maximum wavelet scale, is the axisymmetric scaling function;

[0102] Using Taking the convolution ensures rotational invariance within the range determined by J0; the scattering transform iteratively decomposes the signal into various directions and scales, and collects the scattering path coefficients of different lengths to obtain the scattering characteristics of the spherical grid signal;

[0103] H sct = Concat({S[p]f|p∈∪Ω i , 0 ≤ i ≤ M - 1})

[0104] where M is the number of layers of the anisotropic spherical scattering network, and Ω i is the set composed of all frequency-decreasing propagation paths of length i;

[0105] For the RGB three-color spherical signal input X = [X (R) , X (G) , X (B) , perform anisotropy respectively to obtain the anisotropic scattering characteristics where

[0106]

[0107] Aggregate the multi-azimuth and multi-scale information output by the anisotropic spherical scattering network through the channel attention mechanism, that is, the scattering characteristics, specifically:

[0108] For x = R, G, B, perform max pooling and average pooling respectively in the node dimension to obtain the channel coefficients a max , Combine a max , a meanThrough a two-layer shared parameter perceptron, the channel attention weight a is obtained by adding a nonlinear activation layer. channel :

[0109] a channel =σ(MLP(a max )+MLP(a mean ))

[0110] Based on the channel attention weight a channel , the scattering characteristics Compression to single channel features

[0111]

[0112] The compressed scattering features The first multi-scale feature is fused with the original signal to obtain the first multi-scale feature as the input of the second feature extractor.

[0113] Preferably, the second feature extractor comprises K densely connected feature extraction units, each of which comprises a graph scattering attention mechanism and a dense convolutional network;

[0114] The step of passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature includes:

[0115] The input of the first feature extractor unit is the first multi-scale feature Output d1-dimensional point features in, is the field of real numbers;

[0116] The input of the kth feature extraction unit is the output of the k-1th feature extraction unit Through cross-layer connections, X k-1 The dense convolutional network output feature X of the kth feature extraction unit k,out Concatenate to obtain the output features of the kth feature extractor unit Among them, k=2,…,K.

[0117] Preferably, the step of passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature comprises:

[0118] For the kth feature extractor unit, the input feature X k-1 Compress to the specified dimension through a convolution layer to obtain compressed features

[0119] The obtained compressed features are passed through a graph scattering attention mechanism, and the features output by different scattering paths in the scattering network are aggregated by weight learning to obtain an adaptive multi-scale feature representation H that includes the low-pass and band-pass features of the compressed features. gsan ;

[0120] The adaptive multi-scale feature representation H of the input features gsan aggregates all lattice point features within l-hop through a three-layer dense graph convolutional network to output the depth features of this feature extractor unit.

[0121] Concatenate the output features X of the first k feature extractor units k = Concat(X 1,out , …, X k,out ) as the input of the (k + 1)-th feature extractor unit (1 ≤ k < K) or as the final output of the feature extractor (k = K).

[0122] Preferably, the feature expansion of the second multi-scale feature is realized based on the adaptive filter learning of Chebyshev polynomials and spherical pixel recombination to obtain spherical grid expansion features and reconstruct the upsampled spherical grid signal, including:

[0123] Adopt the adaptive filter learning based on Chebyshev polynomials to perform feature expansion on the output features X of the second feature extractor K , specifically:

[0124] When upsampling from the i-th order spherical grid signal to the (i + 1)-th order spherical grid signal, the spherical signal representation needs to be expanded to 4 times the original, and this feature expansion adopts the adaptive filter learning based on Chebyshev polynomials

[0125]

[0126] T i (λ) is the Chebyshev polynomial, H is the highest order limit of the Chebyshev polynomial, and θ i and are both learnable parameters; filter X through training 4 different adaptive filters F1(λ), F2(λ), F3(λ), F4(λ), and obtain the expanded spherical signal features through cross-layer connection K where P = 1 / 2(I + AD

[0128] -1 ) is the lazy random walk matrix of the spherical grid, and Θ i is a learnable parameter;

[0129] Spherical mesh features generated based on different filters The extended features are reorganized into spherical icosahedral mesh pixels to generate high-order icosahedral spherical meshes and their corresponding spherical mesh features, specifically:

[0130] Expanding from the i-order spherical grid signal feature to the (i+1)-order spherical grid signal feature, the grid points are divided into four groups according to the (i+1)-order spherical grid grid point index, and Fill in, the corresponding relationship is: (i+1) order spherical triangle mesh index and spherical triangle mesh grid point features correspond to each other, in the order of: 0~10×4^i+1 correspondence 10×4 i +2~20×4 i +1 for 20×4 i +2~30×4 i +1 for 30×4 i +2~40×4 i +1 for

[0131] Will Fuse into the (i+1)th order spherical mesh to realize pixel reorganization of the spherical triangular mesh and obtain the spherical mesh extension feature X out ;

[0132] Based on high-order spherical mesh feature representation out , the upsampled spherical signal pixel reconstruction Z = FC(σ(FC(X out ))).

[0133] Preferably, the up-sampled spherical grid signal is subjected to ERP reverse projection to obtain a super-resolution spherical image, comprising:

[0134] It is determined that for any two-dimensional coordinates (h, w), there is the following relationship with the spherical coordinates (φ, θ):

[0135]

[0136] Through the spherical coordinates, we can get the triangular surface where the spherical coordinates are located and the spherical coordinates of the three vertices;

[0137] Based on the spherical coordinates, the vertex coordinates of the triangular surface and the pixel value, the pixel value at the two-dimensional pixel space (h, w) is obtained by the triangular surface interpolation method;

[0138] Through the relationship between two-dimensional coordinates and spherical coordinates, the upsampled spherical signal pixels are reconstructed Z and projected onto the two-dimensional pixel plane from both coordinates and pixel values.

[0139] According to a second aspect of the present invention, there is provided a spherical image super-resolution system, comprising:

[0140] ERP projection module: construct a parameterized spherical icosahedral triangular mesh, forward project the equidistant rectangular projection image ERP from two-dimensional space onto the icosahedral triangular mesh, and obtain a spherical mesh signal;

[0141] A first feature extraction module: passing the spherical grid signal through a first feature extractor to obtain a first multi-scale feature;

[0142] A second feature extraction module: passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature;

[0143] Feature extension module: the second multi-scale feature is extended based on Chebyshev polynomial adaptive filter learning and spherical pixel reorganization to obtain spherical grid extension features and reconstruct upsampled spherical grid signals;

[0144] ERP image reconstruction module: ERP reverse projection is performed on the upsampled spherical grid signal to obtain a super-resolution spherical image.

[0145] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the methods described above, or to run the system described above when executing the program.

[0146] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above, or to run the system described above.

[0147] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the existing spherical image super-resolution technology:

[0148] Embodiments of the present invention The spherical image super-resolution method and system provided by embodiments of the present invention can perform high-quality representation learning and detail portrayal on low-resolution spherical image inputs, especially in ultra-high magnification image super-resolution tasks, to generate super-resolution spherical images with more complete details and less distortion.

[0149] The spherical image super-resolution method and system provided by the embodiments of the present invention adopt a parameterizable spherical icosahedral triangular mesh, which solves the problem of difficulty in node encoding due to the discontinuity of the unfolded graph of the spherical icosahedral triangular mesh, improves the efficiency of spherical convolution, and greatly reduces the computational complexity of the interpolation algorithm for projecting the spherical mesh signal into the two-dimensional pixel space.

[0150] The spherical image super-resolution method and system provided by the embodiment of the present invention adopts ERP projection based on bilinear interpolation of pixel coordinates (from two-dimensional pixel space to spherical grid space) and ERP inverse projection based on grid triangle grid point coordinate interpolation (from spherical grid space to two-dimensional pixel space) to realize the mutual conversion of spherical signals between two-dimensional pixel space and spherical grid space, effectively reduce the noise introduced by ERP projection and ERP inverse projection, and improve the quality of super-resolution images.

[0151] The spherical image super-resolution method and system provided by the embodiment of the present invention adopts a spherical signal preprocessing method of anisotropic spherical scattering to extract multi-azimuth and multi-scale features of spherical signals, aggregates spherical signal scattering features through a channel attention mechanism, and splices the spherical signal preprocessing features with the original spherical image. The spherical signal preprocessing method not only expands the feature channels of the input spherical image, but also has rotation invariance to the angle, thereby enhancing the overall expression ability of the model.

[0152] The spherical image super-resolution method and system provided by the embodiment of the present invention adopts a feature extractor unit based on a graph scattering attention mechanism and a dense convolutional network to extract multi-scale deep features of spherical signals. Compared with the traditional graph convolutional network, the graph scattering attention mechanism can take into account both low-frequency and medium-high-frequency information in the input spherical grid image, avoid the problem of feature over-smoothing that may occur in multi-layer graph convolutional networks, and adaptively aggregate features through the attention mechanism, thereby enhancing the expression ability while discarding redundant features.

[0153] The spherical image super-resolution method and system provided by the embodiments of the present invention adopt multiple densely connected feature extractor units as feature extractors, which can enhance the flow of features between different feature extractor units, increase the diversity of hidden layer features, and enhance the expression ability of the feature extractor.

[0154] The spherical image super-resolution method and system provided by the embodiment of the present invention uses a method based on spherical icosahedron triangular grid pixel reorganization to aggregate spherical signal extension features. Compared with traditional spherical feature upsampling, the spherical feature expansion operation based on spherical grid pixel reorganization is more convenient and has a more obvious geometric meaning, which improves the feature expansion efficiency while ensuring a better feature expansion effect.

[0155] The spherical image super-resolution method and system provided by the present method embodiment were tested on multiple ERP image data sets with different scenes and information distributions. Compared with the existing methods (see the embodiments), good results were achieved, and a certain improvement in the effect was achieved in the ultra-high magnification ERP image super-resolution task. BRIEF DESCRIPTION OF THE DRAWINGS

[0156] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0157] Figure 1 FIG. 4 is a flowchart of a spherical image super-resolution method in one embodiment of the present invention.

[0158] Figure 2 It is a schematic diagram of the operation of projecting an ERP image from a two-dimensional pixel space to a spherical grid space in a preferred embodiment of the present invention.

[0159] Figure 3 It is a working schematic diagram of a feature preprocessing feature extractor (first feature extractor) in a preferred embodiment of the present invention.

[0160] Figure 4 It is a working schematic diagram of a spherical signal depth feature extractor (second feature extractor) in a preferred embodiment of the present invention.

[0161] Figure 5 It is a schematic diagram of the operation of spherical feature expansion and signal reconstruction in a preferred embodiment of the present invention.

[0162] Figure 6 It is a schematic diagram of the operation of projecting a spherical image from a spherical grid space to a two-dimensional pixel space in a preferred embodiment of the present invention.

[0163] Figure 7 The figure is a schematic diagram of the module composition of a spherical signal super-resolution system based on a spherical icosahedron triangular grid in one embodiment of the present invention. DETAILED DESCRIPTION

[0164] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0165] An embodiment of the present invention provides a spherical image super-resolution method, which can perform high-quality representation learning and detail portrayal on low-resolution spherical image input, especially in ultra-high magnification image super-resolution tasks, to generate a super-resolution spherical image with more complete details and less distortion.

[0166] like Figure 1 As shown, the spherical image super-resolution method provided by this embodiment may include the following steps:

[0167] S100, constructing a parameterized spherical icosahedral triangular mesh, forward projecting an equidistant rectangular projection image (ERP image) from a two-dimensional space onto the parameterized sphere, and obtaining a spherical mesh signal;

[0168] S200, passing the spherical grid signal through a first feature extractor to obtain a first multi-scale feature;

[0169] S300, passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature;

[0170] S400, implementing feature expansion of the second multi-scale feature based on adaptive filter learning of Chebyshev polynomials and spherical pixel reorganization to obtain spherical grid expansion features and reconstruct an upsampled spherical grid signal;

[0171] S500, reversely projecting the upsampled spherical grid signal to a two-dimensional pixel space through ERP to obtain a super-resolution spherical image.

[0172] The above embodiment parameterizes the regular icosahedral spherical grid, greatly improving the processing efficiency of the spherical grid signal. At the same time, through multi-scale characterization learning, both the local and global information of the spherical signal are taken into account, ensuring the accurate representation of the spherical signal and better characterizing the details of the signal. In the ultra-high magnification spherical image super-resolution task, the present invention achieves better results under multiple indicators and has great advantages in panoramic camera image processing.

[0173] The problem of difficulty in node encoding due to the discontinuity of the unfolded graph of the spherical icosahedral triangular mesh leads to low efficiency of spherical convolution. Therefore, in a preferred embodiment of the present invention, a preferred spherical parameterization method is provided, so as to further project the ERP image from the two-dimensional space onto the parameterized sphere to obtain a spherical mesh signal. Specifically, Figure 2 As shown, S100 can adopt the following steps:

[0174] S101, calculating the spherical coordinates of all grid points in the spherical icosahedron triangular grid;

[0175] S102, based on the calculated spherical coordinates of all grid points, recursively generate a spherical icosahedral triangular mesh and grid point indexes, and update the neighbor indexes of the grid points;

[0176] S103, based on the spherical icosahedron triangular mesh and grid point index of S102, and updating the neighbor index of the grid point, using the spherical icosahedron triangular mesh grid point search algorithm to establish the corresponding relationship between the spherical mesh and the spherical coordinates;

[0177] S104, projecting the ERP image from the two-dimensional space onto the parameterized sphere to obtain the spherical coordinates of the ERP image, and obtaining the spherical grid signal of the ERP image based on the correspondence between the spherical grid and the spherical coordinates obtained in S103.

[0178] In a preferred embodiment, S101, the spherical coordinates of all grid points in the spherical icosahedron triangular mesh are calculated, and the specific process is as follows:

[0179] According to the latitude, the i-order spherical grid divides the sphere into 3×2 i-1 +1 different layer.

[0180] Respectively put the 0th to 2nd i-1 , 2nd i-1 ~2×2 i-1 and 2×2 i-1 ~3×2 i-1 Layers are defined as the upper, middle, and lower parts of the spherical mesh.

[0181] The number of grid points in the jth layer is recorded as N(j),

[0182]

[0183] Each layer of grid points in the spherical icosahedron grid is evenly distributed at the latitude, and the latitude spacing of the upper, middle and lower parts of the grid is consistent. According to the properties of the icosahedron, it is easy to get the second i-1 Layer and 2 i The latitudes of the layers are arctan(1 / 2) and -arctan(1 / 2) respectively.

[0184] The spherical coordinates of the kth grid point in the jth layer are

[0185]

[0186] In a preferred embodiment, S102, based on the calculated spherical coordinates of all grid points, a spherical icosahedral triangular mesh and grid point index are recursively generated, and the neighbor index of the grid point is updated. The specific process is as follows:

[0187] S102.1 recursively generates a spherical icosahedral triangular mesh based on the calculated spherical coordinates of all grid points.

[0188] When a low-order spherical mesh is upsampled to generate a high-order spherical mesh, the low-order spherical mesh points and their indices can be embedded in the high-order spherical mesh.

[0189] For a regular icosahedral spherical triangular mesh of order i, the mesh has a total of 10×4 i-1 +2 grid points. When generating an (i+1)-order triangle mesh from an i-order triangle mesh, 3×10×4i-1 Except for the two poles of the spherical mesh, each grid point of the i-order triangular mesh is downsampled to the upper right, right and lower right to generate 3×10×4 i-1 downsampling grid points.

[0190] S102.2 Recursively generate the grid point index of the spherical icosahedron.

[0191] For a 1st-order spherical icosahedral triangular grid, the North Pole and South Pole grid points of the spherical grid are indexed as 0 and 1 respectively. Taking the 0 degree longitude as the dividing line, the indexes of the 5 grid points in the northern hemisphere are 2-6 from west to east, and the indexes of the 5 grid points in the southern hemisphere are 7-11 from west to east.

[0192] For the grid points other than the poles with index idx in the i-order triangular mesh, the grid points of the (i+1)-order triangular mesh generated from the i-order triangular mesh are downsampled. The downsampled grid points on the upper right, right, and lower right are idx+10×4 respectively. i ,idx+2×10×4 i ,idx+3×10×4 i .

[0193] S102.3 Updates all neighboring point indices of spherical grid points of any order.

[0194] In this embodiment, it is specified that the regular icosahedral spherical triangular mesh stores the adjacent point index of each grid point in the order of "upper right, right side, lower right side, upper left side, left side, lower left side".

[0195] For the 12 vertices of the regular icosahedral mesh, since the vertices have only 5 neighboring points, the neighboring point index needs to be adjusted appropriately. For the poles numbered 0 and 1, the spherical mesh uses the grid point's own index as the sixth neighboring point index; for grid points with indices 2-6, the upper left and upper right have the same grid point index; for grid points with indices 7-11, the lower right and lower left have the same grid point index.

[0196] The grid point's neighboring point index information is recorded as n c =(c1,c2,…,c6), define the index generation function f i (idx a ,idx b )=f(idx a ,idx b ,n a ,n b ,i), where idx a ,idx b are the indices of grid points a and b respectively, n a ,n bare the neighboring point indices of grid points a and b respectively, and i is the order of the grid. Define p1 and p2 as idx respectively. a In n b The minimum value of the subscript and idx b In n a If a is a point, then p1=p2,idx a =idx b ; If b is a pole, then p2=p1,idx b =idx a .at this time,

[0197]

[0198] When generating the grid point index of the i-order spherical triangle mesh based on the (i-1)-order spherical triangle mesh and the grid point index, for the grid point c and its adjacent grid point index information n in the (i-1)-order spherical triangle mesh, c =(c1,c2,…,c6), the index information of the adjacent points of c in the i-order spherical triangle mesh is

[0199] n′ c =(f i (idx c ,c1),f i (idx c ,c2),…,f i (idx c ,c6))

[0200] The adjacent point index information of the grid point c in the (i-1)-order spherical triangular mesh generated by upsampling to the upper right, right, and lower right are n' up ,n' mid ,n' low If the grid point c is derived from the grid points indexed 2-6 in the 1st-order spherical grid by stepwise downsampling, then

[0201] n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c4,c5),f i (idx c ,c5),idx c )

[0202] n′ mid =(f i (c1,c2),c2,f i (c2,c3),f i (idx c,c1),idx c ,f i (idx c ,c3))

[0203] n′ low =(f i (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c6),f i (c3,c6))

[0204] If the grid point c is derived from the grid points indexed 7-11 in the 1st-order spherical grid by stepwise downsampling, then

[0205] n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c1,c4),f i (idx c ,c4),idx c )

[0206] n′ mid =(f i (c1,c2),c2,f i (c2,c3),f i (idx c ,c1),idx c ,f i (idx c ,c3))

[0207] n′ low =(f i (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c5),f i (c5,c6)

[0208] In a preferred embodiment, S103, based on the spherical icosahedral triangular mesh and grid point index of S102, and updating the neighbor index of the grid point, a spherical icosahedral triangular mesh grid point search algorithm is used to establish a corresponding relationship between the spherical mesh and the spherical coordinates (i.e., for an i-order spherical triangular mesh with N grid points, given any spherical coordinates, determine the triangular mesh surface containing the spherical coordinates). The specific process is as follows:

[0209] S103.1 Determine an approximate grid point v of arbitrary spherical coordinates (φ',θ').

[0210] Since all grid points except the poles in the high-order spherical icosahedral triangular mesh are derived from the grid points with indices 2-11 in the 1st-order triangular mesh through recursive downsampling, the grid points with indices 2-11 are named "generator points". For any spherical coordinate (φ',θ'), the layer j closest to the coordinate is determined by the latitude of the different layers of the triangular mesh, and one of the grid points v=cor(k v ,j v ).

[0211] S103.2 Based on the neighboring point information n of the approximate grid point v v Traverse and determine the triangular mesh surface containing the spherical coordinates (φ',θ').

[0212] Let v0 = Cor(k0,j0) be the generating point of v, then we have

[0213] |k v -k0|+|j v -j0|<2 i-1

[0214] Let the index of v0 be idx0,k v = kmod(N(j) / 5). Based on the above relationship, we can calculate the index idx0 of the generated point v0 and the displacement shift of v0.

[0215]

[0216] For the grid points at the top of the grid in the i-th order grid (0 <j<2 i-1 ), define the displacement vectors for the upper right, right, and lower right, respectively: (0,-1), (1,0), (1,1); for the second i-1 The grid points of the layer define the displacement vectors for the upper right, right and lower right sides respectively: (0,-1), (1,0), (0,1); for the grid points of the remaining layers (2 i-1 <j≤3×2 i-1 ), respectively define the displacement vectors for the upper right, right side, and lower right side: (1,-1), (1,0), (0,1). Based on the displacement defined above, the displacement shift of the approximate grid point v to its "generating point" v0 is reversed. All displacements generated when sampling the spherical grid points are recorded as choice,

[0217]

[0218] For the grid point v and the displacement shift, starting from unit = 1, find a suitable displacement choice[m] from choice so that shift-choice[m]×unit is divisible by 2×unit. Update shift = shift-choice[m]×unit, unit = 2×unit, and record a i =m∈{0,1,2,3}). Repeat this step until unit=2 i-1 , shift=(0,0). Based on the generation point index idx0 and the recorded a2, a3, ..., a i , we can calculate Cor(k v ,j v ) index idx v

[0219]

[0220] In the above embodiment, a parameterizable spherical icosahedral triangular mesh is used to recursively generate grid point indexes and neighborhood information, and a recursive search method is used to find the approximate grid point index of any spherical coordinate and the spherical mesh triangular face on which it is located. This solves the problem of difficulty in node encoding due to the discontinuity of the unfolded graph of the spherical icosahedral triangular mesh, improves the efficiency of spherical convolution, and greatly reduces the computational complexity of projecting the spherical mesh signal into the two-dimensional pixel space.

[0221] In a preferred embodiment, S104 is implemented to project the ERP image from the two-dimensional space onto the parameterized sphere to obtain the spherical coordinates of the ERP image, and based on the correspondence between the spherical grid obtained in S103 and the spherical coordinates, the spherical grid signal of the ERP image is obtained. The specific process is as follows:

[0222] Given any spherical coordinates (φ, θ), there is the following transformation relationship between them and the two-dimensional coordinates (h, w):

[0223]

[0224] Where H and W are the height and width of the two-dimensional ERP image, respectively.

[0225] Then, bilinear interpolation in pixel space is used to determine the pixel value of each spherical grid point during projection:

[0226] For the spherical grid point coordinates (φ0, θ0) and its corresponding two-dimensional coordinates (h0, w0), the coordinates of the four two-dimensional image pixel centers closest to (h0, w0) are (h1, w1), (h2, w2), (h3, w3), (h4, w4), which are located at the upper left, upper right, lower left and lower right of (h0, w0), respectively. Their corresponding pixel values ​​are val1, val2, val3, val4, respectively. Then the pixel value of the spherical grid point is

[0227]

[0228] Where S1 = (h4-h0) × (w4-w0), S2 = (h3-h0) × (w0-w3), S3 = (h0-h2) × (w2-w0), S4 = (h0-h1) × (w0-w1). The spherical grid pixel value val will be used for the subsequent feature extraction of the spherical image signal and the reconstruction of the spherical image.

[0229] By implementing S103, the above embodiment can know the spherical coordinates (φ, θ) corresponding to the two-dimensional coordinates (h, w) and the spherical mesh triangle surface. The above embodiment performs projection based on the spherical mesh signal, effectively reducing the computational complexity of the interpolation algorithm.

[0230] When the two-dimensional ERP image is projected from the two-dimensional space onto the parameterized sphere and the spherical mesh signal is obtained, further feature extraction is required. In a preferred embodiment of the present invention, the first feature extractor includes a scattering network based on anisotropic spherical wavelet and a channel attention mechanism. S200 can adopt the following steps, such as Figure 3 As shown:

[0231] S21, input spherical image X=[X (R) ,X (G) ,X (B) ] Through the anisotropic spherical scattering network, the multi-azimuth and multi-scale information of the image is extracted. f represents the signal of a certain channel of the spherical image, and the anisotropic spherical wavelet is used to calculate the wavelet coefficients of the image at different azimuths.

[0232]

[0233] in is the spherical convolution, is the wavelet function on the sphere, ρ=(α,β,γ)∈SO(3) rotation group, wavelet scale Indicates Ψ j Similarly, the scale of the transformation is W Φ (ω) can be calculated as

[0234]

[0235] where ⊙ is the axisymmetric convolution on the sphere, and the rotation operator is written as Then For the input signal The scattering propagation in the first step of the scattering convolutional network based on the anisotropic spherical wavelet is represented by Using the modulus of the wavelet coefficient as the signal on the rotation group L 2 (SO(3)), it can be composed of linear combination. Denote the scattering propagation U[j]f(α,β,γ g ) at the azimuth angle (α,β) as U[j]f(γ g ). For the propagation path p = ((j1,g1),…,(j m ,g m )) (j i >j i+1 , i = {1,…,m}) with decreasing frequency of length m, the corresponding scattering propagation is

[0236]

[0237] The scattering coefficient output by path p is calculated as

[0238]

[0239] where 0 ≤ J0 < J, J is the maximum wavelet scale, is the axisymmetric scaling function. Taking the convolution with ensures the rotational invariance within the range determined by J0. The scattering transform iteratively decomposes the signal into various directions and scales, and collects the scattering path coefficients of different lengths to obtain the multi-scale feature representation of the spherical signal

[0240] H sct = Concat({S[p]f|p ∈ ∪Ω i , 0 ≤ i ≤ M - 1})

[0241] where M is the number of layers of the anisotropic spherical scattering network, and Ω i is the set composed of all propagation paths with decreasing frequency of length i. For the RGB three-color spherical signal input X = [X (R) , X (G) , X (B) , perform anisotropy respectively to obtain the anisotropic scattering features where

[0242]

[0243] Similarly.

[0244] S22, using the channel attention mechanism to output features of the anisotropic spherical scattering network are compressed into single channel features. For example, the maximum pooling and mean pooling are performed in the node dimension to obtain the channel coefficient a max , A max ,a mean Through a two-layer shared parameter perceptron, the channel attention weight a is obtained by adding a nonlinear activation layer. channel :

[0245] a channel =σ(MLP(a max )+MLP(a mean ))

[0246] Based on the channel attention weight a channel , the scattering characteristics Compression to single channel features

[0247]

[0248] S23, by splicing the scattering features aggregated by S202 with the signals of the corresponding channels of the original image, the spherical signal preprocessing features X0 are obtained.

[0249]

[0250] The above embodiment adopts a spherical signal preprocessing method of anisotropic spherical scattering to extract multi-azimuth and multi-scale features of spherical signals, aggregates spherical signal scattering features through a channel attention mechanism, and splices the spherical signal preprocessing features with the original spherical image. This spherical signal preprocessing method not only expands the feature channels of the input spherical image, but also has rotation invariance to the angle, thereby enhancing the overall expression ability of the model.

[0251] Furthermore, in a preferred embodiment of the present invention, the second feature extractor is composed of K feature extraction units composed of graph scattering attention mechanisms and dense convolutional networks, and K is a positive integer. Where:

[0252] The input of the first feature extractor unit is the fusion feature of the anisotropic spherical scattering feature and the original spherical signal (When the input spherical image is RGB three-channel, d in =6), output spherical mesh features in, is the field of real numbers;

[0253] The input of the kth feature extraction unit is the output of the k-1th feature extraction unit Through cross-layer connection, the output features of the dense convolutional network of the k-th feature extraction unit are concatenated with the input features to obtain the output features of the k-th feature extraction unit where k = 2, …, K.

[0254] Further, in a preferred embodiment of S300, the spherical grid preprocessed features are input into a feature extractor based on a graph scattering attention mechanism and a dense convolutional network, as Figure 4 shown, including:

[0255] S31, for the k-th feature extraction unit, compress the input feature X k-1 to a specified dimension through a one-dimensional convolutional layer to obtain a compressed feature

[0256] S32, pass the obtained compressed feature through a graph scattering attention mechanism (GSAN), and aggregate the features output from different scattering paths in the scattering network by weight learning to obtain an adaptive multi-scale feature representation H containing the low-pass and band-pass features of the compressed feature gsan ;

[0257] S33, calculate the difference between the adaptive multi-scale feature representation H of the input feature gsan and all the lattice point features within its l-hop, and concatenate it with its own feature;

[0258] S34, based on a three-layer dense graph convolutional network, learn the S33 concatenated features and output the depth features of this feature extraction unit

[0259] S35, concatenate the output features X of the first k feature extraction units k = Concat(X 1,out , …, X k,out ) as the input of the (k + 1)-th feature extraction unit (1 ≤ k < K) or as the final output of the feature extractor (k = K).

[0260] In a preferred embodiment of S32, pass the compressed spherical grid signal X cmp through an M-layer graph scattering network based on diffusion wavelets and a series of graph convolutional networks, and use the attention mechanism to aggregate the output features of different scattering paths in the scattering network and the low-pass features obtained in the graph convolutional network to obtain an adaptive multi-scale feature representation H of the spherical grid signal gsan , including:

[0261] S321, based on the lazy random walk matrix P = 1 / 2(I + AD -1), the feature extraction part of the graph scattering attention mechanism includes bandpass features based on the scattering network and low-pass features based on the graph convolutional network. The graph scattering network uses diffuse wavelets to extract multi-scale features of the input signal.

[0262]

[0263] For the scattering path and the input signal X cmp , the scattering transfer characteristics are

[0264]

[0265] Different paths Scattering transfer characteristics of the output The scattering characteristics are obtained by taking the modulus

[0266] S322, the graph convolution part uses a multi-order normalized ring adjacency matrix Extract low-pass features X of different scales of receptive fields gcn,i =W i X cmp ;

[0267] S323, the number of feature channels output by the graph scattering network and the number of feature channels output by the graph convolution network are respectively recorded as C gst and C gcn , based on the output feature X sct,i (1≤i≤C sct ) and X gcn,i (1≤i≤C gcn ), the attention mechanism is used to fuse features. The shared attention parameter is denoted as a, and the point-by-point attention weights of different feature channels are

[0268] e gcn,i =[X cmp ||X gcn,i ]a,e sct,i =[X cmp ||X sct,i ]a

[0269] Where || represents feature concatenation. gcn,i and e sct,i , calculate the normalized attention weight α gcn,i ,α sct,i

[0270]

[0271] S324, aggregate multi-channel features through attention weights to obtain adaptive multi-scale feature representation H gsan

[0272]

[0273] Where ⊙ is the Hadamard product of the vector, σ(·)=LeakyRelu(·);

[0274] In a preferred embodiment of S33 and S34, the spherical grid signal output by the graph scattering network attention mechanism is adaptively represented by a multi-scale feature H gsan Aggregate all grid features within l-hop through a three-layer dense graph convolutional network and output the deep features of the feature extractor unit include:

[0275] S331, point-by-point aggregation using residual splicing The l-hop neighborhood characteristics

[0276]

[0277] Splice All get Represents the number of grid points within the distance grid point v l-hop. Concatenate features point by point to obtain the output feature vector of the dense convolutional network

[0278] S341, H dense,in Through a three-layer dense convolutional network, hidden features are learned. dense,in Through 2D convolution and nonlinear activation layers, with H dence,in Concatenate as the input of the first hidden

[0279]

[0280] H dense,hd1 Through a layer of 2D convolution and nonlinear activation layer, with H dense,hd1 Concatenate as the input of the second hidden layer

[0281]

[0282] H dense,hd2 Through a layer of 2D convolution and nonlinear activation layer, with H dense,hd2 Concatenate as the output feature of the dense convolutional network

[0283]

[0284] S342, using maximum pooling to get the output of the dense convolutional network

[0285]

[0286] S343, through cross-layer splicing, X k,out and the output feature X of the k-1th feature extractor unit k-1 concatenated as the output X of the kth feature extractor unit k =Concat(X k-1 ,X k,out )=Concat(X 1,out ,…,X k,out ).of

[0287] In the above embodiment, a feature extractor unit based on a graph scattering attention mechanism and a dense convolutional network is used to extract multi-scale deep features of spherical signals. Compared with the traditional graph convolutional network, the graph scattering attention mechanism can take into account both low-frequency and medium-high-frequency information in the input spherical mesh image, avoiding the problem of feature over-smoothing that may occur in multi-layer graph convolutional networks. At the same time, the attention mechanism adaptively aggregates features, enhances the expression ability and discards redundant features.

[0288] In a preferred embodiment of the present invention, S400, an adaptive filter based on Chebyshev polynomials is used to train the extended spherical features, the extended features are integrated into a spherical grid with a higher resolution by spherical grid pixel reorganization, and a super-resolution spherical signal is reconstructed from the spherical grid extended features. The following steps may be used, such as Figure 5 As shown:

[0289] S41, using the output feature X of the feature extractor K As the input of the feature expansion part, an adaptive filter based on Chebyshev polynomial is used to learn the feature extractor output feature X K Perform 4x feature expansion:

[0290]

[0291] Here, T i (λ) is the i-th order Chebyshev polynomial, H is the highest order limit of the Chebyshev polynomial, θ i and i All are learnable parameters. By training 4 different adaptive filters F j (λ)(j=1,2,3,4) for X K Filtering, through cross-layer connection to obtain extended spherical signal features

[0292]

[0293] Where P = 1 / 2 (I + AD -1 ) is the lazy random walk matrix of the spherical grid, Θ iis a learnable parameter;

[0294] S42, spherical mesh features generated based on different filters The extended features are reorganized into spherical icosahedral grid pixels, and the low-order spherical grid extended features are integrated into the spherical triangular grid with higher resolution. For the (i+1)-order spherical grid, the grid points are divided into four groups according to the spherical grid grid point index, and Fill in, the corresponding relationship between index and grid feature is

[0295]

[0296] in express The index is 2 to 10×4 i +1 grid feature;

[0297] S43, based on high-order spherical mesh feature representation X out (Output of the spherical icosahedron pixel reconstruction layer), upsampled spherical signal pixel reconstruction is achieved through a two-layer fully connected network

[0298] Z=FC(σ(FC(X out ))).

[0299] The above embodiment uses a method based on spherical icosahedron triangle grid pixel reorganization to aggregate spherical signal expansion features. Compared with traditional spherical feature upsampling, the spherical feature expansion operation based on spherical grid pixel reorganization is more convenient and has a more obvious geometric meaning, which improves the feature expansion efficiency while ensuring a better feature expansion effect.

[0300] In a preferred embodiment of the present invention, S500 projects the super-resolution spherical mesh image into a two-dimensional pixel space by using triangular surface geometric interpolation to output a super-resolution ERP image Z out , you can take the following steps, such as Figure 6 As shown:

[0301] S51, for any two-dimensional pixel coordinate (h, w), use the ERP projection relationship to calculate the spherical coordinate (φ, θ) corresponding to the two-dimensional coordinate

[0302]

[0303] S52, determining the spherical coordinates of the approximate grid point v of the spherical coordinates (φ, θ) through the spherical grid grid point coordinates. Based on a recursive spherical grid grid point search algorithm, determining the index of the approximate grid point v.

[0304] The coordinates of the spherical icosahedron triangular grid points are used to determine one of the grid points v=Cor(k v ,jv ). The grid points with indices 2-11 in the first-order spherical grid are called "generator points". For any grid point other than the north and south poles in any higher-order spherical grid, the grid point is derived from a "generator point" through multiple recursive upsampling. Let v0 = Cor(k0, j0) be the generator point of v, and we have

[0305] |k v -k0|+|j v -j0|<2 i-1

[0306] Let the index of v0 be idx0,k v = kmod(N(j) / 5). Based on the above relationship, we can calculate the index idx0 of the generated point v0 and the displacement shift of v0.

[0307]

[0308] For the grid points at the top of the grid in the i-th order grid (0 <j<2 i-1 ), define the displacement vectors for the upper right, right, and lower right, respectively: (0,-1), (1,0), (1,1); for the second i-1 The grid points of the layer define the displacement vectors for the upper right, right and lower right sides respectively: (0,-1), (1,0), (0,1); for the grid points of the remaining layers (2 i-1 <j≤3×2 i-1 ), respectively define the displacement vectors for the upper right, right side, and lower right side: (1,-1), (1,0), (0,1). Based on the displacement defined above, the displacement shift of the approximate grid point v to its "generating point" v0 is reversed. All displacements generated when sampling the spherical grid points are recorded as choice,

[0309]

[0310] For the grid point v and the displacement shift, starting from unit = 1, find a suitable displacement choice[m] from choice so that shift-choice[m]×unit is divisible by 2×unit. Update shift = shift-choice[m]×unit, unit = 2×unit, and record a i =m∈{0,1,2,3}). Repeat this step until unit=2 i-1 , shift=(0,0). Based on the generation point index idx0 and the recorded a2, a3, ..., a i , we can calculate Cor(k v ,j v ) index idx v

[0311]

[0312] S53, based on the index of the approximate grid point v and its adjacent grid point information n v , determine the triangular face where (φ,θ) is located, and the indices and spherical coordinates of the three vertices of the triangular face.

[0313] Based on the index of the approximate grid point (φ',θ') of (φ,θ), determine the six neighbors of (φ',θ') through the adjacent node index information of the approximate grid point (φ',θ'), which are recorded as v1, v2, ..., v6. Traverse the triangular faces Δv1vv2, Δv2vv3, Δv3vv6, Δv5vv6, Δv4vv5, Δv1vv4 related to the approximate grid point v to determine the triangular mesh surface containing the spherical coordinates (φ,θ).

[0314] S54, based on the spherical coordinates (φ, θ) and the spherical coordinates of the three vertices of the triangular surface (φ1, θ1), (φ2, θ2), (φ3, θ3), triangular surface geometric interpolation is used to obtain the pixel value at the two-dimensional pixel space (h, w).

[0315] The three vertices of the triangle where the spherical coordinates (φ, θ) are located are recorded as points A, B, and C, and the corresponding spherical coordinates are (φ1, θ1), (φ2, θ2), (φ3, θ3), and the pixel values ​​are val A ,val B ,val C . Let (φ,θ) be point O, then the pixel value at (h,w) in the two-dimensional pixel space is

[0316]

[0317] The area of ​​the triangle can be calculated by vector cross product in Cartesian coordinate system. By using the above triangle interpolation method, the upsampled spherical grid signal Z is projected onto the two-dimensional pixel plane to obtain the super-resolution ERP image Z out .

[0318] In order to obtain better super-resolution, in a preferred embodiment, the spherical signal super-resolution method obtains learnable parameters based on end-to-end training according to training data, including:

[0319] The spherical signal super-resolution method uses the output super-resolution signal I pred and truth value I gt The mean square error (MSE) between the output super-resolution image and the true value is used to measure the difference between the output super-resolution image and the true value.

[0320]

[0321] Where C is the number of channels of the ERP image, Mean((I pred (c)-I gt (c)) 2 ) is the output super-resolution signal I pred and truth value I gt The mean square error of the pixel under channel c. The training of the model uses the mean square error mixed regularization term as the loss function of the training model

[0322]

[0323] Where β is a hyperparameter, and Θ is a learnable parameter in the spherical signal super-resolution deep network. The gradient of the parameter Θ is calculated according to the loss function Loss, and the parameter Θ in the spherical image super-resolution model is updated by back propagation. The above loss function is minimized by iterative optimization to obtain the parameters of the spherical image super-resolution model. Specifically, the spherical signal super-resolution deep model can be trained using the mean square error defined on a spherical grid or defined in a two-dimensional pixel space. When the mean square error defined in a two-dimensional pixel space is used for training, I pred =Z out ,I gt =Z gt is the true value of the high-resolution ERP image.

[0324]

[0325] Among them I pred (h,w,c) and I gt (h, w, c) represents the output super-resolution ERP image and the true value at channel c and pixel (h, w), respectively. H and W are the height and width of the super-resolution ERP image. When the mean square error defined on the spherical triangular mesh is used for training, I pred =Z,I gt =Proj(Z gt ) is the projection of the high-resolution true value image on the high-order spherical triangular mesh based on the bilinear geometric interpolation method.

[0326]

[0327] Among them I pred (v,c) and I gt (v,c) represents the output upsampled spherical grid signal and the true value at channel c and grid point v, respectively, and N represents the number of grid points on the spherical triangular grid.

[0328] For better understanding, the technical solution provided by the above embodiment of the present invention is further described in detail below in combination with a specific application example.

[0329] In this specific application example, the ERP image super-resolution is achieved. The spherical signal super-resolution method based on spherical triangular mesh, spherical scattering and spherical pixel recombination mainly includes the following 6 steps:

[0330] Step 1: Generate a spherical icosahedral triangular mesh with a grid point number that matches the number of pixels in the ERP image. Project the spherical grid point coordinates (φ, θ) to the two-dimensional pixel space (h, w) through the correspondence between the two-dimensional pixel coordinates and the spherical coordinates in the ERP projection. Find the four pixel center coordinates (h1, w1), (h2, w2), (h3, w3), (h4, w4) closest to (h, w) and their corresponding pixel values ​​val1, val2, val3, val4, and obtain the pixel value val at the spherical grid point (φ, θ) through bilinear interpolation to obtain the spherical image X.

[0331] Step 2: Spherical image X = [X (R) ,X (G) ,X (B) ] Through an anisotropic spherical scattering network, the multi-azimuth and multi-scale information of each channel is learned separately The channel attention mechanism is used to aggregate the scattering features of each channel respectively, compress the scattering features of each channel to 1 dimension, and obtain the aggregated scattering features X and Stitching to obtain the preprocessing features of the spherical image

[0332] Step 3: The spherical surface preprocessing feature X0 is passed through a series of densely connected feature extractor units based on graph scattering attention mechanism and dense convolutional network to extract the multi-scale hierarchical features of the spherical grid signal. The input of the first feature extractor unit is the fusion feature of the anisotropic spherical scattering feature and the original spherical signal. Export spherical mesh features in, is the real number domain. The input of the kth feature extraction unit is the output of the k-1th feature extraction unit By taking the dense convolutional network output features of the kth feature extraction unit And the k-1th feature extractor unit output features Splicing to get the output feature of the kth feature extractor unit d k =d in +k(d m +3d hd ),(k=1,…,K).

[0333] Step 4: The deep feature X output by the feature extractor KX is respectively converted to K Expanded to i is the number of low-order spherical grid points. The (i+1)-order spherical grid is grouped by index.

[0334] The grid points are divided into 4 groups, Fill in to form the depth feature X defined on the high-order spherical grid grid points out . Upsampled spherical image reconstruction Z = FC(σ(FC(X out ))).

[0335] Step 5: Based on the coordinate correspondence of the ERP projection, the two-dimensional pixel coordinates (h, w) of the super-resolution ERP image are projected onto the sphere to obtain the spherical coordinates (φ, θ). The approximate grid coordinates (φ', θ') of (φ, θ) are found through the spherical grid coordinates, and the index and adjacent point index information of (φ', θ') are obtained through the recursive spherical grid fast search algorithm, and then the triangular face containing (φ, θ) and the spherical coordinates of the vertices of the triangular face (φ1, θ1), (φ2, θ2), (φ3, θ3) are determined. The pixel value at (φ, θ) is obtained through the triangular face interpolation algorithm, that is, the pixel value of (h, w). This operation is performed to obtain the center coordinates of each pixel in the two-dimensional pixel space, and the output super-resolution ERP image Z is obtained. out .

[0336] Step 6: When training with the mean square error defined in two-dimensional pixel space, calculate the output super-resolution ERP image Z out and the ground truth Z of the high-resolution ERP image gt The difference between , and at the same time calculate the regularization term of the trainable parameters in the deep network and calculate the loss function

[0337]

[0338] When training is performed using the mean square error defined in the spherical grid pixel space, step 5 needs to be replaced by:

[0339] In step 5, the high-resolution ERP image truth value Z is projected using the same projection method as step 1 gt Projected onto a high-order spherical grid, the high-resolution spherical image truth Proj(Z gt ).

[0340] Calculate the reconstructed super-resolution spherical mesh image Z and the true value Proj(Z gt ), and calculate the regularization term of the trainable parameters in the deep network and the loss function

[0341]

[0342] Repeat steps 2 to 6 until the network converges. The updated learnable parameters Θ are used for the ERP signal super-resolution task.

[0343] In this specific application case, the comparative evaluation methods of the implementation effects are Bicuic, SRCNN, VDSR, LapSRN, MemNet, MSRN, EDSR, D-DBPN, RCAN, EBRN, 360-SS, LAU-Net and SphereSR. The data of all the comparative models are from the experimental parts of the two papers "LAU-Net: Latitude Adaptive Upscaling Network for Omnidirectional Image Super-resolution" and "SphereSR: 360° Image Super-Resolution with ArbitraryProjection via Continuous Spherical Image Representation". The embodiment of the present invention uses the training set in the ODI-SR dataset that is the same as the two papers for model training, and uses the test set in the ODI-SR dataset and the SUN dataset for testing. The training dataset consists of 1200 samples, the true value of which is a high-definition ERP image of 2048×1024. According to the different super-resolution magnifications, 128×64 (×16 magnification), 256×128 (×8 magnification), 512×256 (×4 magnification) and 1024×512 (×2 magnification) inputs are provided. The ODI-SR test set and SUN dataset each contain 100 samples, the true value is also a high-definition ERP image of 2048×1024, and the above-mentioned inputs of different resolutions are provided according to the different super-resolution magnifications. Tables 1 and 2 provide the super-resolution results of spherical images with ×8 magnification under 256×128 input and ×16 magnification under 128×64 input, respectively. WS-PSNR is used to measure the difference between the output super-resolution image and the high-resolution true value, and WS-SSIR is used to measure the difference between the output super-resolution image and the high-resolution true value in terms of overall structure.

[0344] Table 1

[0345] Table 2

[0346] Compared with SRCNN, 360-SS, VDSR and other models that process ERP images in two-dimensional pixel space, the method of the embodiment of the present invention processes spherical image signals in spherical space, more clearly defines the neighborhood range of spherical image pixel points, and can completely eliminate the influence of distortion when representing learning. Compared with LAU-Net, the method of the embodiment of the present invention is based on a spherical triangular mesh, and different numbers of grid points are allocated at different latitudes, so there is no problem of discontinuous edges when reconstructing and splicing images of different latitudes. Compared with SphereNet, which is also based on a spherical triangular mesh, the method of the embodiment of the present invention optimizes the index and grid feature storage method of the spherical icosahedral triangular mesh, extends the pixel reorganization for the two-dimensional pixel space to the spherical mesh, and adopts a simpler convolution kernel to improve the efficiency of spherical mesh signal processing. At the same time, by introducing an anisotropic spherical scattering network to extract the multi-azimuth and multi-scale features of the spherical grid signal, and the graph scattering attention mechanism and the dense convolutional network to further extract the deep features of the spherical grid signal, the method of the embodiment of the present invention can take into account the low-pass and band-pass characteristics of the spherical grid signal in the characterization learning stage, thereby ensuring the diversity of its features, enhancing the expression ability of the model, and having more advantages in the ultra-high magnification spherical image super-resolution task.

[0347] ERP image super-resolution is widely used in virtual modeling, autonomous driving and other aspects. However, due to the limitations of hardware and communication costs, the industry often cannot directly obtain high-resolution images or directly process high-resolution images. It is necessary to convert low-resolution images into high-resolution images through signal reconstruction. The spherical signal super-resolution method based on a spherical icosahedral triangular mesh provided in the above embodiment of the present invention can generate a relatively smooth and detail-enhanced high-resolution spherical signal based on a low-resolution spherical image, which is of great significance for subsequent mesh modeling, 3D rendering, etc. Therefore, the spherical image super-resolution method based on a spherical triangular mesh provided in the above embodiment of the present invention is of relatively important significance in industrial spherical image processing.

[0348] An embodiment of the present invention provides a spherical image super-resolution system based on a spherical icosahedron triangular grid.

[0349] like Figure 7 As shown, the spherical image super-resolution system based on a spherical icosahedron triangular grid provided by this embodiment may include:

[0350] ERP projection module: Project the two-dimensional ERP image onto the i-order spherical icosahedron triangular mesh through geometric interpolation;

[0351] Feature pre-extraction module: anisotropic spherical scattering network is used to learn the multi-directional and multi-scale features of spherical signals, spherical scattering features are aggregated through the channel attention mechanism, and the learned spherical signal representation is fused with the original spherical signal;

[0352] Deep feature learning module: It uses multiple inclusion graph scattering attention mechanisms (GSAN) and dense convolutional networks to extract multi-scale deep features of input spherical signals;

[0353] Spherical feature expansion module: It uses Chebyshev polynomial-based adaptive filter training to expand the output of multi-scale depth features of spherical signals, and obtains upsampled spherical signal depth features by reorganizing spherical triangular grid pixels;

[0354] ERP image reconstruction module: reconstructs super-resolution spherical signals through a two-layer fully connected network, and projects the spherical signals to a plane through ERP inverse projection to complete super-resolution ERP image reconstruction.

[0355] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0356] An embodiment of the present invention provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0357] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0358] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0359] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0360] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0361] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0362] The spherical image super-resolution method and system based on the icosahedral spherical triangular mesh provided by the above-mentioned embodiment of the present invention firstly projects the two-dimensional ERP image onto the i-order spherical icosahedral triangular mesh through geometric interpolation to obtain a spherical mesh image; then, an anisotropic spherical scattering network is used to learn the multi-directional and multi-scale features of the spherical signal, and the scattering features are aggregated through the channel attention mechanism, and the learned spherical signal representation is fused with the original spherical signal to obtain the spherical image preprocessing features; then, multiple graph-containing scattering attention mechanisms and dense convolutional networks are used to extract the multi-scale depth features of the input spherical signal; then, adaptive filter training is performed to expand the output multi-scale depth features of the spherical signal, and the upsampled spherical signal depth features are obtained through spherical triangular mesh pixel reorganization; finally, the reconstructed spherical mesh image is projected to the two-dimensional pixel space through geometric interpolation to obtain a super-resolution ERP image. Compared with the existing methods, the present invention solves the problem of difficulty in node encoding of the spherical icosahedral triangular mesh due to the discontinuity of the unfolded graph, improves the efficiency of spherical convolution, and greatly reduces the computational complexity of the interpolation algorithm for projecting the spherical mesh signal into the two-dimensional pixel space. It takes into account both the local and global features of the spherical mesh signal during representation learning, ensures the accurate representation of the spherical image, and makes the geometric meaning of feature upsampling on the sphere more distinct by extending pixel reorganization to the spherical triangular mesh. It has greater advantages in the ultra-high magnification spherical image super-resolution task, and has practical application value in industrial applications such as virtual modeling and autonomous driving.

[0363] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0364] All matters not covered in the above embodiments of the present invention are well known in the art.

[0365] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A spherical image super-resolution method, characterized in that: include: Constructing a parameterized spherical icosahedral triangular mesh, forward projecting the equidistant rectangular projection image ERP from the two-dimensional space onto the icosahedral triangular mesh, and obtaining a spherical mesh signal; Passing the spherical grid signal through a first feature extractor to obtain a first multi-scale feature; Passing the first multi-scale features through a second feature extractor to obtain second multi-scale features; The second multi-scale feature is extended by adaptive filter learning based on Chebyshev polynomials and spherical pixel reorganization to obtain spherical grid extension features and reconstruct upsampled spherical grid signals; ERP reverse projection is performed on the up-sampled spherical grid signal to obtain a super-resolution spherical image.

2. A spherical image super-resolution method according to claim 1, characterized in that: The method of constructing a parameterized spherical icosahedral triangular mesh comprises: Calculate the spherical coordinates of all grid points in the spherical icosahedron triangular mesh; Based on the calculated spherical coordinates of all grid points, recursively generate the spherical icosahedral triangular mesh and grid point index, and update the neighbor index of the grid point; The corresponding relationship between the spherical grid and the spherical coordinates is established through the spherical icosahedron triangle grid point search method.

3. A spherical image super-resolution method according to claim 2, characterized in that: The step of calculating the spherical coordinates of all grid points in the spherical icosahedron triangular grid comprises: According to the latitude, the i-order spherical grid divides the sphere into 3×2 i-1 +1 different layer, respectively 0~2 i-1 , 2nd i-1 ~2×2 i-1 and 2×2 i-1 ~3×2 i-1 Layers are defined as the upper, middle, and lower parts of the spherical mesh; The number of grid points in the jth layer is recorded as N(j), Each layer of grid points in the spherical icosahedron grid is evenly distributed at the latitude, and the latitude spacing of the upper, middle and lower parts of the grid is consistent. According to the properties of the icosahedron, the second i-1 Layer and 2 i The latitudes of the layers are arctan(1 / 2) and -arctan(1 / 2); Based on the latitude information, the spherical coordinates of the kth grid point in the jth layer are obtained as 4. The spherical image super-resolution method according to claim 2, characterized in that: The method of recursively generating a spherical icosahedral triangular mesh and a grid point index based on the calculated spherical coordinates of all grid points, and updating the neighbor index of the grid points includes: Recursively generate a spherical icosahedron triangular mesh, specifically: When a low-order spherical mesh is upsampled to generate a high-order spherical mesh, the low-order spherical mesh grid points and their indexes can be embedded in the high-order spherical mesh; For a regular icosahedral spherical triangular mesh of order i, the mesh has a total of 10×4 i-1 +2 grid points; When generating an (i+1)-order triangle mesh from an i-order triangle mesh, you need to add 3×10×4 i-1 Each grid point of the i-order triangular grid is downsampled to the upper right, right and lower right, except for the two poles of the spherical grid, to generate 3×10×4 i-1 downsampling grid points; Recursively generate the grid point index of the spherical icosahedron triangle mesh, specifically: For the 1st-order spherical icosahedral triangular grid, the North Pole and South Pole grid points of the spherical grid are indexed as 0 and 1 respectively. The indexes of the 5 grid points in the northern hemisphere from west to east are 2-6 respectively, and the indexes of the 5 grid points in the southern hemisphere from west to east are 7-11 respectively. For the grid points other than the poles with index idx in the i-order triangular mesh, the grid points of the (i)+1)-order triangular mesh generated from the i-order triangular mesh are downsampled. The indices of the downsampled grid points on the upper right, right, and lower right are idx+10×4 respectively. i ,idx+2×10×4 i ,idx+3×10×4 i ; Update the neighbor index of the grid point, specifically: The icosahedral spherical triangular mesh stores the neighboring point index of each grid point in the order of "upper right, right, lower right, upper left, left, lower left"; For the 12 vertices of the regular icosahedral grid, for the poles numbered 0 and 1, the spherical grid uses the grid point's own index as the sixth adjacent point index; for grid points with indices 2-6, the upper left and upper right are the same grid point index; for grid points with indices 7-11, the lower right and lower left are the same grid point index; For an i-order triangular mesh, the grid point's neighboring point index information is recorded as n c =(c1,c2,…,c6), define the index generation function f i (idx a ,idx b )=f(idx a ,idx b ,n a ,n b ,i), where idx a ,idx b are the indices of grid points a and b respectively, n a ,n b are the neighboring point indices of grid points a and b respectively, and i is the order of the grid; Define p1 and p2 as idx respectively a In n b The minimum value of the subscript and idx b In n a The minimum value of the subscript in the ; if a is a pole, then p1=p2,idx a =idx b ; If b is a pole, then p2=p1,idx b =idx a ;at this time, When generating the grid point index of the i-order spherical triangle mesh based on the (i-1)-order spherical triangle mesh and the grid point index, for the grid point c and its adjacent grid point index information n in the (i-1)-order spherical triangle mesh, c =(c1,c2,…,c6), the index information of the adjacent points of c in the i-order spherical triangular mesh is n′ c =(f i (idx c ,c1),f i (idx c ,c2),…,f i (idx c ,c6)) The adjacent point index information of the grid point c in the (i-1)-order spherical triangular mesh generated by upsampling to the upper right, right, and lower right are n' up ,n' mid ,n' low ; If the grid point c is derived from the grid points indexed 2-6 in the 1st-order spherical grid by stepwise downsampling, then n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c4,c5),f i (idx c ,c5),idx c ) n′ mid =(f i (c1,x2),c2,f i (c2,x3),(f i (idx c ,c1),idx c ,f i (idx c ,c3)) n′ low =(f i (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c6),f i (c3,c6)) If the grid point c is derived from the grid points indexed 7-11 in the 1st-order spherical grid by stepwise downsampling, then n′ up =(c1,f i (c1,c2),f i (idx c ,c2),f i (c1,c4),f i (idx c ,c4),idx c ) n′ mid =(f i (c1,c2),c2,f i (c2,c3),f i (idx c ,c1),idx c ,fi(idx c ,c3)) n′ low =(f i (idx c ,c2),f i (c2,c3),c3,idx c ,f i (idx c ,c5),f i (c5,c6))。 5. The spherical image super-resolution method according to claim 2, characterized in that: The method of finding grid points of a spherical icosahedron triangle grid to establish a corresponding relationship between a spherical grid and spherical coordinates includes: Determine the approximate grid point v of any spherical coordinate (φ',θ'), specifically: Based on the fact that all grid points except the poles in the high-order spherical icosahedral triangular mesh are derived from the grid points with indices 2-11 in the 1st-order triangular mesh through recursive upsampling, the grid points with indices 2-11 are named "generated points"; For any spherical coordinate (φ',θ'), obtain its latitude information, and determine the layer j that is closest to the coordinate by comparing it with the latitude information of different layers of the triangular mesh; The grid point v=Cor(k) closest to (φ',θ') is determined by φ' and the distance between the grid points in the jth layer. v ,j v ); According to the neighboring point information n of the approximate grid point v v Traverse and determine its index, specifically: Let v0 = Cor(k0,j0) be the generating point of v, then we have |k v -k0|+|j v -j0|<2 i-1 ; Let the index of v0 be idx0,k v = kmod(N(j) / 5), N(j) is the number of grid points in the jth layer; According to the above relationship, the index idx0 of the generated point v0 is calculated, and v is equivalent to the displacement shift of v0. For the grid points at the top of the grid in the i-th order grid (0 <j<2 i-1 ), define the displacement vectors for the upper right, right, and lower right, respectively: (0,-1), (1,0), (1,1); for the second i-1 The grid points of the layer define the displacement vectors for the upper right, right and lower right sides respectively: (0,-1), (1,0), (0,1); for the grid points of the remaining layers (2 i-1 <j≤3×2 i-1 ), define the displacement vectors for the upper right, right side and lower right sides respectively: (1,-1),91,0),(0,1); According to the defined displacement, reverse calculation is performed based on the displacement shift of the approximate grid point v to its "generating point" v0; All displacements generated when sampling the spherical grid points are recorded as choice, For the grid point v and the displacement shift, starting from the grid unit length unit=1, find a suitable displacement choice[m] from choice so that shift-choice[m]×unit is divisible by 2×unit; Update shift = shift-choice[n]×unit, unit = 2×unit, and record a i =m∈{0,1,2,3}); Repeat this step until unit=2 i-1 , at this time shift = (0,0); Based on the generation point index idx0 and the records a2, a3, ..., a i , calculate Cor(k v ,j v ) index idx v Based on Cor(k v ,j v ) index idx v , determine the triangular mesh surface containing the spherical coordinates (φ',θ').

6. A spherical image super-resolution method according to claim 2, characterized in that: The ERP image is projected from the two-dimensional space onto the parameterized sphere to obtain the spherical grid signal, specifically: Through the transformation relationship between arbitrary spherical coordinates (φ, θ) and two-dimensional coordinates (h, w), the ERP image is projected from the two-dimensional space onto the parameterized sphere to obtain the spherical grid coordinates corresponding to the two-dimensional coordinates (h, w): Where h and W are the height and width of the two-dimensional ERP image, respectively; For the spherical grid point coordinates (φ0, θ0) and its corresponding two-dimensional coordinates (h0, w0), the coordinates of the four two-dimensional image pixel centers closest to (h0, w0) are (h1, w1), (h2, w2), (h3, w3), (h4, w4), which are located at the upper left, upper right, lower left and lower right of (h0, w0), respectively. Their corresponding pixel values ​​are val1, val2, val3, val4, respectively. Then the pixel value of the spherical grid point is Where S1 = (h4-h0) × (w4-w0), S2 = (h3-h0) × (w0-w3), S3 = (h0-h2) × (w2-w0), S4 = (h0-h1) × (w0-w1); the spherical grid pixel value val will be used for the subsequent feature extraction of the spherical image signal and the reconstruction of the spherical image; Based on the obtained spherical grid point coordinates corresponding to the two-dimensional coordinates (h, W), the triangular mesh surface of the spherical mesh where the spherical grid point coordinates corresponding to the two-dimensional coordinates (h, w) are located is obtained through the correspondence between the spherical mesh and the spherical coordinates.

7. The spherical image super-resolution method according to claim 1, characterized in that: The first feature extractor includes an anisotropic spherical scattering network and a channel attention mechanism; The step of passing the spherical grid signal through a first feature extractor to obtain a first multi-scale feature is specifically as follows: The multi-scale spherical wavelet in the anisotropic spherical scattering network is used to extract the multi-directional and multi-scale features of the spherical signal based on different azimuths and rotation angles. Specifically: The basis functions of the directional spherical wavelet are related to both longitude and latitude, and the wavelet coefficients Calculated as Where f(ω) is the input spherical signal, is the spherical convolution, is the wavelet function on the sphere, ρ=(α,β,γ)∈SO(3) rotation group, is the rotation group, is the spherical angular position; Anisotropic spherical wavelets have localized characteristics in scale, position and direction. Indicates Ψ j Angular locality; by combining the function f and the wavelet Ψ j Expand into spherical harmonic space to obtain their respective spherical harmonic coefficients and Transformation Writing is the Wigner D equation; Based on the above transformation, the scale factor W Φ (ω) is calculated as Where ⊙ is the axisymmetric convolution on the sphere, and the rotation operator is written as but The scaling function is an axisymmetric function. Writing the scaling coefficients in spherical harmonic space The scattering network constructed based on the above anisotropic spherical wavelet has rotation invariance: For the input spherical grid signal The first step of scattering propagation in scattering convolutional network based on anisotropic spherical wavelet express; For an azimuth with line N, the wavelet coefficient at any azimuth γ is written as a linear combination of N basic wavelet coefficients Among them, the rotation angle γ g ∈[0,2π), is the interpolation function; The modulus of the wavelet coefficients is used as the rotation group L 2 The signal on (SO(3)) is given by Linear combination; The scattered propagation U[j]f(α,β,γ g ) is abbreviated as U[j]f(γ g ), for a frequency-decreasing propagation path of length m, p = ((j1,g1),…,(j m ,g m ))(j i >j i+1 , i={1,…,m}), the corresponding scattering propagation is The scattering coefficient of the output of path p is calculated as where \(0\leq J_0 < J\), \(J\) is the maximum wavelet scale, is an axisymmetric scaling function; use The convolution ensures the rotation invariance within the range determined by J0; the scattering transform iteratively decomposes the signal into various directions and scales, and collects the scattering path coefficients of different lengths to obtain the scattering characteristics of the spherical grid signal; Where M is the number of layers of the anisotropic spherical scattering network, Ω i is the set of all frequency-decreasing propagation paths of length i; For the RGB three-color spherical signal input X=[X (R) ,X (G) ,X (B) ] are anisotropically analyzed to obtain anisotropic scattering characteristics in The multi-azimuth and multi-scale information output by the anisotropic spherical scattering network is aggregated through the channel attention mechanism, namely, the scattering features, which are specifically: for x=R, G, B, perform maximum pooling and mean pooling in the node dimension to obtain the channel coefficient a max , A max ,a mean Through a two-layer shared parameter perceptron, the channel attention weight a is obtained through the summation and nonlinear activation layer. channel : a channel =σ(MLP(a max )+MLP(a mean )) Based on the channel attention weight a channel , the scattering characteristics Compression to single channel features The compressed scattering features The first multi-scale feature is fused with the original signal to obtain the first multi-scale feature as the input of the second feature extractor.

8. The spherical image super-resolution method according to claim 1, characterized in that: The second feature extractor includes K densely connected feature extraction units, each of which includes a graph scattering attention mechanism and a dense convolutional network; The step of passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature includes: The input of the first feature extractor unit is the first multi-scale feature Output d1-dimensional point features in, is the field of real numbers; The input of the kth feature extraction unit is the output of the k-1th feature extraction unit Through cross-layer connections, X k-1 The dense convolutional network output feature X with the kth feature extraction unit k,out Concatenate to obtain the output features of the kth feature extractor unit Among them, k=2,…,K.

9. A spherical image super-resolution method according to claim 8, characterized in that: The step of passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature includes: For the kth feature extractor unit, the input feature X k-1 Compress to the specified dimension through a convolution layer to obtain compressed features The obtained compressed features are passed through a graph scattering attention mechanism, and the features output by different scattering paths in the scattering network are aggregated by weight learning to obtain an adaptive multi-scale feature representation H containing compressed features, low-pass and band-pass features. gsan ; The adaptive multi-scale feature representation H of the input feature gsan Aggregate all grid features within l-hop through a three-layer dense graph convolutional network and output the deep features of the feature extractor unit Concatenate the output features X of the first k feature extractor units k = Convcat(X 1,out , …, X k,out ) as the input of the (k + 1)-th feature extractor unit (1 ≤ k < K) or as the final output of the feature extractor (k = K).

10. The spherical image super-resolution method according to claim 1, characterized in that: The method of implementing feature expansion based on adaptive filter learning of Chebyshev polynomials and spherical pixel reorganization for the second multi-scale feature, obtaining spherical grid expansion features, and reconstructing upsampled spherical grid signals comprises: Adaptive filter learning based on Chebyshev polynomials is used to output feature X from the second feature extractor. K Perform feature expansion, specifically: When upsampling from the i-th order spherical grid signal to the (i+1)-th order spherical grid signal, the spherical signal representation needs to be expanded to 4 times the original. This feature expansion adopts adaptive filter learning based on Chebyshev polynomials. T i (λ) is the Chebyshev polynomial, H is the highest order limit of the Chebyshev polynomial, θ i and i All are learnable parameters; by training four different adaptive filters F1(λ), ​​F2(λ), F3(λ), F4(λ) on X K Filtering, through cross-layer connection to obtain extended spherical signal features Where P = 1 / 2 (I + AD -1 ) is the lazy random walk matrix of the spherical grid, Θ i are learnable parameters, A is the adjacency matrix of the spherical grid, and D is the degree matrix of the spherical grid; Spherical mesh features generated based on different filters The extended features are reorganized into spherical icosahedral mesh pixels to generate high-order icosahedral spherical meshes and their corresponding spherical mesh features, specifically: Expanding from the i-order spherical grid signal feature to the (i+1)-order spherical grid signal feature, the grid points are divided into four groups according to the (i+1)-order spherical grid grid point index, and Fill in, the corresponding relationship is: (i+1) order spherical triangle mesh index and spherical triangle mesh grid point features correspond to each other, in the order of: 0~10×4^i+1 correspondence 10×4 i +2~20×4 i +1 for 20×4 i +2~30×4 i +1 for 30×4 i +2~40×4 i +1 for Will Fuse into the (i+1)th order spherical mesh to realize pixel reorganization of the spherical triangular mesh and obtain the spherical mesh extension feature X out ; Based on the spherical mesh feature representation X out , the upsampled spherical signal pixel reconstruction Z = FC(σ(FC(X out ))).

11. The spherical image super-resolution method according to claim 1, characterized in that: The method of using ERP reverse projection on the upsampled spherical grid signal to obtain a super-resolution spherical image comprises: It is determined that for any two-dimensional coordinates (h, w), there is the following relationship with the spherical coordinates (φ, θ): Through the spherical coordinates, we can get the triangular surface where the spherical coordinates are located and the spherical coordinates of the three vertices; Based on the spherical coordinates, the vertex coordinates of the triangular surface and the pixel value, the pixel value at the two-dimensional pixel space (h, w) is obtained by the triangular surface interpolation method; Through the relationship between two-dimensional coordinates and spherical coordinates, the upsampled spherical signal pixels are reconstructed Z and projected onto the two-dimensional pixel plane from both coordinates and pixel values.

12. A spherical image super-resolution system, characterized in that: Includes: ERP projection module: construct a parameterized spherical icosahedral triangular mesh, forward project the equidistant rectangular projection image ERP from two-dimensional space onto the icosahedral triangular mesh, and obtain a spherical mesh signal; A first feature extraction module: passing the spherical grid signal through a first feature extractor to obtain a first multi-scale feature; A second feature extraction module: passing the first multi-scale feature through a second feature extractor to obtain a second multi-scale feature; Feature extension module: the second multi-scale feature is extended based on Chebyshev polynomial adaptive filter learning and spherical pixel reorganization to obtain spherical grid extension features and reconstruct upsampled spherical grid signals; ERP image reconstruction module: ERP reverse projection is performed on the upsampled spherical grid signal to obtain a super-resolution spherical image.

13. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 11, or to run the system described in claim 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 11, or to run the system described in claim 12.

Images