Astronomical image splicing processing method based on spherical Fourier neural operator

Abstract

The invention discloses an astronomical image splicing processing method based on a spherical Fourier neural operator. The astronomical image splicing processing method mainly comprises the steps of preprocessing and spherical mapping, fusion repair, loss design and fine adjustment, sensor non-consistency modeling and robustness enhancement, inverse transformation and post-processing. According to the method, the plane images collected by the sensors are mapped to celestial sphere surface signals, then parallel fusion and restoration are carried out in the spherical frequency domain, and finally the fusion result is back-projected back to the required view, so that seamless and high-precision large-view-field celestial sphere panoramic stitching is realized. According to the method, black edge gaps, tedious calibration and geometric distortion in multi-sensor splicing can be eliminated fundamentally, the observation efficiency can be improved, and a powerful guarantee can be provided for rapid response of astronomical events and data processing pipelines.

Description

Technical Field

[0001] The present invention belongs to the technical field of astronomical image processing methods, and in particular relates to an astronomical image splicing processing method based on a spherical Fourier neural operator. Background Art

[0002] In astronomical observations, multi-sensor joint imaging technology is an important means of expanding the field of view and improving observation efficiency. By collaborating with multiple sensors, a wider area of ​​the sky can be covered, capturing more information about celestial objects. However, this technology has long faced three major challenges in engineering practice, severely hindering its scientific value. First, the physical independence of the sensors leads to "black edges" in the image stitching area. Due to mechanical deviations during the installation of different sensors and differences in optical system distortion, the imaging planes of the sensors cannot be perfectly aligned. When two images are stitched together, the physical gap between the sensors creates a "black edge" with no photosensitive data in the overlapped area. Second, the sensor parameter calibration process is complex and inefficient. Each sensor's intrinsic parameters (such as focal length, optical center position, and rotation angle) require high-precision calibration to ensure the consistency of the image coordinate system. Existing technologies typically rely on manual star point matching, calculating parameters by identifying identical stars in two images. However, this process is time-consuming and labor-intensive, and is susceptible to interference from factors such as noise and atmospheric disturbances. Finally, traditional methods treat sensor images as planar projections (such as latitude and longitude projections), but astronomical images are essentially projections of local slices of the celestial surface. When the stitching area approaches the celestial poles, the geometric distortion of the planar projection is significantly amplified, resulting in deviations in the star positions.

[0003] Multi-camera or multi-sensor array stitching techniques usually rely on planar feature matching (e.g. ) combined Algorithm, Matthew Brown proposed a Features and A fully automatic panoramic stitching method is proposed. This method does not require manual initialization, is robust to rotation, scaling and illumination changes, and achieves seamless high fidelity through multi-band mixing; however, it does not correct lens distortion and parallax, is prone to artifacts in dynamic scenes, and has high computational overhead. Jinxian Qi et al. proposed a method based on improved Feature extraction and improvement The purified image stitching method introduces Detection operator and The descriptor realizes the detection and description of rotation-invariant and more evenly distributed features, combined with Euclidean distance rough matching, distance threshold pre-screening and Refinement effectively improves the real-time performance and registration accuracy of the mosaic, but this method is sensitive to parameters (such as distance threshold), and does not specially correct lens distortion and parallax, and has limited adaptability to large angle changes or dynamic scenes.

[0004] The phase correlation-based frequency domain registration method registers and fuses the section images collected by each sensor after plane projection, Foroosh et al. proposed a phase correlation sub-pixel registration method based on frequency domain multi-phase decomposition, which realizes non-interpolation sub-pixel displacement estimation by analyzing the normalized mutual power spectrum structure. It has high calculation accuracy, is sensitive to noise and light changes, and is suitable for multispectral images. However, it is only applicable to the translation model and cannot handle rotation and non-rigid transformation, and is strongly dependent on the identification of the main peak of sparse images.

[0005] A manual parameter optimization method using star points or star pattern calibration is used to correct internal and external parameters, John A proposed a method of using star pattern images for geometric calibration, which is used for the "Orion" optical navigation camera of the United States NASA ( ). This method extracts the centroid position of the star point through image preprocessing, and combines the known star direction to use Nonlinear least squares algorithm to jointly optimize the internal parameters of the camera, lens distortion parameters and the attitude error between the camera and the star tracker, and to construct a complete calibration model containing ten parameters, which can realize high-precision automatic calibration on orbit to meet the strict requirements of autonomous navigation geometry in deep space missions. However, this method has certain limitations, such as the high correlation between the principal point position and the camera attitude error, which is difficult to estimate accurately at the same time, so the principal point coordinates need to be fixed in advance.

[0006] With the development of large-scale sky survey projects (such as ), the number of sensors has increased from dozens to hundreds, and the traditional method has been unable to meet the engineering requirements of high precision and high efficiency. Therefore, a new method is needed to fundamentally solve the geometric distortion and realize automatic calibration and repair. SUMMARY

[0007] To overcome at least one of the deficiencies of the prior art described above, the present application provides an astronomical image mosaic processing method based on spherical Fourier neural operator (SFO) , which aims to fundamentally eliminate the three major bottlenecks of "black edge" gaps, tedious calibration and geometric distortion in multi-sensor mosaic.

[0008] To achieve the above purpose, the present application provides the following technical solutions:

[0009] An astronomical image stitching processing method based on spherical Fourier neural operator, which maps the planar images collected by each sensor to the spherical surface signal, then parallel fusion and repair in the spherical frequency domain, finally the fusion result is back projected to the required view to realize seamless, high-precision large field of view celestial sphere panoramic stitching. The method specifically comprises: (1) preprocessing and spherical mapping, using star point matching or installation record to obtain the rough attitude of each camera, and the back projection of each pixel to the unit sphere and interpolation to the unified grid to obtain discrete spherical signals. Fusion repair, input the multi-channel spherical signal and its external parameter into , first do spherical harmonic transform, then learn frequency domain filter weight according to order mode, and complete "black edge" filling, fine correction and view fusion with the help of stacked point (3) loss design and fine tuning, in the training process, the "rotation fine tuning" loss is constructed to automatically correct the rough external parameter, the "black edge recovery" loss is used to guide the network to fill the overlapping missing area, and the "smooth regular" loss is added to suppress high-frequency artifacts. The weighted combination of the three forms an end-to-end joint optimization target, which makes the best balance between geometric alignment, data completion and visual smoothing. (4) modeling of non-uniform sensors and robustness enhancement, the brightness of each sensor image is standardized; gamma transformation, Gaussian noise and random occlusion are added in the training to simulate illumination, noise and defects; and the model's resistance to unknown minor changes is improved through adversarial perturbation training. (5) inverse transformation and post-processing, the inverse harmonic transform is performed on the fused spherical image to obtain a continuous and seamless celestial sphere distribution, and then the final compensation image or panoramic image is generated by cutting or back projecting to the observation plane as needed.

[0010] 1. preprocessing and spherical mapping

[0011] The goal of this stage is to convert the planar images collected by multiple cameras into signal representation on a unified spherical grid, to realize frequency domain fusion processing in spherical space.

[0012] 1.1 rough calibration

[0013] The rough external parameter (rotation matrix or quaternion) of each camera is obtained by traditional star point matching or sensor installation record.

[0014] For the th sensor, its rotation matrix and intrinsic parameter are obtained by traditional star point matching or installation record.

[0015] 1.2 dynamic attitude correction

[0016] Considering the device vibration or attitude drift in actual shooting, a dynamic attitude estimation mechanism is introduced. The gyro or IMU data is fused, and the disturbance term is calculated by extended Kalman filter (EKF) and correct in real time:

[0017]

[0018] 1.3 Pixel back-projection

[0019] According to these extrinsic parameters, each planar image is back-projected pixel by pixel to the unit sphere, and interpolated on an equi-azimuthal or equi-polar grid to generate a discrete spherical signal. The planar pixel coordinates are converted to unit sphere coordinates :

[0020]

[0021] After projection, the discrete spherical signal is obtained by interpolation on an equi-azimuthal or equi-polar grid , preparing for subsequent frequency domain processing.

[0022] 2. SFNO fusion and repair

[0023] This stage constructs a frequency domain neural network based on spherical Fourier transform, which performs information fusion, black edge filling and geometric fine-tuning on multiple spherical signals.

[0024] 2.1 Spherical harmonic transform

[0025] Each channel is decomposed into spherical harmonic basis functions :

[0026]

[0027] 2.2 Cross-modal fusion

[0028] Considering multi-band observation information (such as visible light, infrared, ultraviolet), a channel attention mechanism is introduced.

[0029] 2.2.1 Weight calculation:

[0030]

[0031] 2.2.2 Fusion output:

[0032]

[0033] 2.3 Frequency domain filtering

[0034] For each order modality, the network learns a set of scalar weights and performs filtering:

[0035] ​​​

[0036] 2.4 points repair

[0037] for each fusion coefficient application layer full-connection network:

[0038]

[0039] wherein are the repaired spherical harmonic coefficients.

[0040] 2.5 multi-channel fusion

[0041] In the frequency domain, all N sensor channels are fused into a global signal:

[0042]

[0043] fusion coefficient learned dynamically from each channel by a point-wise MLP, to emphasize sensor outputs with higher information quality.

[0044] 2.6 rotational equivariance

[0045] SFNO guarantees equivariance to any spherical rotation R∈SO(3):

[0046]

[0047] is the Wigner D-matrix, ensuring that stitching is not affected by overall rotation.

[0048] 2.7 inverse spherical harmonic transform

[0049] The fused frequency domain signal is converted back to the spherical domain:

[0050]

[0051] dynamic parameter adjustment:

[0052] During model training, real-time monitoring of spherical reconstruction error and black border recovery quality, combined with adaptive learning rate scheduling (such as error trend-based learning rate decay / rise) and frequency domain filter weight update strategies, dynamically adjusts the gating fusion coefficients and filter bandwidth parameters in SFNO to balance the convergence speed and stitching accuracy at different training stages.

[0053] 3. loss design and fine-tuning

[0054] In order to improve the training effect and scientific credibility, the application designs a plurality of loss functions, supports external parameter correction, missing data filling, smooth constraint and physical consistency.

[0055] 3.1 Rotation fine-tuning loss

[0056] Allow the network to make small adjustments to the rough external parameters To reduce the difference with the true rotation :

[0057]

[0058] 3.2 Black border recovery loss

[0059] In the overlapping gap area , use the reference image as the truth, and use L1 loss:

[0060]

[0061] 3.3 Smooth regularization

[0062] In order to suppress high-frequency artifacts, spherical gradient regularization is added:

[0063]

[0064] 3.4 Star consistency loss

[0065]

[0066] 3.5 Star trajectory structure constraint

[0067]

[0068] 3.6 Comprehensive loss

[0069] The above items are combined with weights:

[0070]

[0071] 4. Sensor non-uniformity modeling and robustness enhancement

[0072] In order to enhance the adaptability of the system to different sensor physical differences and complex observation conditions, the application designs the following modules:

[0073] 4.1 Photometric normalization correction module

[0074] Uniform standardization processing is performed on different sensor images:

[0075]

[0076] And For the mean value of brightness and standard deviation, As the learnable weight.

[0077] 4.2 Disturbance robust training mechanism

[0078] 4.2.1 Gamma disturbance simulation:

[0079]

[0080] By randomly sampling a gamma value Nonlinearly transform the original image (usually between 0.8 to 1.2), so as to enhance the model's robustness to light changes.

[0081] 4.2.2 Gaussian noise and occlusion simulation

[0082]

[0083]

[0084] Where, represents a Gaussian white noise with mean 0 and variance ; is a binary occlusion mask, randomly generated; is a certain constant (such as the mean value of the image background) used for occlusion area filling; the occlusion ratio can be set to 5%~20%, covering the local area to simulate the missing of stars, instrument occlusion or dead zone.

[0085] 4.3 Adversarial disturbance training

[0086]

[0087] Add the most "deceptive" tiny disturbance to the input image , so that the model output produces the maximum deviation. By minimizing the loss in this worst-case scenario during training, the model's ability to resist unknown input disturbances is enhanced.

[0088] 5. Inverse transform and post-processing

[0089] This module is responsible for projecting the completed spherical image back to a visual image or observation data, and supports various output customization processing.

[0090] 5.1 Signal inverse transform

[0091] After training or inference is completed, the spherical panoramic signal is mapped back to each sensor plane as needed:

[0092]

[0093] Reprojected to the pixel plane Generate seamless compensation images. If the entire celestial sphere needs to be displayed, the compensation image can be directly For spherical panoramic images.

[0094] 5. 2 ROI Cropper

[0095] According to the observation needs, a specific area can be selected:

[0096]

[0097] 5.3 Multi-format output

[0098] Support FITS format (scientific research), PNG / JPEG (display), star table CSV output, etc.

[0099] 5.4 Star point extraction interface

[0100] Provide a DOG+ threshold star point extraction interface to assist target detection and subsequent task automation.

[0101] The method of the present application uses the following evaluation indicators:

[0102] 1. Star point position error (RMSE)

[0103] This method measures the accuracy of the star point position after splicing with the root mean square error (Root Mean Square Error, RMSE) at the pixel level. Let the coordinates of the true star point in the planar image be , and the star point coordinates obtained after splicing and back projection be , then:

[0104]

[0105] wherein is the coordinate of the true star point in the pixel plane, is the position detected after back projection. This indicator reflects the ability of the method in geometric alignment and distortion correction. The lower the value, the more accurate the star point matching, and the smaller the distortion in the polar region and the splicing seam. In practical applications, the RMSE will usually be calculated separately for different declination bands to observe the differences in alignment performance between the polar region and the equatorial region.

[0106] 2. Photometric consistency error (L1 / L2)

[0107] The stitching process not only aligns the geometry, but also ensures smooth transition of image brightness and hue. Lower photometric consistency error means that there is no obvious brightness step or color unevenness after stitching, which can better support subsequent star measurement and light variation analysis. In the context of large-scale stitching, attention is also paid to the photometric mapping consistency between different sensors to reduce the brightness drift caused by sensor differences. In order to evaluate whether the brightness and contrast of the stitched image are smoothly connected, the photometric difference is calculated in the overlapping area or between the reference panorama, and the L1 error and L2 error are used respectively:

[0108]

[0109] For overlapping or evaluation area, For the pixel value of the stitching output, For the pixel value of the true or high-quality reference image. Lower photometric consistency error means that there is no obvious brightness discontinuity or color distortion after stitching.

[0110] 3. Structure fidelity (SSIM) for the filling effect of the "black edge" area, the structure similarity index is used to evaluate the restoration degree of texture and contrast:

[0111]

[0112] The index comprehensively considers local brightness, contrast and structure information, Local mean and variance respectively, Covariance, constant Used to stabilize the calculation. The closer to 1, the more real the structure details of the filled area. Good fidelity not only restores star points, but also preserves the details of background nebula and dark spots, which is particularly important for scientific image processing.

[0113] 4. Smooth artifact detection High-frequency artifacts or discontinuous edges often appear at the multi-view fusion seams. To quantitatively detect high-frequency artifacts or edge discontinuities, the spherical gradient energy can be calculated on the stitching output:

[0114]

[0115] When The value is low and evenly distributed, indicating that the image has smooth brightness transition everywhere, and there is no obvious fault or artifact. When The value is high or concentrated at the seams, indicating that there are mutations in those areas, which need to be further filtered or repaired.

[0116] Compared with the prior art, the beneficial effects of the present application are:

[0117] The method of the present invention fundamentally eliminates the "black edge" gap, tedious calibration and geometric distortion in multi-sensor stitching, improves end-to-end processing efficiency, and can achieve seamless, high-precision large-field celestial sphere panoramic stitching. In actual astronomical surveys and rapid response observation scenarios, the operating efficiency of the algorithm is very important, from image reading, spherical mapping, The time overhead of the entire process, from inference to final output, determines whether near-real-time requirements can be met. Lower processing time not only improves observation efficiency but also provides strong support for rapid response to astronomical events and data processing pipelines. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 It is an overall framework diagram of the embodiment;

[0119] Figure 2 It is a flow chart of the preprocessing and spherical mapping method;

[0120] Figure 3 It is a flow chart of the SFNO fusion and repair method;

[0121] Figure 4 It is a flow chart of the sensor non-uniformity modeling and loss design method;

[0122] Figure 5 It is a flow chart of the image inverse transformation and post-processing method. DETAILED DESCRIPTION

[0123] The present invention will be described in further detail below with reference to the accompanying drawings.

[0124] The overall system framework of the astronomical image splicing processing method based on spherical Fourier neural operator of the present invention is as follows Figure 1 As shown, the method specifically includes the following steps:

[0125] 1. Preprocessing and spherical mapping (such as Figure 2 shown)

[0126] 1.1 Rough calibration

[0127] Combining traditional star point matching algorithms with sensor installation records, we can make a preliminary estimate of the external parameters of each camera. We use the star point features extracted from the captured star image to perform a one-to-one matching with the known star catalog to calculate the camera's attitude information in the spatial coordinate system. These external parameters are usually expressed in the form of rotation matrices or quaternions. The rotation matrix of a sensor can be obtained by matching the star points or combining the geometric position records when it is installed. At the same time, the internal parameters of the camera need to be calibrated, including focal length And the image principal point coordinates And other key values.

[0128] 1.2 Dynamic pose correction

[0129] In actual shooting process, the system is often affected by external vibration or its own attitude drift, resulting in the initial extrinsic parameters of camera or sensor slightly offset. In order to solve this problem, dynamic pose estimation mechanism can be introduced in the calibration process, to track and compensate the small disturbance of the device in real time. The specific method is to input the angular velocity and acceleration information collected by gyroscope or inertial measurement unit (IMU) into extended Kalman filter (EKF), and constantly estimate the attitude disturbance at the current time in the prediction-update cycle of the filter . Then, the disturbance is applied to the initial attitude by the following formula, to realize online correction of the overall attitude:

[0130]

[0131] The rotation correction matrix from the initial attitude to the real-time attitude is reflected by , and the optimal estimation is given by combining the sensor noise model and the kinematics model.

[0132] 1.3 Pixel back-projection

[0133] After completing the camera intrinsic and extrinsic calibration, each planar image can be back-projected to the unit sphere pixel by pixel to obtain the discrete spherical signal. Specifically, first, normalize the pixel plane coordinates (u, v) to the direction vector in the camera coordinate system:

[0134]

[0135] Then, realize the coordinate system transformation by transposing the extrinsic parameter matrix :

[0136]

[0137] Subsequently, calculate the corresponding spherical coordinate angle:

[0138]

[0139] On the equal latitude or longitude grid or mesh, the bilinear or spherical neighbor interpolation is performed on the position of , and the discrete spherical signal is obtained.

[0140] 2. SFNO fusion and repair (as shown in Figure 3 )

[0141] ​​​2.1 Spherical harmonic transform

[0142] After the spherical signal mapping is completed, the discrete spherical data of each channel is projected onto a set of orthogonal spherical harmonic basis functions to obtain the frequency domain coefficients for subsequent filtering and fusion. Specifically, for the i-th channel, its coefficients in the harmonic domain are denoted as which are calculated by:

[0143]

[0144] denotes the harmonic order, ≤ ≤ denotes the intra-order directional component. By multiplying the spherical function with the conjugate basis function and weighting and then integrating the polar and azimuthal angles, the amplitude information of each frequency component is extracted. The final generated coefficient matrix can fully characterize the global spectral structure of the spherical signal.

[0145] 2.2 Cross-modal fusion

[0146] Different wavebands (e.g., visible light, infrared, ultraviolet, etc.) often carry complementary target information and background features. To fully utilize the advantages of multi-band data, this method specially designs a channel attention mechanism in the spherical frequency domain processing: according to the global statistical features and local spectral response of each waveband signal, the attention weight is adaptively allocated, so as to highlight the useful information of high-quality channels, while suppressing the wavebands with more noise or distortion.

[0147] For the coefficient of the i-th channel at the j-th order , first calculate its significance score:

[0148]

[0149] and are the linear mapping weights of the harmonic feature and channel description vector , respectively, is the projection vector,

[0150] is the hyperbolic tangent activation function. Subsequently, the scores of all channels are normalized to obtain the attention weight:

[0151] ​​​​​

[0152] This mechanism can highlight the information of high-quality channels in the frequency domain while suppressing bands with severe noise or distortion, significantly improving the effect of multi-band fusion.

[0153] 2.3 Frequency Domain Filtering

[0154] For different order modes, automatically learns a set of scalar filter weights In the specific filtering operation, the Harmonic coefficient of the channel at this mode Multiply it by the corresponding weight, that is:

[0155]

[0156] Frequency domain filtering can selectively enhance or suppress high-frequency or low-frequency components, thereby removing noise and highlighting signal details.

[0157] 2.4 Point-wise MLP Repair

[0158] After completing the frequency domain filtering, in order to further repair the loss of details caused by noise suppression or interpolation, each fused spherical harmonic coefficient Apply a point Multilayer Perceptron ( Specifically, let the network share layer, then there are:

[0159]

[0160] forward The layers are all transformed with bias linearly. Activation to capture nonlinear features; the last layer only performs linear mapping and outputs the repaired coefficients This coefficient-by-coefficient The restoration strategy can flexibly compensate for the high-frequency details lost during filtering and automatically learn the weights of each layer in end-to-end training. , which can improve the accuracy of spherical signal reconstruction.

[0161] 2.5 Multi-channel Fusion

[0162] After completing the frequency domain filtering and repair of each channel, all The information of each sensor channel is aggregated into a unified global signal. To this end, the present invention adopts a weighted summation multi-channel fusion strategy:

[0163]

[0164] It is The channel is fused in the first order mode. The harmonic coefficients after repair in the first order mode, is the channel fusion weight. The weight is obtained by dynamically learning the characteristics of each channel in all modes of the order, so as to adaptively highlight the sensor output with higher information quality, and suppress the channel with larger noise or distortion.

[0165] 2.6 Rotational equivariance

[0166] The SFNO network strictly guarantees the rotational equivariance of any spherical surface transformation when the input spherical signal undergoes an overall rotation in space , and its transformation result in the spherical harmonic domain satisfies:

[0167]

[0168] is the Wigner matrix, which represents the linear transformation relationship of the harmonic basis function under the action of rotation. This equivariance means that regardless of how the camera or observation coordinates are adjusted, the global rotation is only mapped to the transformation of the frequency domain coefficient, and does not change the processing and fusion results of the network on the signal content.

[0169] 2.7 Inverse Spherical Harmonic Transform

[0170] After completing the frequency domain fusion and repair of all channels, the global harmonic coefficients obtained need to be reconstructed back to the spherical signal in order to generate the final stitched image. This process is called inverse spherical harmonic transform (IHT), and its expression is:

[0171]

[0172] where is the spherical harmonic basis function, is the frequency domain coefficient obtained after fusion in the previous step, represents the order of the harmonic (spatial frequency), is the direction index within the order. By weighting and summing each order and direction basis function according to the corresponding coefficient, the inverse transform can accurately restore the continuous function on the sphere.

[0173] Dynamic parameter adjustment:

[0174] ​During the model training process, the spherical reconstruction error and the black border recovery quality are monitored in real time, and adaptive learning rate scheduling (such as learning rate decay / rise based on error trend) and frequency domain filter weight updating strategy are combined to dynamically adjust the gating fusion coefficient and filter bandwidth parameter in the formula to balance the convergence speed and splicing accuracy at different training stages.

[0175] 3. Loss design and fine-tuning

[0176] By introducing the loss function, the output result can be effectively optimized during the model training process, ensuring that the repaired data not only performs well in geometric alignment, but also maintains consistency in physical properties (such as luminosity, structure, etc.). In addition, to further improve the accuracy and robustness of the model, an external parameter correction mechanism is designed to address the differences between sensors.

[0177] 3.1 Rotation fine-tuning loss

[0178] During actual shooting, device vibration or attitude drift may cause errors in the rotation matrix. To reduce these errors and improve the accuracy of splicing, the invention introduces a rotation fine-tuning loss , which fine-tunes the external parameters of each sensor to ensure that the difference between its rotation matrix and the true value is minimized. The rotation fine-tuning loss function is as follows:

[0179]

[0180] is the predicted rotation matrix, is the fine-tuning matrix, is the true rotation matrix. By optimizing this loss function, the algorithm can more accurately adjust the attitude of each sensor, reducing rotation errors and improving the accuracy of image splicing, ensuring that images at different angles can be seamlessly aligned. The norm is a common matrix error metric that calculates the square root of the sum of the squares of the matrix elements. Here, it calculates the difference between the predicted rotation matrix and the true rotation matrix.

[0181] 3.2 Black border recovery loss

[0182] Even after attitude fine-tuning, due to physical differences or calibration errors between sensors, "black borders" may appear in the overlapping area of the images during splicing. These black borders are usually caused by a lack of light-sensing data or blank areas. To solve this problem, the invention designs a black border recovery loss , which calculates the loss between the spliced image and the reference image to recover these missing areas. The black border recovery loss function is defined as:​

[0183]

[0184] denotes the difference between the stitched image and the reference image, is a binary mask matrix indicating which pixels belong to the missing region (black border), and only the difference of these regions is calculated. The norm is used to measure the difference of the black border region in the image, The loss is more conducive to recovering the missing region in this case.

[0185] 3.3 Smooth regularization

[0186] In the stitching of large-scale astronomical images, unnatural high-frequency artifacts will appear at the stitching seams, which not only affects the image quality, but also affects the subsequent astronomical analysis. Therefore, the smooth regularization loss is introduced to constrain the stitched image to prevent excessive gradient changes and ensure smooth transition in space. The specific loss function is:

[0187]

[0188] Gradient The gradient calculates the local rate of change of the image on the sphere, measuring the degree of change of adjacent pixels at each point in the image. The infinitesimal area on the sphere represents the area of each small region on the sphere. The integral is used to calculate the smoothness of all pixels on the entire sphere.

[0189] 3.4 Star consistency loss

[0190] In the process of astronomical image stitching, it is very important to maintain the consistency of the star magnitude, because the brightness of the star should remain consistent in all images. The invention designs the star consistency loss , which calculates the difference between the stitched image and the true star magnitude, ensuring that the stitched image maintains the correct brightness and luminosity distribution. The loss function is:

[0191]

[0192] is the star magnitude predicted by the model, while is the actual star magnitude value, which can usually be obtained from reference images or known astronomical databases. By optimizing this loss, the luminosity consistency of the stitching result can be guaranteed, so that there is no significant deviation in the star magnitude. This process, combined with smooth regularization, can further improve the overall quality of the image.

[0193] 3.5 Star trajectory construction

[0194] For dynamic astronomical image data, especially time series data, the trajectory prediction and tracking of celestial bodies is crucial. For this purpose, a celestial body trajectory construction loss is introduced, which optimizes the model by calculating the Euclidean distance between the predicted trajectory and the true trajectory, ensuring the consistency of the celestial body position between adjacent frames in time. The loss function is:

[0195]

[0196] is the position of the star predicted by the model, while is the true position of the star at time . By calculating the error between the predicted trajectory and the true trajectory, the model can be optimized to ensure the accuracy of the position of the celestial body in the time series. Minimizing this loss function can improve the accuracy of the trajectory prediction of celestial bodies in dynamic astronomical images, so that the model can better track the movement of celestial bodies.

[0197] 3.6 Comprehensive loss function

[0198] In order to consider all the loss functions comprehensively, the network needs to optimize multiple goals simultaneously during the training process, including rotation fine-tuning, black border recovery, smoothing regularization, star magnitude consistency and trajectory construction, etc. By weighted summation, each loss function is fused into a unified objective function .

[0199]

[0200] is the weight coefficient, used to adjust the contribution of each loss to the total loss. By adjusting the weight coefficient, the influence of each loss can be flexibly balanced, so that the model can be optimized in multiple aspects. Each loss function corresponds to a different problem in image stitching, and the optimization of the comprehensive loss function ensures the overall quality of the stitching result, covering geometric alignment, brightness consistency, motion trajectory and other dimensions.

[0201] 4. Sensor non-uniformity modeling and robustness enhancement (as shown in Figure 4 )

[0202] 4.1 Photometric normalization correction

[0203] In the preprocessing and fusion stage of multi-sensor stitching, the brightness distribution of images taken by each camera often has large differences. In order to eliminate the influence of this brightness non-uniformity, the invention adopts a photometric correction module based on standardization:

[0204]

[0205] The original brightness value collected by the current sensor, and are the mean and standard deviation of the global brightness of the sensor, is a learnable scaling factor. By first subtracting the mean and then dividing by the standard deviation, the images of different sensors can be mapped to a uniform distribution with zero mean and unit variance; then multiplying by This process can effectively reduce the photometric offset between different devices.

[0206] 4.2 Perturbation Robust Training Mechanism

[0207] In order to improve the model's adaptability to complex situations such as illumination changes, noise interference, and occlusion in real observations, the present invention introduces a variety of simulated disturbances in the training process to enhance generalization ability.

[0208] 4.2.1 Gamma Perturbation

[0209] Randomly sample gamma values A nonlinear gamma transform is applied to the normalized image (typically between 0.8 and 1.2). This operation simulates the nonlinear brightness response in low-light or overexposed scenes, enabling the model to maintain stitching accuracy under varying lighting distributions. By incorporating gamma enhancement into the training set, SFNO learns to be insensitive to changes in grayscale curves, thereby improving the robustness of edge fusion.

[0210]

[0211] 4.2.2 Gaussian Noise and Occlusion

[0212] During the training phase, zero-mean, Gaussian white noise is first superimposed on the normalized image:

[0213]

[0214] To simulate the intrinsic noise and quantization error of the sensor; then randomly generate a binary mask ,according to

[0215]

[0216] Replace the 5%–20% region with a constant These two steps not only expand the diversity of training data but also force SFNO to learn to suppress random noise and fill in missing areas during frequency domain fusion.

[0217] 4.3 Adversarial Perturbation Training

[0218] By generating the smallest perturbations that are most "deceptive" to the model , and conduct adversarial training under the following objectives:

[0219]

[0220] This process continuously seeks input perturbations that maximize the output error and allows the model to learn to resist them. By optimizing the model parameters, SFNO not only optimizes black edge recovery and fusion accuracy under normal conditions, but also remains stable in the face of unprecedented extreme noise, sudden illumination changes, or sensor failures, thereby significantly enhancing the overall robustness of the system.

[0221] 4.4 Optimization and parameter adjustment during training

[0222] During the training process of SFNO image stitching, optimization and parameter adjustment are key to ensuring efficient model convergence, reducing overfitting, and maintaining image stitching quality. The following sections describe how to optimize the training process through techniques such as gradient clipping, learning rate scheduling, and optimization algorithm selection, ensuring robustness and efficiency in the face of complex image stitching tasks.

[0223] 4.4.1 Gradient Clipping

[0224] In use When training a model, the training process involves large-scale parameter updates. In this case, gradient explosion will lead to unstable training, which will affect the quality and effect of image stitching. By clipping the gradient, we can ensure that the gradient of each update is within a reasonable range. The operation method is to calculate the L2 norm of the gradient. When the gradient exceeds the set threshold ( ), scale it to the maximum value. Its L2 norm is as follows:

[0225]

[0226] If ‖g‖2 > τ, then scale the gradient g to the threshold τ:

[0227]

[0228] Here, g' is the clipped gradient and τ is the preset gradient threshold. This formula ensures that the magnitude of the gradient does not exceed the specified threshold at each update, thus avoiding the gradient explosion problem.

[0229] 4.4.2 Learning Rate Scheduling

[0230] exist In image stitching, since the number of parameters updated in each training is large, reasonable learning rate scheduling can also help the model converge quickly in the early stage, and fine-tune the parameters in the later stage to avoid overfitting or gradient disappearance. Common scheduling methods include and Learning rate scheduling.

[0231] (a) Learning rate scheduling dynamically reduces the learning rate through a cosine function. Set the initial learning rate as , the maximum learning rate as , the training period as , the current training period as , and the learning rate The calculation formula is as follows:

[0232]

[0233] is the learning rate of the current training period; is the maximum learning rate; is the minimum learning rate (usually set to a small constant value) This method ensures that the learning rate is large at the beginning of training, which can quickly converge, and gradually decreases in the later period, helping the model to find the optimal solution more finely.

[0234] (b) Strategy first increases the learning rate and then decreases the learning rate, the formula is as follows:

[0235]

[0236] Here, indicates that the learning rate is increased in the first half of the training and gradually decreased in the second half. This strategy is widely used to accelerate convergence, especially when the training time is long.

[0237] 4.4.3 Optimization algorithm

[0238] Optimization algorithms can accelerate convergence and reduce training time, especially when dealing with large-scale image stitching tasks. Common optimization algorithms include and . Optimization algorithms adaptively adjust the learning rate of each parameter by calculating the first moment (mean) and second moment (variance) of the gradient. This is particularly important for

[0239] Image stitching models, because the model involves multiple frequency domain features, and these features may have different update speeds at different training stages.

[0240] Calculate the mean and variance of the gradient:

[0241]

[0242] In this formula, and The mean and variance of the gradient, The algorithm uses these two values to adaptively adjust the learning rate of each parameter; and are the decay rates of the first and second moments, usually set to be close to 1, and control the influence of the historical gradient on the current gradient update, more control the influence of the mean, control the influence of the variance.

[0243] Bias correction:

[0244]

[0245] and are the mean and variance after bias correction. Since in the early stage of training, and will be affected by the initialization value (usually initialized to 0), so they need to be bias corrected. But as the training progresses, the effect of bias correction will gradually weaken.

[0246] 4.4.4 Monitoring of the training process and early stopping mechanism

[0247] In image stitching, monitoring of the training process is very important. Through visualization tools (such as TensorBoard, or other visualization tools), you can monitor the loss, accuracy, learning rate, and validation set performance in real time during the training process.

[0248] To avoid model overfitting in image stitching, the present invention uses an early stopping mechanism during training. By monitoring the loss value of the validation set, if the validation loss does not significantly decrease after a certain number of training, the training is stopped in advance.

[0249] If the loss of the validation set has not improved for consecutive times, the training is stopped and the current optimal model is saved.

[0250] 5. Inverse transform and post-processing (such as Figure 5 shown)

[0251] In the image generation stage after stitching and repairing, the network needs to perform post-processing and output on the final stitched image. The present invention introduces inverse transform and other post-processing steps to ensure the final quality of the image, while supporting multiple output formats and custom region selection.

[0252] 5.1 Signal inverse transform

[0253] During training or inference, the spliced ​​spherical signal Convert back to the sensor plane image. This process involves mapping the signal in the spherical coordinate system to the corresponding plane coordinate system. To achieve this, the rotation matrix Back-project the spherical signal to the image plane coordinates (u′, v′).

[0254]

[0255] This inverse transformation step can transform every point on the sphere Mapping back to the sensor plane, forming a compensated image with the same resolution as the original image. If you need to display the entire celestial sphere image, The network can directly generate a panoramic image, ensuring the continuity and integrity of the entire view.

[0256] 5.2 cropping

[0257] In actual applications, users may only focus on specific areas ( In order to meet these requirements, the present invention also designs a A cropping mechanism allows extracting interesting parts from a panoramic image based on a user-specified region. Specifically, area It is determined by two angle intervals, namely:

[0258]

[0259] and The longitude and latitude range is defined by the user. This cropping method can effectively crop specific areas in the image, reducing unnecessary calculations and displays, thereby improving processing efficiency and accuracy.

[0260] 5.3 Multi-format output

[0261] To meet the needs of different fields and users, the system supports multiple output formats, including those commonly used in scientific research. Format for display format, and star catalogs for subsequent analysis By supporting these diverse output formats, the system can meet the different needs of researchers, data analysts, and general users. For example, The format is easily compatible with other astronomical data processing software. The format is suitable for display and sharing. The file can extract and save relevant information of the celestial body for further data processing and analysis.

[0262] 5.4 Star point extraction interface

[0263] For future further automation of astronomical observations and subsequent tasks, the system provides a star point extraction interface. The interface uses a threshold method to extract star points in the image, which can efficiently detect and label the positions of celestial bodies in the stitched image. By providing such an interface, the system not only automatically completes target detection, but also helps users quickly obtain important astronomical data. This automation function saves a lot of manual annotation work in practical applications, improves processing efficiency and ensures data accuracy.

[0264] The method of the present invention can be applied to the following fields:

[0265] 1. Astronomy and astronomical surveys

[0266] Multi-sensor, large-scale observation tasks in astronomy, such as the Large Synoptic Survey Telescope and the European Space Telescope , require large-scale astronomical image data from multiple sensor arrays. These projects often require complex image stitching to integrate observations from different sensors to generate comprehensive, seamless sky images.

[0267] 1.1 Multi-band image stitching

[0268] The present invention can accurately fuse image data from different sensors, solve the problem of geometric distortion and chromatic aberration between bands, and is especially suitable for large-scale astronomical survey projects such as the and , which can provide researchers with high-precision all-sky images and more consistent data for celestial body observation and analysis. 1.2 High-resolution image generation

[0269]

[0270] The technology can generate fine stitched images at high resolution, preserving astronomical details, which is of great significance to astronomical research, especially in the analysis of galaxy, nebula and star structure, and can improve the effectiveness and accuracy of data. 2. Space telescopes and deep space exploration

[0271] With the deployment of advanced telescopes such as the Hubble Space Telescope and the Webb Space Telescope

[0272] , space observation systems are gradually developing towards higher precision and wider bands. In these devices, multi-band sensors are often used for celestial observations, and data processing and stitching face new challenges. ​​

[0273] 2.1 Multi-sensor Fusion

[0274] This method can process data from different sensors, ensure the geometric consistency of imaging in different bands, and reduce stitching errors caused by differences in the field of view between different sensors. Especially in deep space exploration, due to long-term data accumulation and observation differences between different detectors, traditional methods cannot efficiently handle stitching tasks. The application of the algorithm significantly improves the splicing efficiency and accuracy.

[0275] 2.2 Adaptation to low bandwidth and high noise environments

[0276] Deep space probes usually transmit data in a low-bandwidth environment and face interference such as high radiation. The technology can effectively cope with these environmental challenges through efficient frequency domain data processing and neural network restoration, so that high-quality splicing results can still be obtained even under high noise and low bandwidth conditions.

[0277] 3. Satellite Remote Sensing and Earth Observation

[0278] Satellite remote sensing systems, such as (Landsat) and Sentinel satellites continuously observe the Earth's surface using a variety of sensors, acquiring ground-based data at different wavelengths. Accurately stitching data from these different satellites and sensors to create seamless, high-precision images of the Earth's surface remains a major challenge in remote sensing.

[0279] 3.1 Cross-sensor stitching

[0280] The present invention can effectively handle the geometric distortion and photometric inconsistency problems between different satellites and sensors, and is suitable for efficient stitching of remote sensing images. Technology can accurately align observation data from different satellites to ensure the continuity and accuracy of the final stitched image, supporting surface change monitoring and resource management.

[0281] 3.2 Large-scale remote sensing data processing

[0282] Remote sensing data often contains a large number of high-resolution images, and traditional image stitching methods are difficult to process such a huge amount of data. The technology has strong parallel processing capabilities, can efficiently process large-scale data, ensure the timeliness and accuracy of image stitching, and is particularly suitable for long-term observation tasks and large-scale geographic information systems ( ) data processing.

[0283] 4. Scientific research and popular science exhibition

[0284] With the advancement of astronomical observation technology, more and more astronomical data is being recorded and used for scientific research, analysis, and public outreach. Presenting these astronomical images to researchers and the public, especially stitching multiple images into high-quality panoramas, is a crucial task.

[0285] 4.1 High-quality image display

[0286] pass This technology can produce high-quality astronomical panoramas, avoiding the seams common in traditional stitching methods and providing more precise image details. Seamless stitching helps researchers better analyze astronomical data, while also providing the public with a more stunning and intuitive visual experience, sparking their interest in the universe.

[0287] 4.2 Popularization of Science and Education

[0288] Astronomical image stitching technology also plays an important role in popularizing science and education. High-quality stitched images can be widely used in science exhibitions, educational courses, and other fields, helping the public better understand the mysteries of the universe and the basics of astronomy.

[0289] The above embodiments are only typical implementations of the present invention and are not intended to limit the present invention. Any equivalent replacement or improvement made within the scope of the claims of the present invention shall fall within the scope of protection of the present invention.

Claims

1. An astronomical image stitching processing method based on spherical Fourier neural operator, characterized in that: The planar images collected by each sensor are mapped to celestial surface signals, then fused and repaired in parallel in the spherical frequency domain, and finally the fusion results are back-projected back to the desired view to achieve seamless, high-precision large-field-of-view celestial panoramic stitching. The method comprises the following steps: Step 1: Preprocessing and spherical mapping: Obtain the rough pose of each camera using star point matching or installation records, back-project each pixel onto the unit sphere and interpolate to a uniform grid to obtain a discrete spherical signal. Step 2: Fusion repair; input multi-channel spherical signals and their external parameters , first do spherical harmonic transformation, then learn the frequency domain filter weights according to the order mode, with the help of stacked points Complete "black edge" filling, fine-tuning correction and view fusion; Step 3: Loss design and fine-tuning; During the training process, a rotation fine-tuning loss is constructed to automatically correct the rough external parameters, a black edge recovery loss is constructed to guide the network to fill the overlapping missing areas, and a smooth regularization loss is added to suppress high-frequency artifacts. The three are weighted and combined to form an end-to-end joint optimization goal. Achieve the best balance between geometric alignment, data completion, and visual smoothing; Step 4: Modeling sensor inconsistencies and enhancing robustness. First, brightness normalize each sensor image. Add gamma transform, Gaussian noise, and random occlusion to training to simulate illumination, noise, and defects. Adversarial perturbation training is used to improve the model's resilience to unknown, subtle changes. Step 5: Inverse transformation and post-processing: Perform inverse harmonic transformation on the fused spherical image to obtain a continuous and seamless celestial sphere distribution, and then crop or back-project it back to each observation plane as needed to generate the final compensated image or panoramic image.

2. The astronomical image stitching processing method based on spherical Fourier neural operator according to claim 1 is characterized in that: Step 1 includes the following steps: Step 1-1: Rough calibration: Use traditional star point matching or sensor installation records to obtain the rough external parameters of each camera. For each sensor, its rotation matrix and intrinsic parameters are obtained through traditional star point matching or installation records; Step 1-2: Dynamic attitude correction: introduce a dynamic attitude estimation mechanism, fuse gyroscope or IMU data, calculate the perturbation term through extended Kalman filtering, and make real-time corrections; Steps 1-3: pixel backprojection; According to the external parameters, each plane image is back-projected pixel by pixel onto the unit sphere, and Interpolation is performed on the grid to generate discrete spherical signals in preparation for subsequent frequency domain processing.

3. The astronomical image stitching processing method based on spherical Fourier neural operator according to claim 1, characterized in that: Step 2 includes the following steps: Step 2-1: Spherical harmonic transform; decompose each channel into spherical harmonic basis functions; Step 2-2: Cross-modal fusion: Considering multi-band observation information, introducing a channel attention mechanism, performing weight calculation and fusion output; Step 2-3: Frequency domain filtering: For each mode, the network learns a set of scalar weights and performs filtering. Step 2-4: Point Repair; Apply a fully connected network to each fusion coefficient for repair; Step 2-5: Multi-channel fusion: fuse all sensor channels into a global signal in the frequency domain; Step 2-6: Rotational equivariance; using SFNO to ensure rotational equivariance for any spherical surface; Step 2-7: Inverse spherical harmonic transform; convert the fused frequency domain signal back to the spherical domain.

4. The astronomical image stitching processing method based on spherical Fourier neural operator according to claim 1, characterized in that: Step 2 includes dynamic parameter adjustment. During the model training process, the spherical reconstruction error and black edge restoration quality are monitored in real time. Combined with the adaptive learning rate scheduling and frequency domain filter weight update strategy, the gated fusion coefficient and filter bandwidth parameters in SFNO are dynamically adjusted to maintain a balance between convergence speed and splicing accuracy at different training stages.

5. The astronomical image stitching processing method based on spherical Fourier neural operator according to claim 1, characterized in that: Step 3 includes the following steps: Step 3-1: Obtain rotation fine-tuning loss; allow the network to make small adjustments to the coarse extrinsic parameters to reduce the difference from the true rotation; Step 3-2: Obtain black edge restoration loss; on the overlapping gap area, use the reference image as the ground truth and adopt L1 loss; Step 3-3: Perform smoothing regularization; to suppress high-frequency artifacts, add spherical gradient regularization; Step 3-4: Obtain magnitude consistency loss; Step 3-5: Perform star trajectory structure constraints; Step 3-6: Calculate the comprehensive loss.

6. The astronomical image stitching processing method based on spherical Fourier neural operator according to claim 1, characterized in that: Step 4 includes the following steps: Step 4-1: Photometric normalization correction; perform unified standardization on images from different sensors; Step 4-2: Perturbation robust training; including gamma perturbation simulation and Gaussian noise and occlusion simulation; Step 4-3: Adversarial perturbation training; add the most deceptive tiny perturbations to the input image to cause the largest deviation in the model output.

7. The astronomical image stitching processing method based on spherical Fourier neural operator according to claim 1, characterized in that: Step 5 includes the following steps: Step 5-1: Signal inverse transformation: After training or inference is completed, the spherical panoramic signal is mapped back to each sensor plane as needed, and then projected onto the pixel plane to generate a seamless compensated image. Step 5-2: Select a specific area for ROI cropping based on observation requirements; Step 5-3: Output in multiple formats; Step 5-4: Provide the DOG+ threshold star point extraction interface to assist in target detection and subsequent task automation.

8. The astronomical image stitching processing method based on spherical Fourier neural operator according to claim 1, characterized in that: Evaluation indicators including star point position error, photometric consistency error, structural fidelity, and smoothing artifact detection are used to evaluate the stitching accuracy of the proposed method.

Images