Astronomical image stitching method and device based on deep learning, equipment and medium

Abstract

The application relates to the technical field of picture processing, and discloses an astronomical image splicing method, device and equipment based on deep learning and a medium, the method comprising the following steps: performing key point detection on a pretreated original image based on an XFeat neural network, and acquiring pixel coordinates of corresponding key points; wherein the original image comprises a first image and a second image; performing feature extraction based on the pixel coordinates, and outputting a key point feature descriptor vector; matching each key point feature descriptor vector in the first image with the nearest neighbor in the second image, generating a coarse matching matching candidate set, and performing offset prediction based on the coarse matching matching candidate set; optimizing and adjusting the matching result based on the predicted offset, obtaining a fine matching key point pair set; calculating a homography matrix based on the key point pair set; and performing image transformation and fusion based on the homography matrix. The application can accurately extract and match key points of astronomical images, and improves splicing quality and effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of picture processing, in particular to an astronomical image splicing method and device based on deep learning, equipment and medium. BACKGROUND

[0002] In astronomical research, large field of view astronomical images have important application value. However, due to the size limitation of the field of view of astronomical cameras and the fisheye effect of wide-angle cameras, direct shooting of large field of view astronomical images faces many challenges. In order to solve this problem, image splicing technology emerges as the times require. This technology splices multiple small field of view pictures with overlapping parts into a large field of view panoramic astronomical image through registration, transformation and fusion, so as to facilitate celestial body analysis and research.

[0003] The basic principle of the image splicing algorithm mainly includes key point extraction, key point matching, homography matrix calculation, image transformation and fusion and the like. In these steps, the extraction and matching of image key points are crucial, which provides a basis for subsequent homography matrix calculation and directly affects the final splicing quality. However, traditional key point extraction methods such as SIFT (Scale-Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), BRISK (Binary Robust Invariant Scalable Keypoints) and the like, as well as key point matching methods such as BFMatcher (Brute-Force Matcher), KNN (K-Nearest Neighbors) and the like, but due to the small changes in shape, color and texture and the like of astronomical images, the number of feature points detected by these traditional methods is not large, and it is difficult to fully capture important information in the image. When feature matching, a certain amount of false matching will occur, which affects the continuity and consistency of image splicing, and thus ultimately affects the effect and quality of astronomical splicing.

[0004] Therefore, the number of feature points detected by the image splicing algorithm in the prior art is not large, and false matching is prone to occur during feature matching, which is a problem to be solved. SUMMARY

[0005] The main purpose of the present application is to provide an astronomical image splicing method and device based on deep learning, equipment and medium, which aims to solve the technical problem that the number of feature points detected by the image splicing algorithm in the prior art is not large, and false matching is prone to occur during feature matching.

[0006] The first aspect of the present application provides an astronomical image splicing method based on deep learning, comprising:

[0007] input the preprocessed original image into a key point detection module of XFeat for key point detection; the key point detection comprises detecting image points with preset key features; wherein the original image comprises a first image and a second image;

[0008] output a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility of the corresponding position being a key point;

[0009] extract key point positions from the key point heat map and map them to the size of the corresponding original image to obtain the pixel coordinates of the corresponding key points in the original image;

[0010] perform feature extraction based on the pixel coordinates and output a key point feature descriptor vector;

[0011] use a nearest neighbor search algorithm to match the nearest neighbors of each key point feature descriptor vector in the first image in the second image to generate a coarsely matched matching candidate set, wherein the matching candidate set comprises multiple pairs of potential matching key points;

[0012] input the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learn and predict the corresponding offset based on a preset model;

[0013] optimize and adjust the matching result based on the predicted offset to obtain a set of finely matched key point pairs;

[0014] calculate a homography matrix based on the set of key point pairs;

[0015] perform image transformation and fusion based on the homography matrix.

[0016] Further, the step of performing feature extraction based on the pixel coordinates and outputting a key point feature descriptor vector comprises:

[0017] determine the position of the key point in the image according to the pixel coordinates;

[0018] extract a local feature descriptor of the key point within a preset range of the position, which is used to reflect the local feature information within the preset range of the key point;

[0019] process and integrate the extracted local feature descriptor to output a 64-dimensional key point feature descriptor vector.

[0020] Further, the step of optimizing and adjusting the matching result based on the predicted offset to obtain a set of finely matched key point pairs comprises:

[0021] adjust the positions of each pair of potential matching key points in the coarsely matched matching candidate set according to the predicted offset.

[0022] After adjusting the positions of the key point pairs, the matching degree between the adjusted key point pairs is evaluated based on a preset similarity metric index;

[0023] When the matching degree exceeds a preset threshold, the corresponding key point pair is considered to be a precise match and is included in a precise matching key point pair set.

[0024] Further, the step of calculating a homography matrix based on the key point pair set comprises:

[0025] Extracting key point coordinate information in the matching successful key point pair set, the key point coordinate information including corresponding key point coordinates in the first image and the second image;

[0026] Calculating a homography matrix based on the key point coordinate information, the homography matrix being used to describe the mapping relationship between the first image and the second image, the mapping relationship including perspective transformation, affine transformation or combined relationship;

[0027] Based on a preset evaluation index, the calculated homography matrix is verified and optimized to obtain the final homography matrix.

[0028] Further, the step of performing image transformation and fusion based on the homography matrix comprises:

[0029] Performing corresponding image transformation on the first image or the second image using the homography matrix, so that the first image and the second image are aligned to the same perspective;

[0030] Image stitching of the first image and the second image under the same perspective;

[0031] Image fusion of the stitched first image and second image for smooth transition to complete image stitching.

[0032] Further, before the step of key point detection by the key point detection module of the XFeat neural network after the preprocessed original image is input, the key point detection comprises:

[0033] Preprocessing the original image, the preprocessing including grayscale operation to convert the original image from 3 channels to a single channel.

[0034] The second aspect of the present application proposes an astronomical image stitching device based on deep learning, comprising:

[0035] An image input module for inputting a preprocessed original image to a key point detection module of an XFeat neural network for key point detection, the key point detection including detecting image points with preset key features.

[0036] a heat map output module configured to output a key point heat map based on the detection result, wherein a value of each pixel in the key point heat map represents a possibility that a corresponding position is a key point;

[0037] a position extraction module configured to extract key point positions from the key point heat map and map the key point positions to a size of a corresponding original image to obtain pixel coordinates of corresponding key points in the original image;

[0038] an extraction module configured to perform feature extraction based on the pixel coordinates and output a key point feature descriptor vector;

[0039] a matching module configured to match, by using a nearest neighbor search algorithm, each key point feature descriptor vector in the first image to a nearest neighbor in the second image to generate a coarse matching candidate set containing multiple pairs of potential matching key points;

[0040] a prediction module configured to take, as input, a nearest neighbor pair of each key point pair on an original coarse-grained feature map, learn and predict a corresponding offset based on a preset model;

[0041] an optimization module configured to optimize and adjust a matching result based on the predicted offset to obtain a fine matching key point pair set;

[0042] a calculation module configured to calculate a homography matrix based on the key point pair set;

[0043] a fusion module configured to perform image transformation and fusion based on the homography matrix.

[0044] A third aspect of the present application provides a computer device, comprising:

[0045] a processor;

[0046] a memory configured to store a computer program;

[0047] When the processor executes the computer program, the above-mentioned any one of the astronomical image stitching methods based on deep learning is implemented.

[0048] A fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned any one of the astronomical image stitching methods based on deep learning is implemented.

[0049] Advantages:

[0050] By performing key point detection on the pre-processed first image and the second image, and obtaining the pixel coordinates of the corresponding key points, the key information in the image can be accurately located, providing a basis for subsequent processing. The nearest neighbor of each key point feature descriptor vector in the first image in the second image is matched to obtain a coarse matching result, then model learning is performed based on the coarse matching result to realize offset prediction, and the matching result is optimized based on the offset prediction result to obtain a set of precisely matched key point pairs, which not only improves the fitting effect, but also improves the matching accuracy through secondary matching. The homography matrix is calculated based on the set of key point pairs, and image transformation and fusion are performed based on the homography matrix, which can make the image stitching more accurate and natural. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a step schematic diagram of an astronomical image stitching method based on deep learning according to an embodiment of the present application;

[0052] Figure 2 is a structural schematic block diagram of an astronomical image stitching device based on deep learning according to an embodiment of the present application;

[0053] Figure 3 is a structural schematic block diagram of a computer device according to an embodiment of the present application;

[0054] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0055] In order to make the object, technical scheme and advantages of the present application more clear, the present application will be further described in detail below with reference to the embodiments and the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0056] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements, modules and / or components exist, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components and / or their combinations. It should be understood that when an element is said to be "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any module and all combinations of the associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0058] Reference Figure 1 This application provides a deep learning-based astronomical image stitching method, including the following steps: S1, inputting the preprocessed original image into the XFeat key point detection module for key point detection; the key point detection includes detecting image points with preset key features;

[0059] S2. Output a key point heatmap based on the detection results, wherein the value of each pixel in the key point heatmap represents the probability that the corresponding position is a key point;

[0060] S3. Extract the key point locations from the key point heatmap and map them to the corresponding dimensions of the original image to obtain the pixel coordinates of the corresponding key points in the original image.

[0061] In this step, first, the original image is preprocessed, wherein the original image includes a first image and a second image; the preprocessing includes performing a grayscale operation to convert it from 3 channels to a single channel. This can effectively reduce the data volume of the image, improve the processing speed, and at the same time preserve the structural information of the image. Then, the preprocessed original image is input into the XFeat network for key point detection, which will detect key points with significance and representativeness, which usually correspond to corner points, edge points, intersection points, etc. in the image. Based on the detection result, a key point heat map is output. The value of each pixel in the heat map represents the likelihood that the position is a key point. The key point positions are extracted from the key point heat map and mapped to the size of the original image to obtain the pixel coordinates of the corresponding key points in the original image. In this process, an accurate algorithm is needed to ensure that the extracted key point positions are accurate and can be accurately mapped to the size of the original image. Specifically, when the preprocessed original image is input into the key point detection module of XFeat, the module will use its trained model and algorithm to detect image points with preset key features. These key features may include the sharpness of corner points, the continuity of edges, etc. Based on the detection result, a key point heat map is output. The value of each pixel in the heat map represents the likelihood that the corresponding position is a key point, and the higher the likelihood, the larger the pixel value. When extracting key point positions from the key point heat map, a specific algorithm is used to determine which pixel positions correspond to a likelihood that exceeds a certain threshold, thereby determining these positions as key point positions. For example, through the Non-Maximum Suppression (NMS) algorithm, for each pixel point in the heat map, it is judged whether there are other pixel points in its neighborhood with higher likelihood. If there are, the current pixel point is suppressed, i.e. it is not considered as a key point; otherwise, it is considered as a key point. Specifically, first, a likelihood threshold is set, for example, 0.5. Then, for each pixel point in the heat map with a likelihood higher than the threshold, check the likelihood of other pixel points in its 8-neighborhood (or larger neighborhood). If there are other pixel points with a higher likelihood than the current pixel point, suppress the current pixel point; otherwise, determine the current pixel point as a key point position. Through this method, the key point positions can be accurately extracted from the key point heat map, thereby providing an accurate basis for subsequent processing.

[0062] Then, the key point positions are mapped to the size of the original image through a mapping algorithm to obtain the pixel coordinates of the corresponding key points in the original image. This ensures that the obtained pixel coordinates correspond to the actual size of the original image, providing an accurate basis for subsequent processing.

[0063] S4, feature extraction based on the pixel coordinates, output key point feature descriptor vector;

[0064] According to the obtained pixel coordinates, the specific position of the key point in the image is determined. Within a preset range at this position, the local feature descriptors of the key point are extracted. These descriptors are used to reflect the local feature information in the preset range of the key point, such as color, texture, shape, etc. The extracted local feature descriptors are processed and integrated, and a 64-dimensional key point feature descriptor vector is output through the XFeat network. This vector can more accurately represent the features of the key point, providing a more reliable basis for subsequent matching and stitching, and improving the representativeness and robustness of the features.

[0065] S5, using a nearest neighbor search algorithm, matching each key point feature descriptor vector in the first image with its nearest neighbor in the second image, generating a coarse matching candidate set containing multiple pairs of potential matching key points;

[0066] S6, taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learning and predicting the corresponding offset based on a preset model;

[0067] Using a nearest neighbor search algorithm, each key point feature descriptor vector in the first image can be quickly and effectively matched with its nearest neighbor in the second image. By calculating the similarity between the feature descriptor vectors, the most similar nearest neighbor is found, thereby generating a matching candidate set containing multiple pairs of potential matching key points. Taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, a model is learned based on the preset model. This model, after training, can predict the corresponding offset according to the input nearest neighbor pair information. This model can be adjusted and optimized based on XFeat, or it can be an independent model, and its training goal is to accurately predict the corresponding offset according to the input nearest neighbor pair information. The prediction of the offset can further refine the matching result, taking into account the possible minor deformations and differences in the image, improving the accuracy and reliability of the matching. In this way, the corresponding relationship of the key points in the first image and the second image can be more accurately found, providing a more accurate basis for subsequent image stitching. After obtaining the key point feature descriptor vector, the feature matching stage is entered, which includes two stages: initial feature matching and matching optimization and screening. In the initial feature matching stage, a nearest neighbor search algorithm is used to find the nearest neighbor of each feature descriptor in another set of images, thereby generating a matching candidate set, i.e., the process of coarse matching (initial matching), which contains multiple pairs of potential matching key points.

[0068] S7, based on the predicted offset, the matching result is optimized and adjusted to obtain a set of fine-matched key point pairs;

[0069] However, due to the possibility of large errors in the initial matching, the feature matching stage of XEeat also includes a matching optimization or screening step, which further refines the matching results by learning to predict pixel-level offsets. This step refines the coarse matching results by considering the nearest neighbor pairs of each keypoint pair on the original coarse feature map as input, resulting in more accurate and reliable matching results. By adopting a two-level feature matching mode, further learning of pixel-level offsets is performed based on coarse matching, effectively ensuring the accuracy of feature matching. Based on the offset prediction results, the previous matching results are optimized. By removing some mis-matched keypoint pairs, a more accurate set of matching successful keypoint pairs is obtained. In this process, the XFeat network will use specific algorithms and models to evaluate the accuracy of the matching results and make corresponding adjustments and optimizations based on the evaluation results.

[0070] S8, computing a homography matrix based on the set of keypoint pairs;

[0071] Extracting keypoint coordinate information from the set of matching successful keypoint pairs, including corresponding keypoint coordinates in the first image and the second image. Based on these keypoint coordinate information, a homography matrix is calculated. The homography matrix is used to describe the mapping relationship between the first image and the second image, which can be perspective transformation, affine transformation or their combination. Based on the preset evaluation index, the calculated homography matrix is verified and optimized. By comparing the calculated homography matrix with the preset standard or expected result, its accuracy and reliability are judged. If there is a problem or deviation, appropriate adjustments and optimizations are made to obtain the final homography matrix.

[0072] S9, image transformation and fusion based on the homography matrix;

[0073] Using the homography matrix to perform corresponding image transformation on the first image or the second image, so that they can be aligned to the same perspective, preparing for subsequent image stitching. The first image and the second image under the same perspective are stitched. In this process, appropriate algorithms need to be selected to ensure accurate image stitching and avoid obvious stitching marks or misalignment. The stitched first image and second image are fused for smooth transition. Common image fusion methods include multi-band fusion, Poisson fusion, etc., which can make the two images look more natural at the seam, reduce the stitching marks, and complete high-quality image stitching.

[0074] In an embodiment, the step of extracting features based on the pixel coordinates and outputting keypoint feature descriptor vectors includes:

[0075] S20, determining the position of the key point in the image according to the pixel coordinates;

[0076] S21, extracting a local feature descriptor of the key point within a preset range of the position, for reflecting local feature information within the preset range of the key point;

[0077] S22, processing and integrating the extracted local feature descriptor to output a 64-dimensional key point feature descriptor vector.

[0078] In this embodiment, according to the obtained pixel coordinates, the specific position of the key point in the image can be accurately determined. Within a preset range of the position, a local feature descriptor of the key point is extracted by using a specific algorithm and model. Including using SIFT (Scale-Invariant Feature Transform) algorithm: by detecting key points at different scales and calculating the gradient direction histogram around the key points to describe local features. SURF (Speeded Up Robust Features) algorithm: based on Hessian matrix to detect key points, and using integral image to accelerate the calculation of features. HOG (Histogram of Oriented Gradients) algorithm: by calculating the gradient direction histogram of the region around the key point to describe the local feature. These local feature descriptors can fully reflect the local feature information within the preset range of the key point, such as the change of color, the trend of texture, the characteristics of shape, etc. Next, the extracted local feature descriptor is processed and integrated. This process may include normalizing, screening, combining and other operations on the feature descriptor to remove redundant information and noise and highlight key features. Through these processing and integration, a 64-dimensional key point feature descriptor vector is finally output. This vector can more accurately and comprehensively represent the features of the key point, providing a reliable basis for subsequent key point matching and image stitching. The 64-dimensional vector has high expression ability and can more accurately describe the unique properties of the key point, thereby improving the accuracy and quality of image stitching.

[0079] In an embodiment, the step of optimizing and adjusting the matching results based on the predicted offset for each pair of potential matching key point pairs in the coarse matching matching candidate set to obtain a set of fine matching key point pairs includes:

[0080] S30, adjusting their positions according to the predicted offset;

[0081] S31, after adjusting the positions of the key point pairs, evaluating the matching degree between the adjusted key point pairs based on a preset similarity measure index;

[0082] S32, when the matching degree exceeds the preset threshold, the corresponding key point pair is considered to be a fine matching key point pair and is included in the fine matching key point pair set.

[0083] In the present embodiment, the nearest neighbor pairs of each key point pair on the original coarse-grained feature map have been learned by the preset model, and the corresponding offsets are obtained. These offsets represent the amount of adjustment needed for the key point pairs in position. For each pair of potential matching key point pairs in the coarse matching result, their positions are fine-tuned according to the predicted offsets. Specifically, the coordinates of the key point pairs are added to the corresponding offsets to achieve the adjustment of the positions. Further, after adjusting the positions of the key point pairs, the matching similarity between them is recalculated. Some similarity measure indicators, such as Euclidean distance, cosine similarity, etc., can be used to evaluate the matching degree between the adjusted key point pairs. According to the re-evaluated matching similarity, a threshold is set to screen out the fine matching key point pairs. Only when the matching similarity exceeds the threshold, the key point pair is considered to be a fine matching key point pair and is included in the fine matching key point pair set. Multiple iterations can be performed to continuously adjust the offsets and screen the key point pairs to gradually optimize the fine matching results. In each iteration, the offsets can be updated according to the results of the previous iteration, or some optimization algorithms can be used to improve the accuracy of matching. Through the above steps, the coarse matching results are fine-tuned using the predicted offsets, and a more accurate fine matching key point pair set can be obtained, thereby improving the quality and effect of astronomical image stitching.

[0084] In an embodiment, the step of optimizing and adjusting the matching results based on the predicted offsets to obtain a fine matching key point pair set comprises:

[0085] S40, adjusting and screening the initial matching results according to the offset prediction results;

[0086] S41, removing the mis-matching key point pairs and retaining the accurately matching key point pairs to obtain a fine matching key point pair set.

[0087] In this embodiment, according to the offset prediction result, the initial matching result obtained by the nearest neighbor search algorithm is further adjusted and screened. The matching of the potential key point pairs in the initial matching result, that is, the matching candidate set of the coarse matching, is analyzed by analyzing the distribution and size of the offset, to determine which key point pairs are likely to have matching errors. Then, the mis-matched key point pairs are removed, which may be caused by noise, similar local features and other factors in the image. Through strict screening, the accurately matched key point pairs are retained, thereby obtaining the set of precisely matched key point pairs. The key point pairs in this set have higher accuracy and reliability, and can better reflect the true correspondence between the first image and the second image, providing a more reliable basis for subsequent image processing.

[0088] In an embodiment, the step of calculating a homography matrix based on the set of key point pairs comprises:

[0089] S50, extracting key point coordinate information in the set of successfully matched key point pairs, the key point coordinate information comprising corresponding key point coordinates in the first image and the second image;

[0090] S51, calculating a homography matrix based on the key point coordinate information, for describing the mapping relationship between the first image and the second image, the mapping relationship comprising perspective transformation, affine transformation or combined relationship;

[0091] S52, verifying and optimizing the calculated homography matrix based on a preset evaluation index, to obtain the final homography matrix.

[0092] In this embodiment, first, the coordinate information of the key points is extracted from the set of successfully matched key point pairs. These coordinate information includes the specific coordinate positions of the corresponding key points in the first image and the second image. Based on these key point coordinate information, a specific mathematical algorithm is used to calculate the homography matrix. The homography matrix can describe the mapping relationship between the first image and the second image, which can include perspective transformation, affine transformation or their combination. Perspective transformation can simulate the change of camera angle, and affine transformation can process translation, rotation and scaling of the image, etc. After the homography matrix is calculated, it is verified and optimized based on the preset evaluation index. The evaluation index may include the determinant, eigenvalue, etc. of the matrix, for judging the rationality and accuracy of the matrix. Through adjustment and optimization of the matrix, an accurate and reliable homography matrix is finally obtained, providing accurate guidance for image transformation and stitching.

[0093] In an embodiment, the step of performing image transformation and fusion based on the homography matrix comprises:

[0094] S60, performing corresponding image transformation on the first image or the second image by using the homography matrix, so that the first image and the second image are aligned to the same perspective;

[0095] S61, image stitching of the first image and the second image under the same perspective;

[0096] S62, image fusion of the stitched first image and second image for smooth transition, completing image stitching.

[0097] In this embodiment, the calculated homography matrix is used to perform corresponding image transformation on the first image or the second image. Through this transformation, the first image and the second image can be aligned to the same perspective, ensuring their consistency in spatial position. In the same perspective, the first image and the second image are stitched. In this process, the accuracy and seamlessness of the stitching need to be ensured to avoid obvious stitching marks or misplacement. In order to make the stitched image more natural and smooth, the stitched first image and second image are fused. Image fusion can use various methods, such as multi-band fusion, Poisson fusion, etc. Through these methods, the differences between the two images at the joint can be smoothed, reducing the stitching marks and making the image look more coherent and natural. After completing image fusion, the entire image stitching process is completed, and a panoramic astronomical image with a large field of view is obtained.

[0098] In an embodiment, the pre-processed original image is input to the key point detection module of the XFeat neural network for key point detection; before the step of detecting the image points with the preset key features, it includes:

[0099] S00, pre-processing the original image, the pre-processing including performing a grayscale operation to convert the original image from 3 channels to a single channel.

[0100] In this embodiment, before performing key point detection on the original image, it is first pre-processed. The main operation of pre-processing is to perform grayscale, converting the original image from 3 channels to a single channel. The advantage of this is that it can effectively reduce the data volume of the image and improve the speed of subsequent processing. At the same time, the grayscale operation can preserve the structural information of the image without losing the key content of the image. By converting the image to a single channel, the subsequent key point detection and feature extraction operations can be more efficient, laying the foundation for the entire astronomical image stitching process.

[0101] Reference Figure 2 An astronomical image stitching device based on deep learning, comprising:

[0102] The image input module 100 is configured to input the preprocessed original image to a key point detection module of the XFeat neural network for key point detection; the key point detection includes detecting image points with preset key features; wherein the original image includes a first image and a second image.

[0103] The heat map output module 200 is configured to output a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility of the corresponding position being a key point.

[0104] The position extraction module 300 is configured to extract key point positions from the key point heat map and map them to the size of the corresponding original image to obtain pixel coordinates of the corresponding key points in the original image.

[0105] The extraction module 400 is configured to perform feature extraction based on the pixel coordinates and output a key point feature descriptor vector.

[0106] The matching module 500 is configured to use a nearest neighbor search algorithm to match the nearest neighbor of each key point feature descriptor vector in the first image in the second image, to generate a coarse matching matching candidate set, and the matching candidate set includes multiple pairs of potential matching key point pairs.

[0107] The prediction module 600 is configured to take the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, and learn and predict the corresponding offset based on a preset model.

[0108] The optimization module 700 is configured to optimize and adjust the matching result based on the predicted offset to obtain a fine matching key point pair set.

[0109] The calculation module 800 is configured to calculate a homography matrix based on the key point pair set.

[0110] The fusion module 900 is configured to perform image transformation and fusion based on the homography matrix.

[0111] Further, the above-mentioned extraction module 400 includes:

[0112] The position determination unit is configured to determine the position of the key point in the image according to the pixel coordinates.

[0113] The feature extraction unit is configured to extract a key point local feature descriptor within a preset range of the position, which is used to reflect the local feature information within the preset range of the key point.

[0114] The integration output unit is configured to process and integrate the extracted local feature descriptor to output a 64-dimensional key point feature descriptor vector.

[0115] Further, the above-mentioned optimization module 700 includes:

[0116] an adjusting unit configured to adjust the positions of each pair of potential matching key points in the set of matching candidates of the coarse matching according to the predicted offset;

[0117] an evaluating unit configured to evaluate the matching degree between the adjusted key points after adjusting the positions of the key points based on a preset similarity metric;

[0118] an incorporating unit configured to incorporate the corresponding key points into a set of fine matching key points when the matching degree exceeds a preset threshold.

[0119] Further, the above-mentioned computing module 800 comprises:

[0120] a coordinate extracting unit configured to extract key point coordinate information in the set of matching key points, the key point coordinate information comprising corresponding key point coordinates in the first image and the second image;

[0121] a matrix calculating unit configured to calculate a homography matrix based on the key point coordinate information, the homography matrix being used to describe a mapping relationship between the first image and the second image, the mapping relationship comprising a perspective transformation, an affine transformation or a combination thereof;

[0122] a verifying and optimizing unit configured to verify and optimize the calculated homography matrix based on a preset evaluation index to obtain a final homography matrix.

[0123] Further, the above-mentioned fusion module 900 comprises:

[0124] an image transforming unit configured to perform corresponding image transformation on the first image or the second image using the homography matrix to align the first image and the second image to the same perspective;

[0125] an image splicing unit configured to splice the first image and the second image under the same perspective;

[0126] an image fusing unit configured to fuse the spliced first image and second image to achieve smooth transition and complete image splicing.

[0127] Further, the above-mentioned device further comprises a preprocessing module 010 configured to pre-process the original image, the preprocessing comprising a grayscale operation to convert the original image from 3 channels to a single channel.

[0128] Referring to Figure 3 In the embodiments of the present application, a computer device is also provided, which can be a server, and the internal structure thereof can be as shown in Figure 3The computer device includes a processor, a memory, a network interface and a database connected through a bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores operations, computer programs and databases. The internal memory provides an environment for the operation and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as the deep learning-based astronomical image stitching method. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a deep learning-based astronomical image stitching method, including the steps of: inputting a preprocessed original image into a key point detection module of an XFeat neural network for key point detection; wherein the original image includes a first image and a second image; the key point detection includes detecting image points with preset key features; outputting a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility of the corresponding position being a key point; extracting the key point position from the key point heat map and mapping it to the size of the corresponding original image to obtain the pixel coordinates of the corresponding key points in the original image; performing feature extraction based on the pixel coordinates to output a key point feature descriptor vector; using a nearest neighbor search algorithm to match each key point feature descriptor vector in the first image with its nearest neighbor in the second image to generate a coarse matching matching candidate set, the matching candidate set contains multiple pairs of potential matching key points; taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learning and predicting the corresponding offset based on a preset model; based on the predicted offset, the matching result is optimized and adjusted to obtain a fine matching key point pair set; based on the key point pair set, a homography matrix is calculated; based on the homography matrix, image transformation and fusion are performed.

[0129] An embodiment of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement a deep learning-based astronomical image stitching method, including the steps of: inputting a preprocessed original image into a key point detection module of an XFeat neural network to perform key point detection; wherein the original image includes a first image and a second image; the key point detection includes detecting image points with preset key features; outputting a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility of the corresponding position being a key point; extracting key point positions from the key point heat map and mapping them to the corresponding original image size to obtain the pixel coordinates of the corresponding key points in the original image; performing feature extraction based on the pixel coordinates to output a key point feature descriptor vector; using a nearest neighbor search algorithm to match the nearest neighbor of each key point feature descriptor vector in the first image in the second image to generate a coarsely matched matching candidate set, the matching candidate set including multiple pairs of potential matching key points; taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learning and predicting the corresponding offset based on a preset model; optimizing and adjusting the matching result based on the predicted offset to obtain a set of precisely matched key point pairs; calculating a homography matrix based on the set of key point pairs; and performing image transformation and fusion based on the homography matrix.

[0130] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium provided by the present application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM) and the like.

[0131] The above merely describes the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made according to the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. A deep learning-based astronomical image stitching method, characterized in that, include: The preprocessed original image is input into the keypoint detection module of the XFeat neural network for keypoint detection. The key point detection includes detecting image points with preset key features; The original image mentioned above includes a first image and a second image; Based on the detection results, a key point heatmap is output, wherein the value of each pixel in the key point heatmap represents the probability that the corresponding location is a key point; Extract the key point locations from the key point heatmap and map them to the corresponding dimensions of the original image to obtain the pixel coordinates of the corresponding key points in the original image; Based on the pixel coordinates, feature extraction is performed, and a key point feature descriptor vector is output. Using the nearest neighbor search algorithm, the feature descriptor vectors of each key point in the first image are matched with the nearest neighbors in the second image to generate a coarse matching candidate set, which contains multiple pairs of potentially matching key point pairs; Using the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, the corresponding offset is learned and predicted based on the preset model. The matching results are optimized and adjusted based on the predicted offset to obtain a set of finely matched key point pairs; Calculate the homography matrix based on the set of key points; Image transformation and fusion based on homography matrix.

2. The astronomical image stitching method based on deep learning according to claim 1, characterized in that, The step of extracting features based on the pixel coordinates and outputting keypoint feature descriptor vectors includes: Based on the pixel coordinates, determine the location of the key points in the image; Within a preset range at the location, local feature descriptors of key points are extracted to reflect local feature information within the preset range of key points; The extracted local feature descriptors are processed and integrated to output a 64-dimensional keypoint feature descriptor vector.

3. The astronomical image stitching method based on deep learning according to claim 1, characterized in that, The step of optimizing and adjusting the matching results based on the predicted offset to obtain a set of finely matched keypoint pairs includes: For each pair of keypoints in the coarse matching candidate set that is a potential match, their positions are adjusted according to the predicted offset. After adjusting the positions of the keypoint pairs, the matching degree between the adjusted keypoint pairs is evaluated based on the preset similarity metric. When the matching degree exceeds a preset threshold, the corresponding key point pair is considered to be finely matched and included in the set of finely matched key point pairs.

4. The astronomical image stitching method based on deep learning according to claim 1, characterized in that, The step of calculating the homography matrix based on the set of keypoint pairs includes: Extract the key point coordinate information from the set of successfully matched key point pairs, wherein the key point coordinate information includes the corresponding key point coordinates in the first image and the second image; Based on the key point coordinate information, a homography matrix is ​​calculated to describe the mapping relationship between the first image and the second image. The mapping relationship includes perspective transformation, affine transformation, or combination relationship. Based on preset evaluation indicators, the calculated homography matrix is ​​verified and optimized to obtain the final homography matrix.

5. The astronomical image stitching method based on deep learning according to claim 1, characterized in that, The steps for image transformation and fusion based on the homography matrix include: The first or second image is transformed using a homography matrix to align the first and second images to the same viewpoint. The first and second images from the same viewpoint are stitched together. The first and second images are then merged to create a smooth transition, completing the image stitching process.

6. The astronomical image stitching method based on deep learning according to claim 1, characterized in that, The preprocessed original image is input into the key point detection module of the XFeat neural network for key point detection. Prior to the step of detecting image points with preset key features, the key point detection includes: The original image is preprocessed, including grayscale conversion, which converts the original image from 3 channels to a single channel.

7. A deep learning-based astronomical image stitching device, characterized in that, include: The image input module is used to input the pre-processed raw image into the XFeat neural network's keypoint detection module for keypoint detection. The key point detection includes detecting image points with preset key features; The original image mentioned above includes a first image and a second image; The heatmap output module is used to output a key point heatmap based on the detection results, wherein the value of each pixel in the key point heatmap represents the probability that the corresponding position is a key point; The location extraction module is used to extract the location of key points from the key point heatmap and map it to the corresponding size of the original image to obtain the pixel coordinates of the corresponding key points in the original image. The extraction module is used to extract features based on the pixel coordinates and output key point feature descriptor vectors. The matching module is used to use the nearest neighbor search algorithm to match the nearest neighbors of the feature descriptor vectors of each key point in the first image in the second image, and generate a coarse matching candidate set, which contains multiple pairs of potential matching key point pairs. The prediction module is used to learn and predict the corresponding offset based on a preset model, taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input. The optimization module is used to optimize and adjust the matching results based on the predicted offset to obtain a set of finely matched keypoint pairs. The calculation module is used to calculate the homography matrix based on the set of key point pairs; The fusion module is used for image transformation and fusion based on the homography matrix.

8. The astronomical image stitching device based on deep learning according to claim 7, characterized in that, The extraction module includes: A position determination unit is used to determine the position of key points in the image based on the pixel coordinates; The feature extraction unit is used to extract local feature descriptors of key points within a preset range at the location, which reflect the local feature information of the key points within the preset range. The integrated output unit processes and integrates the extracted local feature descriptors, outputting a 64-dimensional keypoint feature descriptor vector.

9. A computer device, characterized in that, include: processor; Memory, used to store computer programs; The processor executes the computer program to implement the deep learning-based astronomical image stitching method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the deep learning-based astronomical image stitching method according to any one of claims 1 to 6.

Images