Astronomical image splicing method and device based on deep learning, equipment and medium

Abstract

The invention relates to the technical field of picture processing, and discloses an astronomical image splicing method and device based on deep learning, equipment and a medium, and the method comprises the steps: carrying out the key point detection of a preprocessed original image based on an XFat neural network, and obtaining the pixel coordinates of a corresponding key point; wherein the original image comprises a first image and a second image; feature extraction is carried out based on the pixel coordinates, and key point feature descriptor vectors are output; matching the nearest neighbor of each key point feature descriptor vector in the first image in the second image, generating a matching candidate set of rough matching, and performing offset prediction based on the matching candidate set of rough matching; optimizing and adjusting the matching result based on the predicted offset to obtain a finely matched key point pair set; calculating a homography matrix based on the key point pair set; and carrying out image transformation and fusion based on the homography matrix. The key points of the astronomical image can be accurately extracted and matched, and the splicing quality and effect are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a method, device, equipment and medium for astronomical image stitching based on deep learning. Background Art

[0002] In astronomical research, astronomical images with a large field of view have important application value. However, due to the field size limitation of astronomical cameras and the fisheye effect of wide-angle cameras, directly shooting astronomical images with a large field of view faces many challenges. In order to solve this problem, image stitching technology came into being. This technology stitches multiple small field of view images with overlapping parts into a large field of view panoramic astronomical image through registration, transformation and fusion, so as to facilitate celestial body analysis and research.

[0003] The basic principles of image stitching algorithms mainly include key point extraction, key point matching, homography matrix calculation, image transformation and fusion. Among these steps, the extraction and matching of image key points are crucial, which provide the basis for the subsequent homography matrix calculation and directly affect the final stitching quality. However, traditional key point extraction methods such as SIFT (Scale-Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), BRISK (Binary Robust Invariant Scalable Keypoints), and key point matching methods such as BFMatcher (Brute-ForceMatcher) and KNN (K-Nearest Neighbors) have a small number of astronomical image feature points detected by these traditional methods due to the small changes in shape, color and texture of astronomical images. It is difficult to fully capture the important information in the image. When matching features, some mismatches will occur, which will affect the continuity and consistency of image stitching, and ultimately affect the effect and quality of astronomical stitching.

[0004] Therefore, the number of feature points detected by the image stitching algorithm in the prior art is small, and mismatching is prone to occur during feature matching, which is an urgent problem to be solved. Summary of the invention

[0005] The main purpose of this application is to provide an astronomical image stitching method, device, equipment and medium based on deep learning, aiming to solve the technical problem in the prior art that the number of feature points detected by the image stitching algorithm is small and mismatching is easy to occur during feature matching.

[0006] The first aspect of the present application proposes an astronomical image stitching method based on deep learning, comprising:

[0007] Inputting the preprocessed original image into the key point detection module of XFeat to perform key point detection; the key point detection includes detecting image points with preset key features; wherein the original image includes a first image and a second image;

[0008] Output a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point;

[0009] Extract the key point positions from the key point heat map and map them to the size of the corresponding original image to obtain the pixel coordinates of the corresponding key points in the original image;

[0010] Perform feature extraction based on the pixel coordinates and output a key point feature descriptor vector;

[0011] Using a nearest neighbor search algorithm, matching the nearest neighbors of the feature descriptor vectors of each key point in the first image in the second image, and generating a rough matching candidate set, wherein the matching candidate set includes a plurality of potential matching key point pairs;

[0012] Taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learn and predict the corresponding offset based on the preset model;

[0013] The matching results are optimized and adjusted based on the predicted offset to obtain a set of key point pairs with precise matching;

[0014] Calculating a homography matrix based on the set of key point pairs;

[0015] Image transformation and fusion based on homography matrix.

[0016] Furthermore, the step of performing feature extraction based on the pixel coordinates and outputting a key point feature descriptor vector comprises:

[0017] Determine the position of the key point in the image according to the pixel coordinates;

[0018] Extracting a local feature descriptor of the key point within a preset range of the position, so as to reflect the local feature information within the preset range of the key point;

[0019] The extracted local feature descriptors are processed and integrated to output a 64-dimensional key point feature descriptor vector.

[0020] Furthermore, the step of optimizing and adjusting the matching result based on the predicted offset to obtain a set of key point pairs of precise matching includes:

[0021] For each pair of potential matching key point pairs in the matching candidate set of the rough matching, adjusting their positions according to the predicted offset;

[0022] After adjusting the positions of the key point pairs, the matching degree between the adjusted key point pairs is evaluated based on a preset similarity metric;

[0023] When the degree of matching exceeds a preset threshold, the corresponding key point pair is considered to be an exact match and is included in the set of exact matching key point pairs.

[0024] Furthermore, the step of calculating the homography matrix based on the key point pair set includes:

[0025] Extracting key point coordinate information from the set of successfully matched key point pairs, wherein the key point coordinate information includes corresponding key point coordinates in the first image and the second image;

[0026] Calculating a homography matrix based on the key point coordinate information to describe a mapping relationship between the first image and the second image, wherein the mapping relationship includes a perspective transformation, an affine transformation, or a combination relationship;

[0027] Based on the preset evaluation index, the calculated homography matrix is ​​verified and optimized to obtain the final homography matrix.

[0028] Furthermore, the step of performing image transformation and fusion based on the homography matrix includes:

[0029] Performing a corresponding image transformation on the first image or the second image using a homography matrix so that the first image and the second image are aligned to the same viewing angle;

[0030] Performing image stitching on the first image and the second image at the same viewing angle;

[0031] The stitched first image and the second image are fused for smooth transition to complete the image stitching.

[0032] Furthermore, the step of inputting the preprocessed original image into the key point detection module of the XFeat neural network for key point detection; the key point detection includes the step of detecting image points with preset key features, including:

[0033] The original image is preprocessed, and the preprocessing includes graying the original image from three channels to a single channel.

[0034] The second aspect of the present application proposes an astronomical image stitching device based on deep learning, comprising:

[0035] An image input module is used to input the preprocessed original image into the key point detection module of the XFeat neural network for key point detection; the key point detection includes detecting image points with preset key features;

[0036] A heat map output module, used to output a key point heat map based on the detection results, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point;

[0037] A position extraction module is used to extract the key point positions from the key point heat map and map them to the size of the corresponding original image to obtain the pixel coordinates of the corresponding key points in the original image;

[0038] An extraction module, used for performing feature extraction based on the pixel coordinates and outputting a key point feature descriptor vector;

[0039] A matching module, used to match the nearest neighbors of the feature descriptor vectors of each key point in the first image in the second image using a nearest neighbor search algorithm, and generate a rough matching candidate set, wherein the matching candidate set includes a plurality of potential matching key point pairs;

[0040] The prediction module is used to take the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learn and predict the corresponding offset based on the preset model;

[0041] An optimization module is used to optimize and adjust the matching results based on the predicted offset to obtain a set of key point pairs with precise matching;

[0042] A calculation module, used for calculating a homography matrix based on the set of key point pairs;

[0043] Fusion module, used for image transformation and fusion based on homography matrix.

[0044] A third aspect of the present application provides a computer device, comprising:

[0045] processor;

[0046] Memory for storing computer programs;

[0047] Wherein, when the processor executes the computer program, it implements any of the above-mentioned deep learning-based astronomical image stitching methods.

[0048] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the deep learning-based astronomical image stitching method described in any one of the above items is implemented.

[0049] Beneficial effects:

[0050] By performing key point detection on the preprocessed first image and the second image and obtaining the pixel coordinates of the corresponding key points, the key information in the image can be accurately located, providing a basis for subsequent processing. The feature descriptor vectors of each key point in the first image are matched with the nearest neighbors in the second image to obtain a rough matching result, and then model learning is performed based on the rough matching result to achieve offset prediction. The matching result is optimized based on the offset prediction result to obtain a set of key point pairs with precise matching, which not only improves the fitting effect, but also the secondary matching can improve the matching accuracy. The homography matrix is ​​calculated based on the key point pair set, and the image transformation and fusion are performed based on the homography matrix, which can make the image stitching more accurate and natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the steps of an astronomical image stitching method based on deep learning according to an embodiment of the present application;

[0052] Figure 2 It is a structural schematic block diagram of an astronomical image stitching device based on deep learning according to an embodiment of the present application;

[0053] Figure 3 is a schematic block diagram of the structure of a computer device according to an embodiment of the present application;

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It will be appreciated by those skilled in the art that, unless expressly stated, the singular forms "one", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when an element is said to be "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may also be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0058] Reference Figure 1 , the embodiment of the present application provides an astronomical image stitching method based on deep learning, comprising the following steps: S1, inputting the preprocessed original image into the key point detection module of XFeat to perform key point detection; the key point detection includes detecting image points with preset key features;

[0059] S2. Output a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point;

[0060] S3, extracting the key point positions from the key point heat map, and mapping them to the size of the corresponding original image, to obtain the pixel coordinates of the corresponding key points in the original image;

[0061] In this step, first, the original image is preprocessed, wherein the original image includes a first image and a second image; the preprocessing includes a grayscale operation to convert it from a three-channel to a single channel. This can effectively reduce the amount of image data, improve processing speed, and retain the structural information of the image. Then, the preprocessed original image is input into the XFeat network for key point detection, and significant and representative key points are detected, which usually correspond to corner points, edge points, intersection points, etc. in the image. Based on the detection results, a key point heat map is output. The value of each pixel in the heat map indicates the possibility that the position is a key point. The key point position is extracted from the key point heat map and mapped to the original image size, so as to obtain the pixel coordinates of the corresponding key point in the original image. In this process, an accurate algorithm is needed to ensure that the extracted key point position is accurate and can be accurately mapped to the size of the original image. Specifically, when the preprocessed original image is input into the key point detection module of XFeat, the module uses its trained model and algorithm to detect image points with preset key features. These key features may include the sharpness of corner points, the continuity of edges, etc. Based on the detection results, a key point heat map is output. The value of each pixel in the heat map indicates the possibility that the corresponding position is a key point. The higher the possibility, the larger the pixel value. When extracting key point positions from the key point heat map, a specific algorithm is used to determine which pixel positions have a corresponding possibility exceeding a certain threshold, so as to determine these positions as key point positions. For example, through the non-maximum suppression (NMS) algorithm, for each pixel in the heat map, it is determined whether there are other pixels with higher possibilities in its neighborhood. If so, the current pixel is suppressed, that is, it is not regarded as a key point; otherwise, it is regarded as a key point. Specifically, a possibility threshold is first set, such as 0.5. Then, for each pixel in the heat map whose possibility is higher than the threshold, the possibility of other pixels in its 8-neighborhood (or larger neighborhood) is checked. If there are other pixels with a higher possibility than the current pixel, the current pixel is suppressed; otherwise, the current pixel is determined as a key point position. In this way, the key point position can be accurately extracted from the key point heat map, thereby providing an accurate basis for subsequent processing.

[0062] Then, these key point positions are mapped to the original image size through a mapping algorithm to obtain the pixel coordinates of the corresponding key points in the original image. This ensures that the obtained pixel coordinates correspond to the actual size of the original image, providing an accurate basis for subsequent processing.

[0063] S4, extracting features based on the pixel coordinates and outputting a key point feature descriptor vector;

[0064] According to the obtained pixel coordinates, the specific position of the key point in the image is determined. The local feature descriptors of the key point are extracted within the preset range of the position. These descriptors are used to reflect the local feature information within the preset range of the key point, such as color, texture, shape, etc. The extracted local feature descriptors are processed and integrated, and a 64-dimensional key point feature descriptor vector is output through the XFeat network. This vector can more accurately represent the features of the key points, provide a more reliable basis for subsequent matching and splicing, and improve the representativeness and robustness of the features.

[0065] S5, using a nearest neighbor search algorithm, matching the nearest neighbors of the feature descriptor vectors of each key point in the first image in the second image, and generating a rough matching candidate set, wherein the matching candidate set includes multiple pairs of potential matching key point pairs;

[0066] S6, taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learning and predicting the corresponding offset based on the preset model;

[0067] Using the nearest neighbor search algorithm, the nearest neighbors of the feature descriptor vectors of each key point in the first image can be matched quickly and effectively in the second image. By calculating the similarity between the feature descriptor vectors, the most similar nearest neighbors are found, thereby generating a matching candidate set containing multiple pairs of key point pairs with potential matches. The nearest neighbor pairs of each key point pair on the original coarse-grained feature map are used as input and learning is performed based on a preset model. This model is trained to predict the corresponding offset based on the input nearest neighbor pair information. This model can be adjusted and optimized based on XFeat, or it can be an independent model, and its training goal is to accurately predict the corresponding offset based on the input nearest neighbor pair information. The prediction of the offset can further refine the matching results, taking into account the possible slight deformations and differences in the image, and improving the accuracy and reliability of the matching. In this way, the correspondence between the key points in the first image and the second image can be found more accurately, providing a more accurate basis for subsequent image stitching. After obtaining the key point feature descriptor vector, the feature matching stage begins. This stage includes two stages: initial feature matching and matching optimization and screening. In the initial feature matching stage, the nearest neighbor search algorithm is used to find the nearest neighbor of each feature descriptor in another set of images, thereby generating a matching candidate set, which is the rough matching (initial matching) process, which contains multiple pairs of potential matching key point pairs.

[0068] S7, optimizing and adjusting the matching result based on the predicted offset to obtain a set of key point pairs with precise matching;

[0069] However, since there may be large mismatches in the initial matching, the feature matching stage of XFeat also includes a matching optimization or screening step, which further refines the matching results by learning to predict pixel-level offsets. This step fine-tunes the coarse matching results by considering the nearest neighbor pairs of each key point pair on the original coarse-grained feature map as input, thereby obtaining more accurate and reliable matching results. By adopting a two-level feature matching mode, on the basis of coarse matching, further learning to predict pixel-level offsets effectively ensures the accuracy of feature matching. According to the offset prediction results, the previous matching results are optimized. By removing some mismatched key point pairs, a more accurate set of key point pairs that are successfully matched is obtained. In this process, specific algorithms and models are used based on the XFeat network to evaluate the accuracy of the matching results, and corresponding adjustments and optimizations are made based on the evaluation results.

[0070] S8, calculating a homography matrix based on the key point pair set;

[0071] Extract the key point coordinate information from the set of successfully matched key point pairs, which includes the corresponding key point coordinates in the first image and the second image. Based on these key point coordinate information, calculate the homography matrix. The homography matrix is ​​used to describe the mapping relationship between the first image and the second image, which can be a perspective transformation, an affine transformation, or a combination thereof. Based on the preset evaluation indicators, verify and optimize the calculated homography matrix. By comparing the calculated homography matrix with the preset standard or expected results, its accuracy and reliability are judged. If problems or deviations are found, corresponding adjustments and optimizations will be made to obtain the final homography matrix.

[0072] S9, image transformation and fusion based on homography matrix;

[0073] Use the homography matrix to perform corresponding image transformation on the first image or the second image so that they can be aligned to the same perspective, preparing for subsequent image stitching. Stitch the first image and the second image at the same perspective. In this process, it is necessary to select a suitable algorithm to ensure that the image stitching is accurate and avoid obvious stitching marks or misalignment. Fuse the stitched first image and the second image for a smooth transition. Commonly used image fusion methods include multi-band fusion, Poisson fusion, etc. These methods can make the two images look more natural at the seams, reduce stitching marks, and complete high-quality image stitching.

[0074] In one embodiment, the step of performing feature extraction based on the pixel coordinates and outputting a key point feature descriptor vector comprises:

[0075] S20, determining the position of the key point in the image according to the pixel coordinates;

[0076] S21, extracting a key point local feature descriptor within a preset range of the position, so as to reflect local feature information within the preset range of the key point;

[0077] S22, processing and integrating the extracted local feature descriptors, and outputting a 64-dimensional key point feature descriptor vector.

[0078] In this embodiment, according to the obtained pixel coordinates, the specific position of the key point in the image can be accurately determined. Within the preset range of the position, the local feature descriptor of the key point is extracted by using a specific algorithm and model. Including the use of SIFT (Scale-Invariant Feature Transform) algorithm: by detecting key points at different scales, and calculating the gradient direction histogram around the key point to describe the local features. SURF (Speeded Up Robust Features) algorithm: based on the Hessian matrix to detect key points, and using integral images to accelerate the calculation of features. HOG (Histogram of Oriented Gradients) algorithm: by calculating the gradient direction histogram of the area around the key point to describe the local features. These local feature descriptors can fully reflect the local feature information within the preset range of the key point, such as color changes, texture trends, shape characteristics, etc. Next, the extracted local feature descriptors are processed and integrated. This process may include normalizing, screening, combining and other operations on the feature descriptors to remove redundant information and noise and highlight the key features. Through these processes and integrations, a 64-dimensional key point feature descriptor vector is finally output. This vector can more accurately and comprehensively represent the characteristics of the key points, providing a reliable basis for subsequent key point matching and image stitching. The 64-dimensional vector has a high expressive power and can more accurately describe the unique properties of the key points, thereby improving the accuracy and quality of image stitching.

[0079] In one embodiment, the step of optimizing and adjusting the matching result based on the predicted offset for each potential matching key point pair in the matching candidate set of the coarse matching to obtain a set of key point pairs of precise matching includes:

[0080] S30, adjusting their positions according to the predicted offset;

[0081] S31, after adjusting the positions of the key point pairs, evaluating the matching degree between the adjusted key point pairs based on a preset similarity measurement index;

[0082] S32: When the matching degree exceeds a preset threshold, the corresponding key point pair is considered to be an exact match and is included in a set of exact match key point pairs.

[0083] In this embodiment, the nearest neighbor pair of each key point pair on the original coarse-grained feature map has been learned through a preset model to obtain the corresponding offset. These offsets represent the amount by which the key point pair needs to be adjusted in position. For each pair of potential matching key point pairs in the coarse matching result, their positions are fine-tuned according to the predicted offsets. Specifically, the coordinates of the key point pair are added with the corresponding offset to achieve the position adjustment. Further, after adjusting the position of the key point pair, the matching similarity between them is recalculated. A certain similarity metric, such as Euclidean distance, cosine similarity, etc., can be used to evaluate the matching degree between the adjusted key point pairs. According to the re-evaluated matching similarity, a threshold is set to filter out the key point pairs of precise matching. Only when the matching similarity exceeds the threshold, the key point pair is considered to be a precise match and is included in the set of key point pairs of precise matching. Multiple iterations can be performed to continuously adjust the offset and filter the key point pairs to gradually optimize the results of the precise matching. In each iteration, the offset can be updated according to the previous result, or some optimization algorithms can be used to improve the accuracy of the matching. Through the above steps, the predicted offset is used to fine-tune the coarse matching results, and a more accurate set of precise matching key point pairs can be obtained, thereby improving the quality and effect of astronomical image stitching.

[0084] In one embodiment, the step of optimizing and adjusting the matching result based on the predicted offset to obtain a set of key point pairs of precise matching includes:

[0085] S40, adjusting and screening the initial matching results according to the offset prediction results;

[0086] S41. Remove the incorrectly matched key point pairs, retain the accurately matched key point pairs, and obtain a set of precisely matched key point pairs.

[0087] In this embodiment, the initial matching results previously obtained by the nearest neighbor search algorithm are further adjusted and screened according to the offset prediction results. The initial matching results, i.e., the matching conditions of potential key point pairs in the matching candidate set of the rough match, are determined by analyzing the distribution and size of the offset to determine which key point pairs may have errors in matching. Then, those key point pairs that are mismatched are removed, which may be caused by factors such as noise in the image and similar local features. Through strict screening, the key point pairs that are accurately matched are retained, thereby obtaining a set of key point pairs that are precisely matched. The key point pairs in this set have higher accuracy and reliability, can better reflect the true correspondence between the first image and the second image, and provide a more reliable basis for subsequent image processing.

[0088] In one embodiment, the step of calculating the homography matrix based on the set of key point pairs includes:

[0089] S50, extracting key point coordinate information from the set of successfully matched key point pairs, wherein the key point coordinate information includes corresponding key point coordinates in the first image and the second image;

[0090] S51, calculating a homography matrix based on the key point coordinate information, which is used to describe a mapping relationship between the first image and the second image, wherein the mapping relationship includes a perspective transformation, an affine transformation, or a combination relationship;

[0091] S52: Based on preset evaluation indicators, verify and optimize the calculated homography matrix to obtain the final homography matrix.

[0092] In this embodiment, first, the coordinate information of the key points is extracted from the set of key point pairs that are successfully matched. These coordinate information include the specific coordinate positions of the corresponding key points in the first image and the second image. Based on these key point coordinate information, a specific mathematical algorithm is used to calculate the homography matrix. The homography matrix can describe the mapping relationship between the first image and the second image, and this mapping relationship can include perspective transformation, affine transformation or a combination thereof. Perspective transformation can simulate the change of the camera's viewing angle, and affine transformation can handle the translation, rotation and scaling of the image. After the homography matrix is ​​calculated, it is verified and optimized based on preset evaluation indicators. The evaluation indicators may include the determinant, eigenvalue, etc. of the matrix, which are used to judge the rationality and accuracy of the matrix. By adjusting and optimizing the matrix, an accurate and reliable homography matrix is ​​finally obtained, providing precise guidance for the transformation and splicing of the image.

[0093] In one embodiment, the step of performing image transformation and fusion based on the homography matrix includes:

[0094] S60, performing corresponding image transformation on the first image or the second image using a homography matrix, so that the first image and the second image are aligned to the same viewing angle;

[0095] S61, stitching the first image and the second image at the same viewing angle;

[0096] S62: Fusing the stitched first image and the second image for smooth transition to complete image stitching.

[0097] In this embodiment, the calculated homography matrix is ​​used to perform a corresponding image transformation on the first image or the second image. Through this transformation, the first image and the second image can be aligned to the same viewing angle to ensure that they are consistent in spatial position. At the same viewing angle, the first image and the second image are spliced. In this process, it is necessary to ensure the accuracy and seamlessness of the splicing to avoid obvious splicing marks or dislocation. In order to make the spliced ​​image more natural and smooth, the spliced ​​first image and the second image are fused. Image fusion can adopt a variety of methods, such as multi-band fusion, Poisson fusion, etc., through which the difference between the two images at the seam can be smoothly transitioned, the splicing marks can be reduced, and the image looks more coherent and natural. After the image fusion is completed, the entire image splicing process is completed, and a panoramic astronomical image with a large field of view is obtained.

[0098] In one embodiment, the step of inputting the preprocessed original image into the key point detection module of the XFeat neural network for key point detection; the key point detection includes the step of detecting image points with preset key features, including:

[0099] S00, preprocessing the original image, wherein the preprocessing includes performing a grayscale operation to convert the original image from three channels to a single channel.

[0100] In this embodiment, before the key point detection is performed on the original image, it is first preprocessed. The main operation of the preprocessing is to grayscale the original image from the original three channels to a single channel. The advantage of this is that the amount of image data can be effectively reduced and the speed of subsequent processing can be improved. At the same time, the grayscale operation can retain the structural information of the image without causing loss to the key content of the image. By converting the image to a single channel, subsequent key point detection and feature extraction operations can be performed more efficiently, laying the foundation for the entire astronomical image stitching process.

[0101] refer to Figure 2 , an astronomical image stitching device based on deep learning, comprising:

[0102] The image input module 100 is used to input the preprocessed original image into the key point detection module of the XFeat neural network for key point detection; the key point detection includes detecting image points with preset key features; wherein the original image includes a first image and a second image;

[0103] A heat map output module 200 is used to output a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point;

[0104] The position extraction module 300 is used to extract the key point positions from the key point heat map and map them to the size of the corresponding original image to obtain the pixel coordinates of the corresponding key points in the original image;

[0105] An extraction module 400, configured to perform feature extraction based on the pixel coordinates and output a key point feature descriptor vector;

[0106] A matching module 500 is used to match the nearest neighbors of the feature descriptor vectors of each key point in the first image in the second image using a nearest neighbor search algorithm to generate a rough matching candidate set, wherein the matching candidate set includes multiple pairs of potential matching key point pairs;

[0107] A prediction module 600 is used to take the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learn and predict the corresponding offset based on a preset model;

[0108] An optimization module 700 is used to optimize and adjust the matching result based on the predicted offset to obtain a set of key point pairs of precise matching;

[0109] A calculation module 800, configured to calculate a homography matrix based on the key point pair set;

[0110] The fusion module 900 is used to perform image transformation and fusion based on the homography matrix.

[0111] Furthermore, the extraction module 400 includes:

[0112] A position determination unit, used to determine the position of the key point in the image according to the pixel coordinates;

[0113] A feature extraction unit, used to extract a local feature descriptor of a key point within a preset range of the position, so as to reflect the local feature information within the preset range of the key point;

[0114] The integrated output unit is used to process and integrate the extracted local feature descriptors and output a 64-dimensional key point feature descriptor vector.

[0115] Furthermore, the optimization module 700 includes:

[0116] An adjustment unit, configured to adjust the positions of each pair of potential matching key points in the matching candidate set of the rough matching according to the predicted offset;

[0117] An evaluation unit, configured to evaluate the degree of matching between the adjusted key point pairs based on a preset similarity measurement index after adjusting the positions of the key point pairs;

[0118] The inclusion unit is used to consider the corresponding key point pair as an exact match and include it in the set of key point pairs of exact matches when the matching degree exceeds a preset threshold.

[0119] Furthermore, the calculation module 800 includes:

[0120] A coordinate extraction unit, used to extract key point coordinate information from a set of successfully matched key point pairs, wherein the key point coordinate information includes corresponding key point coordinates in the first image and the second image;

[0121] A matrix calculation unit, used for calculating a homography matrix based on the key point coordinate information, for describing a mapping relationship between the first image and the second image, wherein the mapping relationship includes a perspective transformation, an affine transformation or a combination relationship;

[0122] The verification and optimization unit is used to verify and optimize the calculated homography matrix based on preset evaluation indicators to obtain the final homography matrix.

[0123] Furthermore, the above fusion module 900 includes:

[0124] An image transformation unit, used for performing corresponding image transformation on the first image or the second image using a homography matrix, so that the first image and the second image are aligned to the same viewing angle;

[0125] An image stitching unit, used for stitching the first image and the second image at the same viewing angle;

[0126] The image fusion unit is used to fuse the stitched first image and the second image for smooth transition to complete the image stitching.

[0127] Furthermore, the above-mentioned device also includes: a preprocessing module 010, which is used to preprocess the original image, and the preprocessing includes performing a grayscale operation to convert the original image from three channels to a single channel.

[0128] Reference Figure 3 In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected by a bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores operations, computer programs and databases. The memory provides an environment for the operation of the non-volatile storage medium and the operation of the computer program. The database of the computer device is used to store data such as an astronomical image stitching method based on deep learning. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a deep learning-based astronomical image stitching method is implemented, including the steps of: inputting the pre-processed original image into the key point detection module of the XFeat neural network for key point detection; wherein the original image includes a first image and a second image; the key point detection includes detecting image points with preset key features; outputting a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point; extracting the key point position from the key point heat map, and mapping it to the size of the corresponding original image, and obtaining the pixel coordinates of the corresponding key point in the original image. ; Perform feature extraction based on the pixel coordinates and output a key point feature descriptor vector; use a nearest neighbor search algorithm to match the nearest neighbors of each key point feature descriptor vector in the first image in the second image to generate a coarse matching candidate set, wherein the matching candidate set includes multiple pairs of potential matching key point pairs; use the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learn and predict the corresponding offset based on a preset model; optimize and adjust the matching results based on the predicted offset to obtain a precisely matched key point pair set; calculate a homography matrix based on the key point pair set; perform image transformation and fusion based on the homography matrix.

[0129] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a deep learning-based astronomical image stitching method is implemented, comprising the steps of: inputting a preprocessed original image into a key point detection module of an XFeat neural network for key point detection; wherein the original image comprises a first image and a second image; the key point detection comprises detecting image points having preset key features; outputting a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point; extracting the key point position from the key point heat map and mapping it to the corresponding original image size to obtain the original image; The invention relates to a method for obtaining pixel coordinates of corresponding key points in an original image; extracting features based on the pixel coordinates and outputting key point feature descriptor vectors; matching the nearest neighbors of the key point feature descriptor vectors in the first image in the second image by using a nearest neighbor search algorithm to generate a coarse matching candidate set, wherein the matching candidate set includes multiple pairs of potential matching key point pairs; taking the nearest neighbor pairs of each key point pair on the original coarse-grained feature map as input, learning and predicting the corresponding offsets based on a preset model; optimizing and adjusting the matching results based on the predicted offsets to obtain a precisely matched key point pair set; calculating a homography matrix based on the key point pair set; and performing image transformation and fusion based on the homography matrix.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in the present application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0131] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A deep learning-based astronomical image stitching method, characterized in that: include: The preprocessed original image is input into the key point detection module of the XFeat neural network for key point detection; The key point detection includes detecting image points having preset key features; The original image includes a first image and a second image; Output a key point heat map based on the detection result, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point; Extract the key point positions from the key point heat map and map them to the size of the corresponding original image to obtain the pixel coordinates of the corresponding key points in the original image; Perform feature extraction based on the pixel coordinates and output a key point feature descriptor vector; Using a nearest neighbor search algorithm, matching the nearest neighbors of the feature descriptor vectors of each key point in the first image in the second image, and generating a rough matching candidate set, wherein the matching candidate set includes a plurality of potential matching key point pairs; Taking the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learn and predict the corresponding offset based on the preset model; The matching results are optimized and adjusted based on the predicted offset to obtain a set of key point pairs with precise matching; Calculating a homography matrix based on the set of key point pairs; Image transformation and fusion based on homography matrix.

2. The astronomical image stitching method based on deep learning according to claim 1, characterized in that: The step of extracting features based on the pixel coordinates and outputting a key point feature descriptor vector comprises: Determine the position of the key point in the image according to the pixel coordinates; Extracting a local feature descriptor of the key point within a preset range of the position, so as to reflect the local feature information within the preset range of the key point; The extracted local feature descriptors are processed and integrated to output a 64-dimensional key point feature descriptor vector.

3. The astronomical image stitching method based on deep learning according to claim 1, characterized in that: The step of optimizing and adjusting the matching result based on the predicted offset to obtain a set of key point pairs with precise matching includes: For each pair of potential matching key point pairs in the matching candidate set of the rough matching, adjusting their positions according to the predicted offset; After adjusting the positions of the key point pairs, the matching degree between the adjusted key point pairs is evaluated based on a preset similarity metric; When the degree of matching exceeds a preset threshold, the corresponding key point pair is considered to be an exact match and is included in the set of exact matching key point pairs.

4. The astronomical image stitching method based on deep learning according to claim 1, characterized in that: The step of calculating the homography matrix based on the key point pair set comprises: Extracting key point coordinate information from the set of successfully matched key point pairs, wherein the key point coordinate information includes corresponding key point coordinates in the first image and the second image; Calculating a homography matrix based on the key point coordinate information to describe a mapping relationship between the first image and the second image, wherein the mapping relationship includes a perspective transformation, an affine transformation, or a combination relationship; Based on the preset evaluation index, the calculated homography matrix is ​​verified and optimized to obtain the final homography matrix.

5. The astronomical image stitching method based on deep learning according to claim 1, characterized in that: The step of performing image transformation and fusion based on the homography matrix includes: Performing a corresponding image transformation on the first image or the second image using a homography matrix so that the first image and the second image are aligned to the same viewing angle; Performing image stitching on the first image and the second image at the same viewing angle; The stitched first image and the second image are fused for smooth transition to complete the image stitching.

6. The astronomical image stitching method based on deep learning according to claim 1, characterized in that: The pre-processed original image is input into the key point detection module of the XFeat neural network for key point detection; Before the key point detection step includes detecting image points with preset key features, the step includes: The original image is preprocessed, and the preprocessing includes graying the original image from three channels to a single channel.

7. An astronomical image stitching device based on deep learning, characterized in that: include: An image input module is used to input the preprocessed original image into the key point detection module of the XFeat neural network for key point detection; The key point detection includes detecting image points having preset key features; The original image includes a first image and a second image; A heat map output module, used to output a key point heat map based on the detection results, wherein the value of each pixel in the key point heat map represents the possibility that the corresponding position is a key point; A position extraction module is used to extract the key point positions from the key point heat map and map them to the size of the corresponding original image to obtain the pixel coordinates of the corresponding key points in the original image; An extraction module, used for performing feature extraction based on the pixel coordinates and outputting a key point feature descriptor vector; A matching module, used to match the nearest neighbors of the feature descriptor vectors of each key point in the first image in the second image using a nearest neighbor search algorithm, and generate a rough matching candidate set, wherein the matching candidate set includes a plurality of potential matching key point pairs; The prediction module is used to take the nearest neighbor pair of each key point pair on the original coarse-grained feature map as input, learn and predict the corresponding offset based on the preset model; An optimization module is used to optimize and adjust the matching results based on the predicted offset to obtain a set of key point pairs with precise matching; A calculation module, used for calculating a homography matrix based on the set of key point pairs; Fusion module, used for image transformation and fusion based on homography matrix.

8. The astronomical image stitching device based on deep learning according to claim 7, characterized in that: The extraction module comprises: A position determination unit, used to determine the position of the key point in the image according to the pixel coordinates; A feature extraction unit, used to extract a local feature descriptor of a key point within a preset range of the position, so as to reflect the local feature information within the preset range of the key point; The integrated output unit is used to process and integrate the extracted local feature descriptors and output a 64-dimensional key point feature descriptor vector.

9. A computer device, characterized in that: include: processor; Memory for storing computer programs; Wherein, when the processor executes the computer program, the deep learning-based astronomical image stitching method described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the deep learning-based astronomical image stitching method according to any one of claims 1 to 6 is implemented.

Images