A multi-band push-broom imaging method based on cosine similarity image collaborative stitching

Abstract

The application provides a kind of multi-band push-broom imaging method based on cosine similarity image collaborative splicing, comprising the following steps: step one, data preprocessing, the image obtained by infrared detector push-broom imaging system is arranged in time sequence to form data cube, the selected image is one-dimensional expansion, form can be calculated by cosine similarity;Step two, cosine similarity calculation, for selecting the best splicing point, and recording the frame shift information, calculate the frame shift distance between all adjacent frames by this method, store the data in the empty array according to time sequence;Step three, spectral image splicing, complete multi-band spectral image splicing by using the frame shift information calculated by full channel.The multi-band push-broom imaging method based on cosine similarity image collaborative splicing provided by the application realizes the best matching method between two adjacent frames.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of infrared imaging technology, and particularly relates to a multi-band push-broom imaging method based on cosine similarity image cooperative splicing. BACKGROUND

[0002] Infrared is an electromagnetic wave with a wavelength range of 760nm-1mm, also known as infrared ray. According to Planck's blackbody radiation law, any object with an absolute temperature higher than absolute zero can radiate electromagnetic waves. The wavelength and energy of the radiated electromagnetic waves are closely related to the temperature of the emission source. In view of this characteristic, infrared detectors have great application potential in the fields of atmospheric monitoring, forest fire monitoring, geological exploration, and animal migration observation.

[0003] In the field of infrared hyperspectral remote sensing, infrared detectors are mainly divided into refrigeration-type infrared detectors and non-refrigeration-type infrared detectors. Compared with non-refrigeration-type infrared detectors, refrigeration-type infrared detectors need to work with refrigerants to reduce temperature, so they have high imaging sensitivity, high precision, and small errors. However, they also have large power consumption, large volume, short service life, and high cost. Based on the above problems, refrigeration-type infrared detectors are less used in commercial infrared remote sensing fields. Non-refrigeration-type infrared detectors, due to their working environment without refrigeration and relatively simple structure, have become one of the important development directions in the field of infrared hyperspectral remote sensing.

[0004] During the process of shooting on the ground, the multispectral image data recorded by the spaceborne push-broom non-refrigeration infrared camera is a strip along the orbital direction, and the final goal is to obtain a continuous two-dimensional image along the orbital direction after splicing. However, due to the problems of unstable sampling frame rate caused by the mismatch of speed and altitude ratio and the working temperature of the satellite in the actual flight process, the image obtained by directly splicing each frame to the tail of the previous frame is discontinuous.

[0005] Image registration is the process of matching and superimposing two or more images obtained at different times, different sensors (imaging devices), or different conditions (weather, illumination, camera position and angle, etc.). It has been widely used in the fields of remote sensing data analysis, computer vision, and image processing. First, feature extraction is performed on two images to obtain feature points. By performing similarity measurement, matching feature point pairs are found. Then, image space coordinate transformation parameters are obtained through the matching feature point pairs. Finally, image registration is performed by the coordinate transformation parameters. Feature extraction is the key in registration technology, and accurate feature extraction provides a guarantee for the successful performance of feature matching. Therefore, seeking a feature extraction method with good invariance and accuracy is crucial for matching accuracy.

[0006] In existing technologies, image stitching based on low-level features can be divided into four types: stitching based on Harris corner detectors, stitching based on FAST corner detectors, stitching based on SIFT feature detectors, and stitching based on SURF feature detectors. For shooting environments with few feature points, it may result in no matching corner points between adjacent frames to achieve image stitching, and mismatches are prone to occur in small overlapping areas. Image stitching based on cosine similarity, however, is suitable for a wider range of flexible scenarios.

[0007] Compared to traditional image feature matching algorithms based on low-level features, deep learning-based feature matching algorithms can produce larger-scale and higher-quality matches. For example, while image stitching based on the deep learning LoFTR algorithm can achieve better stitching results, its stitching time is still relatively long and its computational cost is high.

[0008] Therefore, how to efficiently achieve optimal matching between two adjacent frames of a multi-channel image under the condition of low computational cost is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] The purpose of this invention is to provide a multi-band pushbroom imaging method based on cosine similarity image collaborative stitching, addressing the problems in the prior art.

[0010] Therefore, the above-mentioned objectives of the present invention are achieved through the following technical solutions:

[0011] A multi-band pushbroom imaging method based on cosine similarity image co-stitching includes the following steps:

[0012] Step 1: Data preprocessing. The images obtained by the pushbroom imaging system of the uncooled infrared detector are arranged in time series to form a data cube. The initial data contains four channels. The data is divided into four time series with the same data set according to the channels. The initial frame is selected from the full-channel dataset. By analyzing the velocity-to-height ratio, the range of overlapping parts of adjacent frames is determined. After retaining some image redundancy during selection, the selected images are expanded in one dimension to form a form that can be used for cosine similarity calculation.

[0013] Step 2: Cosine similarity calculation is used to select the best splicing point and record the inter-frame shift information. Using the data obtained in Step 1, the cosine similarity of each pair of adjacent n columns of the next frame integrated into a one-dimensional array is calculated with the last n columns of the previous frame integrated into a one-dimensional array. The position of the maximum cosine similarity is found, and the inter-frame shift distance is calculated. The inter-frame shift distance of all adjacent frames is calculated in this way, and the data is stored in an empty array according to the time series.

[0014] Step 3: Spectral image stitching. Multi-band spectral image stitching is completed by utilizing the inter-frame shift information obtained from full-channel calculation.

[0015] While adopting the above technical solutions, the present invention may also adopt or combine the following technical solutions:

[0016] As a preferred technical solution of the present invention: the data preprocessing includes four-channel segmentation, initial frame setting, and computational unit integration. First, four-channel segmentation: the data obtained through push-broom imaging is a data cube arranged in a time sequence. The initial data contains four channels, where the first channel is a full-channel image and the other three are single-channel images. These are divided into four identically arranged data cubes according to their channels. Second, initial frame setting: the initial frame is the starting point for image stitching. The first frame in the data cube is selected as the initial frame. After selecting the initial frame, the next frame is selected as the frame to be stitched according to the push-broom time sequence. Finally, computational unit integration: n consecutive columns of images are selected in the full-channel band as computational units. The last n columns of data from the full-channel initial frame are expanded one-dimensionally to form a one-dimensional array. Similarly, each adjacent n columns of images from subsequent full-channel frames are expanded one-dimensionally into a one-dimensional array, resulting in a one-dimensional array 'a' obtained from the one-dimensional expansion of the last n columns of the full-channel initial frame. p = [x1, x2, ..., x m , ..., x mn ], and a one-dimensional array {b1 = [y1, y2, ..., y] for each adjacent n columns of the second frame of the full channel, expanded into a one-dimensional array {b1 = [y1, y2, ..., y] m , ..., y mn ], b2, ..., b p}, where n is the number of columns selected to integrate n columns of data into a calculation unit, m is the number of rows of data in each frame, i.e., the number of pixels in one column, p is the number of calculation units that can be taken from adjacent n columns in each frame of the full channel, and x and y are the pixel grayscale values ​​at the corresponding positions.

[0017] As a preferred technical solution of the present invention: the cosine similarity calculation specifically includes cosine similarity calculation and inter-frame shift distance calculation: First, the cosine similarity calculation is performed, and the cosine similarity of each calculation unit of the next frame with the last calculation unit of the initial frame is calculated. The specific calculation formula is as follows:

[0018]

[0019] After pairwise calculations are completed, find the i corresponding to the maximum cosine similarity, which means finding the n consecutive columns most similar to the last n columns of the initial frame. Then the i+nth column and the rest of the second frame are the parts to be spliced. Assuming a total of l columns in a single frame of the full-channel band, the inter-frame shift information between the initial frame and the second frame is (li-n+1). Store this data in an empty array.

[0020] Similarly, the cosine similarity of the one-dimensional array of the last n columns of the second frame and the one-dimensional array of the one-dimensional array of each adjacent n columns of the third frame is calculated to obtain the inter-frame shift information between the second and third frames. The above operation is repeated until an array recording the inter-frame shift information of all adjacent frames is obtained.

[0021] As a preferred technical solution of the present invention: In step three, the spectral image stitching is based on the inter-frame shift information (li-n+1) obtained in step two. The inter-frame shift information is used to obtain the part of each subsequent frame stitched to the end of the previous frame. Each channel starts from the initial frame and stitches frame by frame according to the time sequence to realize multi-channel spectral image stitching.

[0022] Compared to existing technologies, this invention discloses a multi-band pushbroom imaging method based on cosine similarity image collaborative stitching. This method calculates the inter-frame shift information of high signal-to-noise ratio spectral channels using the cosine similarity principle and shares this shift information with the remaining spectral channels, achieving accurate stitching of all bands and obtaining the optimal match between adjacent frames of the image. In this invention, during pushbroom imaging, the area array detector selects a fixed-width image segment of a certain band in the area array through data preprocessing, ensuring that some overlap information is retained between adjacent image segments; cosine similarity matching is used to calculate the optimal stitching point of adjacent image segments and record the inter-frame shift information; by using the inter-frame shift information obtained in the previous step, the image mismatch parameters are determined, and multi-band spectral image stitching is completed. This method can obtain accurate multi-band spectral images even with velocity-to-height ratio mismatch in infrared pushbroom imaging systems, and has great application prospects in the field of aerial imaging for spaceborne detectors. Attached Figure Description

[0023] Figure 1 This is a flowchart of a multi-band pushbroom imaging method based on cosine similarity image collaborative stitching according to the present invention;

[0024] Figure 2 This is a schematic diagram of a detector array for a multi-band pushbroom imaging method based on cosine similarity image collaborative stitching according to the present invention.

[0025] Figure 3 This is a schematic diagram of the spectral stitching process of a multi-band pushbroom imaging method based on cosine similarity image collaborative stitching according to the present invention. Detailed Implementation

[0026] The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 As shown, the multi-band pushbroom imaging method based on cosine similarity image co-stitching of the present invention specifically includes the following steps:

[0028] Step 1: Data preprocessing. Using the multi-band pushbroom imaging method based on cosine similarity image co-stitching of the present invention, the starting frame is selected and the data is converted into a form that can be used for cosine similarity calculation.

[0029] Step 2: Cosine similarity calculation, used to select the best splicing point and record inter-frame shift information;

[0030] Step 3: Spectral image stitching. Multi-band spectral image stitching is completed by utilizing the inter-frame shift information obtained from full-channel calculation.

[0031] In step one, during data preprocessing, the images obtained from the infrared pushbroom imaging system are arranged into a data cube according to time series. The initial data contains four channels, which are then divided into four identical time series datasets according to channels. An initial frame is selected from the full-channel dataset, and the range of overlapping portions between adjacent frames is determined by analyzing the velocity-to-height ratio. Redundancy in some images is retained during selection to facilitate determining the optimal matching position through column traversal. Finally, to facilitate the implementation of the cosine similarity algorithm, the selected images are unfolded in one dimension.

[0032] Data preprocessing includes four-channel segmentation, initial frame setting, and computational unit integration. First, four-channel segmentation: the data obtained through pushbroom imaging is a data cube arranged in a time sequence. The initial data contains four channels, with the first channel being a full-channel image and the other three being single-channel images. This data is divided into four identically arranged data cubes according to the channel sequence. Second, initial frame setting: the initial frame is the starting point for image stitching. Generally, the first frame in the data cube is selected as the initial frame. After selecting the initial frame, the next frame is selected as the frame to be stitched according to the pushbroom time sequence. Finally, computational unit integration: to reduce noise and interference from repetitive features and improve the accuracy of the stitching result, n consecutive columns of images are selected as computational units across the entire channel band. The last n columns of data from the initial frame are expanded one-dimensionally to form a one-dimensional array. Similarly, each adjacent n columns of images from subsequent frames across the entire channel are expanded one-dimensionally into a one-dimensional array, transforming the data into a form suitable for cosine similarity calculation.

[0033] In this invention, the infrared detector is an uncooled area array infrared detector with four channels: one all-pass and the other three single-pass. The velocity-to-height ratio matching is identical across all channels within the same frame. During system pushbroom, the all-pass segment receives the richest infrared radiation signal, resulting in a better signal-to-noise ratio compared to the single-pass segment. Therefore, the inter-frame shift information calculated using cosine similarity across all channels is more accurate. Furthermore, using the inter-frame shift information from the all-pass segment to assist single-channel image stitching saves significant computational resources. Thus, the inter-frame shift calculated from all channels is also applicable to single channels, and the infrared radiation signal received from all channels is the richest, resulting in a higher image signal-to-noise ratio compared to other single channels. The uncooled infrared detector in this invention obtains four-channel images through pushbroom. By using the all-channel segment to assist single-channel image stitching and utilizing the inter-frame shift information obtained from all channels, four-channel image stitching is achieved, significantly saving operational resources and yielding more accurate stitching results.

[0034] Step two, the cosine similarity calculation, is performed by combining the data obtained from data preprocessing with the one-dimensional array of each adjacent n columns of the next frame and the one-dimensional array of the last n columns of the previous frame. The position of the maximum cosine similarity is found to obtain the inter-frame shift distance. The inter-frame shift distance of all adjacent frames is calculated in this way, and the data is stored in an empty array according to the time series to prepare for the subsequent step three, spectral image stitching.

[0035] The cosine similarity calculation includes cosine similarity calculation and inter-frame shift distance calculation. First, the cosine similarity calculation is performed, which is based on the one-dimensional array 'a' obtained by expanding the last n columns of the initial frame across all channels using the data preprocessing in step one. p = [x1, x2, ..., x m , ..., x mn ], and a one-dimensional array {b1 = [y1, y2, ..., y] for each adjacent n columns of the second frame of the full channel, expanded into a one-dimensional array {b1 = [y1, y2, ..., y] m , ..., y mn ], b2, ..., b p},

[0036] Where n is the number of columns selected to integrate n columns of data into a calculation unit, m is the number of rows in each frame (i.e., the number of pixels in one column), p is the number of calculation units that can be taken from adjacent n columns in each frame across all channels, and x and y are the pixel grayscale values ​​at the corresponding positions. Then, a cosine similarity calculation is performed between each calculation unit in the next frame and the last calculation unit in the initial frame. The specific calculation formula is as follows:

[0037]

[0038] After pairwise calculations are completed, the i corresponding to the maximum cosine similarity is found, which means the n consecutive columns most similar to the last n columns of the initial frame can be found. Then the i+nth column and the subsequent part of the second frame are the parts to be stitched. Assuming that there are l columns in a single frame of the full-channel band, the inter-frame shift information between the initial frame and the second frame is (li-n+1). This data is stored in an empty array.

[0039] Similarly, the cosine similarity of the one-dimensional arrays of the last n columns of the second frame and the one-dimensional arrays of each adjacent n columns of the third frame is calculated to obtain the inter-frame shift information between the second and third frames. This process is repeated until an array recording the inter-frame shift information of all adjacent frames is obtained, preparing for subsequent spectral image stitching.

[0040] In step three, after the inter-frame shift distance is found by cosine similarity calculation in step two, each channel starts from the initial frame and sequentially stitches the image after the column where the maximum cosine similarity of the next frame is located to the end of the previous frame to form the stitched image, until the stitching is completed and the final stitching result is formed.

[0041] The spectral image stitching is based on the inter-frame shift information (li-n+1) obtained in step two. The inter-frame shift information is used to obtain the part of each subsequent frame stitched to the end of the previous frame. Each channel starts from the initial frame and stitches frame by frame according to the time sequence to realize multi-channel spectral image stitching.

[0042] Example 1

[0043] Reference Figure 2 The present invention provides a multi-band pushbroom imaging method based on cosine similarity image collaborative stitching. The image obtained by the infrared pushbroom imaging system consists of 1280*1024 pixels. The image is then divided into four channels by a filter. The first channel is a full channel, and the other three channels are single channels. Each channel has 1280*256 pixels. The content captured between each frame is shifted.

[0044] First, data preprocessing is performed, dividing the raw data into four channels, each with a resolution of 1280*256. After selecting the initial frame, refer to... Figure 3 In the left image (a), n consecutive columns of images are selected as computational units across the entire channel band. In this embodiment, 20 columns are selected, and the last 20 columns are expanded into a one-dimensional array of [1280*20, 1]. Similarly, referring to... Figure 3 a) Expand each of the 20 adjacent columns of the second frame into a one-dimensional array of [1280*20, 1], resulting in 237 groups. Calculate the cosine similarity between each calculation unit of the second frame and the last calculation unit of the initial frame. The calculation formula is as follows:

[0045]

[0046] Assuming the maximum cosine similarity after calculation is the 173rd calculation unit, then columns 193 to 256 of the frame to be spliced ​​are the part to be spliced. The calculated inter-frame shift distance is 64, and 64 is stored in an empty array. Similarly, refer to... Figure 3 a) Calculate the cosine similarity between each computational unit of the third frame and the last computational unit of the second frame. Assuming the maximum cosine similarity after calculation is the 120th computational unit, the inter-frame shift distance is calculated to be 117. Store 117 in an empty array. Repeat the above operation to obtain an array {64, 117, ...} that records the inter-frame shift information.

[0047] Next is the spectral image stitching. Using the inter-frame shift distance information calculated across all channels, we can obtain the portion of each frame stitched to the previous frame after the initial frame for each channel. (Refer to...) Figure 3 b, the shaded area is the part stitched to the end of the previous frame. Each channel is stitched frame by frame according to the time sequence from the initial frame to the next until the stitching is complete, refer to... Figure 3 c. This completes the stitching process for the multi-band pushbroom imaging method based on cosine similarity image co-stitching.

[0048] The above specific embodiments are used to explain and illustrate the present invention, and are only preferred embodiments of the present invention, not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A multi-band pushbroom imaging method based on cosine similarity image co-stitching, comprising the following steps: Step 1: Data preprocessing. The images obtained by the infrared detector pushbroom imaging system are arranged in time series to form a data cube. The initial data contains four channels. The data is divided into four time series with the same data set according to the channels. The initial frame is selected in the full-channel dataset. When generating the calculation unit, all n consecutive columns of images are selected. The selected images are expanded in one dimension to form a form that can be used for cosine similarity calculation. Step 2: Cosine similarity calculation is used to select the best splicing point and record the inter-frame shift information. Using the data obtained in Step 1, the cosine similarity of each pair of adjacent n columns of the next frame integrated into a one-dimensional array is calculated with the last n columns of the previous frame integrated into a one-dimensional array. The position of the maximum cosine similarity is found, and the inter-frame shift distance is calculated. The inter-frame shift distance of all adjacent frames is calculated in this way, and the inter-frame shift distances are stored in an empty array according to the time series. Step 3: Spectral image stitching. Multi-band spectral image stitching is completed by utilizing the inter-frame shift information obtained from full-channel calculation.

2. The multi-band pushbroom imaging method based on cosine similarity image co-stitching as described in claim 1, characterized in that: The data preprocessing includes four-channel segmentation, initial frame setting, and computational unit integration. First, four-channel segmentation: the data obtained through pushbroom imaging is a data cube arranged in a time sequence. The initial data contains four channels, with the first channel being a full-channel image and the other three being single-channel images. This data is divided into four identically arranged data cubes according to the channel sequence. Second, initial frame setting: the initial frame is the starting point for image stitching. The first frame in the data cube is selected as the initial frame. After selecting the initial frame, the next frame is selected as the frame to be stitched according to the pushbroom time sequence. Finally, computational unit integration: n consecutive columns of images are selected in the full-channel band as computational units. The last n columns of data from the full-channel initial frame are expanded one-dimensionally to form a one-dimensional array. Similarly, each adjacent n columns of images from subsequent full-channel frames are expanded one-dimensionally into a one-dimensional array, resulting in a one-dimensional array 'a' obtained from the one-dimensional expansion of the last n columns of the full-channel initial frame. p = [x1, x2, ..., x m , ..., x mn ], and a one-dimensional array {b1 = [y1, y2, ..., y] for each adjacent n columns of the second frame of the full channel, expanded into a one-dimensional array {b1 = [y1, y2, ..., y] m , ..., y mn ],b2,...,b p }, where a p It refers to the one-dimensional array obtained by one-dimensionally expanding the last n columns of the initial frame of all channels, x1, x2, x... m ...x mn This refers to the grayscale value of each pixel after the last n columns of the initial frame of the full-channel image are unfolded in one dimension, b1, b2...b p It refers to the one-dimensional array obtained by one-dimensionally expanding all adjacent n columns of the second frame of the full channel, y1, y2, y3, y4, y5, y6, y7, y8, y9, y1 ...9, y1, y2, y3, y4 m ...y mn It refers to the grayscale value of each pixel in the one-dimensional array after the one-dimensional expansion of a certain continuous n columns of image in the second frame of the full channel. n is the number of columns of data selected and integrated into a calculation unit, m is the number of rows of data in each frame, i.e. the number of pixels in a column, and p is the number of calculation units that can be taken from adjacent n columns in each frame of the full channel.

3. The multi-band pushbroom imaging method based on cosine similarity image co-stitching as described in claim 1, characterized in that: The cosine similarity calculation specifically includes cosine similarity calculation and inter-frame shift distance calculation: First, the cosine similarity calculation is performed, where each calculation unit in the next frame is compared with the last calculation unit in the initial frame using a cosine similarity calculation. The specific calculation formula is as follows: Where i is the ordinal number of the calculation unit in the next frame. It is the last calculation unit of the initial frame. It is the i-th computational unit in the next frame, x t y t These are the pixel grayscale values ​​corresponding to the last calculation unit of the initial frame and the t-th pixel of the i-th calculation unit of the second frame, respectively. After pairwise calculations, the cosine similarity between the one-dimensional unfolded array of each consecutive n columns of the second frame and the one-dimensional unfolded array of the last n columns of the initial frame is obtained. By finding the i corresponding to the maximum cosine similarity value, and based on the fact that a larger cosine similarity value represents a higher degree of image similarity, the most similar consecutive n columns of images in the second frame to the last n columns of the initial frame can be found. Then, the i+n column and the subsequent part of the second frame are the parts to be stitched. Assuming that there are l columns in a single frame of the full-channel band, the inter-frame shift information between the initial frame and the second frame is (li-n+1). This data is stored in an empty array. Similarly, the cosine similarity calculation is performed on the one-dimensional array of the one-dimensional unfolded array of the last n columns of the second frame and the one-dimensional array of the one-dimensional unfolded array of each adjacent n columns of the third frame to obtain the inter-frame shift information between the second frame and the third frame. The above operation is repeated until an array recording the inter-frame shift information of all adjacent frames is obtained.

4. The multi-band pushbroom imaging method based on cosine similarity image co-stitching as described in claim 3, characterized in that: In step three, the spectral image stitching is based on the inter-frame shift information (li-n+1) obtained in step two. The inter-frame shift information is used to obtain the part of each subsequent frame stitched to the end of the previous frame. Each channel starts from the initial frame and stitches frame by frame according to the time sequence to realize multi-channel spectral image stitching.

Images