A Multi-Source Satellite Video Joint Video Stabilization Method Based on Multi-Source Image Registration

Abstract

The present invention discloses a multi-source satellite video joint image stabilization method based on multi-source image registration, belonging to the technical field of satellite video image sequence image stabilization, and comprising the following steps: 1: performing image combination on video images; 2: extracting features of the combined video frames by using a multi-modal registration method to obtain corresponding homologous feature points between each image frame; 3: motion estimation; 4: transformation model fitting; 5: motion compensation. The present invention extracts homologous points between homologous video frames and between multi-source video frames; realizes the corresponding relationship of the positions of the same feature point on homologous and multi-source images through marking of the homologous points; fits the coordinate of the feature points of the frame sequence images into a virtual plane, and resamples all video frames onto this virtual plane to realize the relay image stabilization of multi-source satellite videos.

Description

Technical Field

[0001] The present invention belongs to the technical field of satellite video image sequence stabilization, and specifically relates to a multi-source satellite video joint stabilization method based on multi-source image registration. Background Art

[0002] In recent years, video satellites have gradually emerged. Using the staring videos of video satellites for a series of applications such as regional monitoring and target tracking has become an important technology in satellite remote sensing. Due to the increasing variety of satellite videos capable of staring imaging, each can only express single information and be independently applied, without fully exerting the application capabilities of various types of satellite videos. Therefore, a method is needed to fully combine the capabilities of various types of satellite videos and maximize their application value.

[0003] Classical video stabilization techniques usually achieve the stabilization effect by updating and correcting the geographical information attached to satellite videos. However, due to different types of satellite videos originating from different orbits, having different imaging methods, and the video images showing strong radiation differences, it brings great challenges to the joint stabilization of multi-source satellite videos. Summary of the Invention

[0004] Aiming at the above problems existing in the prior art, the present invention adopts an image registration method based on computer vision, provides a multi-source satellite video joint stabilization method based on multi-source image registration, extracts the corresponding image homologous points between homologous video frames and between multi-source video frames; through marking the homologous points, realizes the corresponding relationship of the positions of the same feature point on homologous and multi-source images; fits the coordinate of the feature points of the frame sequence images into a virtual plane, and resamples all video frames onto this virtual plane to achieve the relay stabilization of multi-source satellite videos.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A multi-source satellite video joint stabilization method based on multi-source image registration, comprising the following steps:

[0007] Step 1: Perform image joint on the video images, and the joint method adopts one of relay type, fusion type or cross type.

[0008] Step 2: Use a multi-modal registration method to extract features from the video frames jointed in Step 1 to obtain the corresponding homologous feature points between each image frame.

[0009] Step 3: Motion Estimation: Mark all pairs of corresponding feature points extracted, search among the feature points in all frames, extract the feature points that exist in all frame images and group them; then calculate the mean of each combination of feature point coordinates as the position of the feature point on the virtual frame, and the mean of the point coordinates obtained from all combinations of feature point coordinates constitutes the set of corresponding feature points on the virtual frame.

[0010] Step 4: Transformation Model Fitting: After obtaining the positions of all feature points on the virtual frame in Step 3, pair the feature points of each frame image with the feature points on the virtual frame respectively for transformation model fitting to obtain the geometric transformation model from the frame image to the virtual frame.

[0011] Step 5: Motion Compensation. Using the transformation model obtained in Step 4, resample the image frame sequence onto the virtual frame, and the arranged sequence is the stabilized video frame image sequence.

[0012] As a further improvement of the above technical solution:

[0013] Step 2 includes:

[0014] Step 21: Feature Detection. Obtain the two-dimensional phase model, and detect Harris corner points on the phase diagram as feature points.

[0015] Step 22: Feature Description. Obtain the feature descriptor of each feature point, and obtain the corresponding feature points between frame images by matching the descriptors.

[0016] In Step 21, calculate the real part and the imaginary part of the image by combining the Fourier transform in the spatial domain with a Log-Gabor filter. The real part and the imaginary part respectively represent the even-symmetric wavelet and the odd-symmetric wavelet. Convolve the image with the even-symmetric wavelet and the odd-symmetric wavelet respectively to obtain the response components. Consider the analysis results in all directions and all orientations, and introduce a noise compensation term to finally obtain the two-dimensional phase model.

[0017] In Step 21, the even-symmetric wavelet is and the odd-symmetric wavelet is The image is The response components obtained by convolution are and , , the amplitude component and the phase component of the image at scale s and direction o can be obtained through and Introduce the noise compensation term as T, and the obtained two-dimensional phase model is , . Among them, is the pixel coordinate. is a weighting function, is the deviation of the phase component, is a number that prevents the denominator from being zero. The image is processed through the phase model to obtain the phase diagram of the image. Harris corner detection is performed on this phase diagram, and the detected points are used as the feature points of the image.

[0018] In step 22, given an image, first, calculate the amplitude using the response component obtained in step 21; then, arrange the convolutional layers in sequence to obtain a convolutional sequence, which is a multi-channel convolutional map. For each pixel position of the convolutional map, obtain an ordered array, and then find the maximum value in the array and its corresponding position channel, and set the position channel as the pixel value of the pixel position.

[0019] In step 22, the image is , and the response component obtained in step 21 is and , and the calculated amplitude is the amplitude at scale s and orientation o. For orientation o, the amplitudes of all scales are summed to obtain the log-Gabor layer , and the obtained multi-channel convolutional map is , where N s is a specific scale value, is the number of directions, with superscript ω = 1, 2, …, and the pixel position is , and the ordered array is , and the maximum value in the array is obtained through the formula and its corresponding position channel , and is set as the pixel value of pixel position in the feature description map, where the feature description map refers to the convolutional map corresponding to pixel position . For each feature point, select a local image block with J×J pixels centered on the feature, and use a Gaussian function with a standard deviation equal to J / 2 to assign weights to each pixel. If the window position changes, this process avoids sudden changes in feature description. Then, divide the local area into 6×6 sub-grids, and construct a distribution histogram without bins for each sub-grid. The feature vector is obtained by connecting all histograms. The dimension of the feature vector is 6×6×n; to obtain invariance to illumination changes, finally, the normalized feature vector is the feature descriptor of the feature point, where J and n are positive integers.

[0020] ​The beneficial effects of the present invention are as follows: By using the technology based on multi-source image registration, fast and accurate feature extraction and motion estimation can be achieved between images of different sensor types. The advantages of multi-source satellite videos can be combined to extend the monitoring duration of the target area or enhance the information presentation ability of satellite videos, making the collaborative application of multi-source satellite videos possible and effectively improving and facilitating the application ability of satellite videos in more aspects. Brief Description of the Drawings

[0021] Figure 1 It is a flowchart of the multi-source satellite video joint image stabilization method based on multi-source image registration of the present invention. Detailed Embodiment

[0022] The following details the specific embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining and illustrating the present invention and are not used to limit the present invention.

[0023] For the sake of description, spatial relative terms such as "above...", "over...", "on the upper surface of...", "above" etc. can be used here to describe the spatial positional relationship between a device or feature shown in the figure and other devices or features. It should be understood that the spatial relative terms are intended to include different orientations in use or operation in addition to the orientation of the device described in the figure. For example, if the device in the drawing is inverted, the device described as "above other devices or structures" or "over other devices or structures" will then be positioned as "below other devices or structures" or "under other devices or structures". Thus, the exemplary term "above..." can include both the orientations of "above..." and "below...". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and corresponding interpretations are made for the spatial relative descriptions used here.

[0024] A multi-source satellite video joint image stabilization method based on multi-source image registration, as Figure 1 shown, includes the following steps:

[0025] Step 1: Perform image joint. The image joint method adopts one of the relay type, fusion type or cross type.

[0026] Among them, in the relay type, multi-source sequential images are arranged end to end, and registration is performed in the adjacent frame manner. The fusion type is divided into two parts: on the one hand, each frame of multi-source images needs to be registered and fused to integrate the information advantages of both, and on the other hand, feature extraction is performed on the fused images in the adjacent frame manner. The cross type is to arrange the multi-source video frame images crosswise and perform feature extraction in the adjacent frame manner. The specific technical solutions of the relay type, fusion type and cross type are prior arts and will not be elaborated here.

[0027] Step 2: Extract homologous features from video frames using a multi-modal registration method;

[0028] Among them, the registration method or algorithm is a feature transformation algorithm insensitive to radiation transformation. This method uses the principle of phase consistency of images to eliminate large radiation intensity differences between multi-source images.

[0029] Step 2 includes the following two steps:

[0030] Step 21: Feature detection: Calculate the real and imaginary parts of the image, which represent the even-symmetric wavelet ( ) and the odd-symmetric wavelet ( ), respectively, by combining the Fourier transform with the Log-Gabor filter (LGF) in the spatial domain; Convolve the image with the even-symmetric and odd-symmetric log-Gabor wavelets respectively to obtain the response components and and , , where the amplitude component and the phase component of the image at scale s and direction o can be obtained through and ; Considering the analysis results in all directions and all orientations and introducing the noise compensation term T, the final two-dimensional phase (PC) model is: . Among them, is the pixel coordinate, is a weighting function, is the deviation of the phase component, is a very small number to prevent the denominator from being zero. The image is processed by the phase model to obtain the phase map of the image, and Harris corner detection is performed on this phase map, and the detected points are used as the feature points of the image.

[0031] Step 22: Feature description. Use the corresponding components and amplitudes obtained in Step 21 for feature description. Specifically, given the image , use the response components and obtained in Step 21 to calculate the amplitude at scale s and direction o: For direction o, the amplitudes of all scales (here N s is a specific scale value) are summed to obtain the log-Gabor layer ; Then, arrange the log-Gabor convolution layers in sequence to obtain the log-Gabor convolution sequence, which is a multi-channel convolution mapping , where are the direction numbers, with superscript ω = 1, 2, …. Therefore, for each pixel position of the convolutional graphs (the convolutional sequence consists of multiple convolutional graphs forming a sequence) , we can obtain an ordered array . Then use to find the maximum value in this array and its corresponding position channel . We set the value of as the pixel value at the position in the feature description graph, where the feature description graph refers to the convolutional graph corresponding to the pixel position .

[0032] For each feature point, select a local image patch with J×J pixels centered on this feature, and use a Gaussian function with a standard deviation equal to J / 2 to assign weights to each pixel. If the window position changes, this process avoids sudden changes in feature description. Then, divide the local area into 6×6 sub-grids, and construct a distribution histogram without bins for each sub-grid. Obtain the feature vector by connecting all histograms. Therefore, the dimension of the feature vector is 6×6×n; to obtain invariance to illumination changes, finally normalize the feature vector to get the feature descriptor of the feature point. By matching the descriptors of the feature points, the corresponding feature points between frame images are screened out. Where J and n are positive integers.

[0033] Figure 1 In image registration for obtaining corresponding points, image registration is used to extract corresponding feature points between frame images. Use the corresponding feature points to fit the transformation matrix. Since video frame images are multi-modal, the phase method can be used for registration without considering the so-called modality. In other words, this step is to obtain the corresponding feature points between frame images through registration for subsequent motion estimation.

[0034] Step 3: Motion estimation: Mark all pairs of corresponding feature points extracted, search among the feature points of all frames, and extract the feature points that exist in all frame images for grouping; then calculate the mean of each combination of feature point coordinates as the position of the feature point on the virtual frame;

[0035] Step 4: Transformation model fitting: After obtaining the positions of all feature points on the virtual frame in Step 3, respectively combine the feature points of each frame image with the feature points on the virtual frame in pairs for transformation model fitting. The transformation model is the transformation matrix obtained by fitting the feature points, which can be an affine transformation, a projective transformation, or a similarity transformation.

[0036] Step 5: Motion compensation: Use the transformation model obtained in Step 4 to resample the image frame sequence onto the virtual frame respectively, and the sequence arranged in order is the stabilized video frame image sequence.

[0037] Finally, it is necessary to state here that the above embodiments are only used to further elaborate on the technical solutions of the present invention, and cannot be construed as limiting the protection scope of the present invention. Some non-essential improvements and adjustments made by those skilled in the art based on the above content of the present invention all fall within the protection scope of the present invention.

Claims

1. A multi-source satellite video joint image stabilization method based on multi-source image registration, characterized in that, Including the following steps: Step 1: Combine video images. The combination method is one of relay type, fusion type or cross type. Step 2: Use a multimodal registration method to extract features from the video frames combined in Step 1, and obtain corresponding homologous feature points between image frames. Step 3: Motion estimation: Mark all pairs of extracted homologous feature points, search among the feature points of all frames, extract the feature points that exist in all frame images and group them; then calculate the mean value of each feature point coordinate combination as the position of the feature point on the virtual frame, and the mean values of the point coordinates obtained from all feature point coordinate combinations constitute the set of homologous feature points of the virtual frame. Step 4: Transformation model fitting: After obtaining the positions of all feature points of the virtual frame in Step 3, respectively combine the feature points of each frame image and the feature points of the virtual frame into pairs for transformation model fitting, and obtain the geometric transformation model from the frame image to the virtual frame. Step 5: Motion compensation. Using the transformation model obtained in Step 4, resample the image frame sequence onto the virtual frame, and the sequentially arranged result is the stabilized video frame image sequence.

2. The multi-source satellite video joint image stabilization method according to claim 1, wherein: Step 2 includes: Step 21: Feature detection to obtain a two-dimensional phase model, and detect Harris corners on the phase diagram as feature points. Step 22: Feature description to obtain the feature descriptor of each feature point, and obtain the homologous feature points between frame images by matching the descriptors.

3. The multi-source satellite video joint image stabilization method according to claim 2, wherein: In Step 21, the real part and the imaginary part of the image are calculated by combining the Fourier transform in the spatial domain through a Log-Gabor filter. The real part and the imaginary part respectively represent the even-symmetric wavelet and the odd-symmetric wavelet. The even-symmetric wavelet and the odd-symmetric wavelet are respectively used to convolve the image to obtain response components. Considering the analysis results in all directions and all orientations, and introducing a noise compensation term, finally a two-dimensional phase model is obtained.

4. The multi-source satellite video joint steady imaging method according to claim 3, wherein: In step 21, the even symmetric wavelet is , and the odd symmetric wavelet is , the image is , and the response components obtained by convolution are and . . The amplitude component and the phase component of the image at scale s and direction o can be obtained through and . By introducing the noise compensation term as T, the obtained two-dimensional phase model is . , where is the pixel coordinate, is a weighting function, is the deviation of the phase component, is a number to prevent the denominator from being zero. The image is processed by the phase model to obtain the phase diagram of the image. Harris corner detection is performed on this phase diagram, and the detected points are used as the feature points of the image.

5. The multi-source satellite video joint image stabilization method according to claim 4, characterized in that: In Step 22, given an image, first, calculate the amplitude using the response components obtained in Step 21; then, arrange the convolutional layers in sequence to obtain a convolutional sequence, which is a multi-channel convolutional mapping. For each pixel position of the convolutional map, obtain an ordered array, and then find the maximum value in the array and its corresponding position channel, and set the position channel as the pixel value of the pixel position.

6. The multi-source satellite video joint image stabilization method according to claim 5, characterized in that: In step 22, the image is , the response components obtained in step 21 are and , and the calculated amplitude is the amplitude at scale s and direction o. For direction o, the amplitudes at all scales are summed to obtain the log-Gabor layer , and the resulting multi-channel convolution map is , where N s is a specific scale value, is the number of directions, the superscript ω = 1, 2, …, the pixel position is , the ordered array is , and the maximum value in the array is obtained through the formula and its corresponding position channel . Set the value of to the pixel value at pixel position in the feature description map. Here, the feature description map refers to the convolution map corresponding to pixel position . For each feature point, a local image block with J×J pixels centered on this feature is selected, and a Gaussian function with a standard deviation equal to J / 2 is used to assign weights to each pixel. If the window position changes, this process avoids sudden changes in feature description. Then, the local area is divided into 6×6 sub-grids, and a distribution histogram without bins is constructed for each sub-grid. The feature vector is obtained by connecting all histograms. The dimension of the feature vector is 6×6×n. To obtain invariance to illumination changes, the finally normalized feature vector is the feature descriptor of this feature point, where J and n are positive integers.​

Images