Multi-source satellite video combined image stabilization method based on multi-source image registration

Abstract

The invention discloses a multi-source satellite video combined image stabilization method based on multi-source image registration, which belongs to the technical field of satellite video image sequence image stabilization, and comprises the following steps of: 1, carrying out image combination on video images, 2, carrying out feature extraction on combined video frames by adopting a multi-mode registration method, and carrying out feature extraction on the combined video frames; obtaining corresponding homonymy feature points among the image frames; 3, motion estimation; 4, fitting a transformation model; and 5, motion compensation. The method comprises the following steps: extracting image homonymy points between homologous video frames and between multi-source video frames; marking homonymy points to realize the corresponding relation of the positions of the same feature point on the homologous and multi-source images; and fitting the coordinates of the frame sequence image feature points to form a virtual plane, and re-sampling all video frames to the virtual plane to realize relay image stabilization of the multi-source satellite video.

Description

Technical Field

[0001] The present invention belongs to the technical field of satellite video image sequence stabilization, and in particular relates to a multi-source satellite video joint stabilization method based on multi-source image registration. Background Art

[0002] In recent years, video satellites have gradually emerged, and using the staring video of video satellites for a series of applications such as regional monitoring and target tracking has become an important technology for satellite remote sensing. Due to the increase in the types of satellite videos that can perform staring imaging, each can only express a single message and be used independently, and the application capabilities of multiple types of satellite videos have not been fully utilized. Therefore, a method is needed to fully combine the capabilities of various types of satellite videos and maximize their application value.

[0003] Classic video stabilization technology usually achieves stabilization by updating and correcting the geographic information attached to satellite videos. However, since different types of satellite videos come from different orbits, have different imaging methods, and video images show strong radiation differences, it brings great challenges to the joint stabilization of multi-source satellite videos. Summary of the invention

[0004] In view of the above problems existing in the prior art, the present invention adopts an image registration method based on computer vision, and provides a multi-source satellite video joint stabilization method based on multi-source image registration, which extracts the same-name points between homologous video frames and multi-source video frames; by marking the same-name points, the corresponding relationship between the positions of the same feature points on the homologous and multi-source images is achieved; the coordinates of the feature points of the frame sequence images are fitted into a virtual plane, and all video frames are resampled to this virtual plane to achieve relay stabilization of multi-source satellite videos.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: A multi-source satellite video joint image stabilization method based on multi-source image registration comprises the following steps: Step 1: Combine the video images in a relay, fusion or cross combination mode. Step 2: Use the multimodal registration method to extract features from the video frames combined in step 1 to obtain the corresponding feature points of the same name between each image frame; Step 3: Motion estimation: Mark all extracted feature point pairs with the same name, search among the feature points of all frames, extract the feature points that exist in all frames and group them; then calculate the average of each feature point coordinate combination as the position of the feature point on the virtual frame, and the point coordinate average of all feature point coordinate combinations constitutes the set of feature points with the same name in the virtual frame; Step 4: Transformation model fitting: After obtaining the positions of all feature points of the virtual frame in step 3, the feature points of each frame image and the feature points of the virtual frame are combined into pairs to fit the transformation model, and the geometric transformation model from the frame image to the virtual frame is obtained; Step 5: Motion compensation. Using the transformation model obtained in step 4, the image frame sequence is resampled to a virtual frame and arranged in order to form a stabilized video frame image sequence.

[0006] As a further improvement of the above technical solution: Step 2 includes: Step 21: Feature detection, obtaining a two-dimensional phase model, and detecting Harris corner points on the phase map as feature points; Step 22: Feature description, obtain the feature descriptor of each feature point, and obtain the feature points with the same name between frame images by matching the descriptors.

[0007] In step 21, the real and imaginary parts of the image are calculated in the spatial domain by combining the Log-Gabor filter with the Fourier transform. The real and imaginary parts represent the even symmetric wavelet and the odd symmetric wavelet respectively. The image is convolved with the even symmetric wavelet and the odd symmetric wavelet respectively to obtain the response component. The analysis results of all directions and all orientations are considered, and the noise compensation term is introduced to finally obtain the two-dimensional phase model.

[0008] In step 21, the even-symmetric wavelet is , the odd symmetric wavelet is , the image is , the response component obtained by convolution is and , ,image Amplitude component at scale s and direction o and phase component Can be and Obtained, the noise compensation term is introduced as T, and the obtained two-dimensional phase model is , .in, is the pixel coordinate, is a weighting function, is the deviation of the phase component, It is a number that prevents the denominator from being 0. The image is processed by the phase model to obtain the phase map of the image. Harris corner point detection is performed on this phase map, and the detected points are used as feature points of the image.

[0009] In step 22, given an image, first, the amplitude is calculated using the response component obtained in step 21; then, the convolution layers are arranged in order to obtain a convolution sequence, which is a multi-channel convolution map. For each pixel position of the convolution map, an ordered array is obtained, and then the maximum value in the array and its corresponding position channel are calculated, and the position channel is set to the pixel value of the pixel position.

[0010] In step 22, the image is , the response component obtained in step 21 is and , the calculated amplitude is the amplitude at scale s and direction o. For direction o, all The amplitudes of the scales are summed to obtain the log-Gabor layer , the resulting multi-channel convolution map is ,in N s is a specific scale value. is the direction number, superscript ω = 1, 2, ..., and the pixel position is , the ordered array is , through the formula Find the maximum value in an array and its corresponding position channel ,Will The value of is set to the pixel position in the feature description map The pixel value of the feature description map refers to the pixel position Corresponding convolution map, for each feature point, select a local image block with J×J pixels centered on the feature, and use a Gaussian function with a standard deviation equal to J / 2 to assign weights to each pixel. If the window position changes, this process avoids sudden changes in the feature description. Then, the local area is divided into 6×6 sub-grids, and a distribution histogram without face elements is constructed for each sub-grid. The feature vector is obtained by connecting all histograms, and the dimension of the feature vector is 6×6×n; in order to obtain invariance to illumination changes, the final normalized feature vector is the feature descriptor of the feature point, where J and n are positive integers.

[0011] The beneficial effects of the present invention are as follows: by utilizing the technology based on multi-source image registration, fast and accurate feature extraction and motion estimation between images of different sensor types can be achieved, and the advantages of multi-source satellite videos can be combined to expand the monitoring time of the target area or enhance the information expression capability of satellite videos, making the coordinated application of multi-source satellite videos possible, and effectively improving and facilitating the application capabilities of satellite videos in more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1It is a flow chart of the multi-source satellite video joint image stabilization method based on multi-source image registration of the present invention. DETAILED DESCRIPTION

[0013] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0014] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0015] A multi-source satellite video joint stabilization method based on multi-source image registration, such as Figure 1 As shown, the following steps are included: Step 1: Perform image combination, which adopts one of the following methods: relay, fusion or cross.

[0016] Among them, the relay type is to arrange the multi-source time-series images head to tail and perform registration in an adjacent frame manner. The fusion type is divided into two parts: on the one hand, it is necessary to register and fuse each frame of the multi-source image to combine the information advantages contained in the two, and on the other hand, the fused image is subjected to feature extraction in an adjacent frame manner. The cross type is to cross-arrange the multi-source video frame images and extract features in an adjacent frame manner. The specific technical solutions of the relay type, fusion type, and cross type are existing technologies and will not be repeated here.

[0017] Step 2: Use multimodal registration method to extract the same-name features from video frames; Among them, the registration method or registration algorithm is a radiation transformation insensitive feature transformation algorithm, which uses the phase consistency principle of the image to eliminate the large radiation intensity differences between multi-source images.

[0018] Step 2 includes the following two steps: Step 21: Feature detection: The real and imaginary parts of the image are calculated by combining the Log-Gabor filter (LGF) in the spatial domain with the Fourier transform to represent the even symmetric wavelet ( ) and odd symmetric wavelet ( ); Use even symmetric and odd symmetric log-Gabor wavelets to image Convolution is performed to obtain the response component and , , where the image Amplitude component at scale s and direction o and phase component Can be and Obtain; Considering the analysis results of all directions and all orientations, and introducing the noise compensation term T, the final two-dimensional phase (PC) model is: .in, is the pixel coordinate, is a weighting function, is the deviation of the phase component, It is a very small number to prevent the denominator from being 0. The image is processed by the phase model to obtain the phase map of the image, and Harris corner point detection is performed on this phase map, and the detected points are used as feature points of the image.

[0019] Step 22: Feature description. Use the corresponding components and amplitudes obtained in step 21 to perform feature description. Specifically, given an image , using the response component obtained in step 21 and , calculate the amplitude at scale s and direction o : For direction o, all Scale (here N s is a specific scale value) are summed to obtain the log-Gabor layer ; Then, the log-Gabor convolution layers are arranged in order to obtain the log-Gabor convolution sequence, which is a multi-channel convolution mapping ,in is the number of directions, with superscript ω = 1, 2, .... Therefore, for each pixel position of the convolutional image (a convolutional sequence is a sequence of multiple convolutional images), , we can get an ordered array Then use Find the maximum value in this array and its corresponding position channel .we will The value of is set to the position in the feature description map The pixel value of the feature description map refers to the pixel position The corresponding convolution graph.

[0020] For each feature point, a local image block with J×J pixels centered on the feature is selected, and a Gaussian function with a standard deviation equal to J / 2 is used to assign a weight to each pixel. This process avoids sudden changes in the feature description if the window position changes. Then, the local area is divided into 6×6 subgrids, and a distribution histogram without facets is constructed for each subgrid. The feature vector is obtained by connecting all histograms. Therefore, the dimension of the feature vector is 6×6×n; in order to obtain invariance to illumination changes, the feature vector is finally normalized to obtain the feature descriptor of the feature point. By matching the descriptors of the feature points, the corresponding feature points between frame images are screened. Where J and n are positive integers.

[0021] Figure 1 Obtaining the same-name points through image registration Image registration is used to extract the same-name feature points between frame images, and use the same-name feature points to fit the transformation matrix. Since video frame images are multimodal, the phase method can be used for registration without considering the so-called mode. In other words, this step is to obtain the same-name feature points between frame images through registration for subsequent motion estimation.

[0022] Step 3: Motion estimation: Mark all extracted feature point pairs with the same name, search among the feature points of all frames, extract the feature points that exist in all frames and group them; then calculate the average of each feature point coordinate combination as the position of the feature point on the virtual frame; Step 4: Transformation model fitting: After obtaining the positions of all feature points of the virtual frame in step 3, the feature points of each frame image and the feature points of the virtual frame are combined into pairs to fit the transformation model. The transformation model is the transformation matrix obtained by fitting the feature points, which can be an affine transformation, a projection transformation or a similarity transformation.

[0023] Step 5: Motion compensation: Using the transformation model obtained in step 4, the image frame sequence is resampled to virtual frames respectively, and the stabilized video frame image sequence is obtained after arranging them in order.

[0024] Finally, it is necessary to explain here that the above embodiments are only used to further illustrate the technical solution of the present invention in detail and cannot be understood as limiting the scope of protection of the present invention. Some non-essential improvements and adjustments made by technicians in this field based on the above content of the present invention all fall within the scope of protection of the present invention.

Claims

1. A multi-source satellite video joint image stabilization method based on multi-source image registration, characterized in that: The following steps are involved: Step 1: Combine the video images in a relay, fusion or cross combination mode. Step 2: Use the multimodal registration method to extract features from the video frames combined in step 1 to obtain the corresponding feature points of the same name between each image frame; Step 3: Motion estimation: Mark all extracted feature point pairs with the same name, search among the feature points of all frames, extract the feature points that exist in all frames and group them; then calculate the average of each feature point coordinate combination as the position of the feature point on the virtual frame, and the point coordinate average of all feature point coordinate combinations constitutes the set of feature points with the same name in the virtual frame; Step 4: Transformation model fitting: After obtaining the positions of all feature points of the virtual frame in step 3, the feature points of each frame image and the feature points of the virtual frame are combined into pairs to fit the transformation model, and the geometric transformation model from the frame image to the virtual frame is obtained; Step 5: Motion compensation. Using the transformation model obtained in step 4, the image frame sequence is resampled to a virtual frame and arranged in order to form a stabilized video frame image sequence.

2. The multi-source satellite video joint image stabilization method according to claim 1, characterized in that: Step 2 includes: Step 21: Feature detection, obtaining a two-dimensional phase model, and detecting Harris corner points on the phase map as feature points; Step 22: Feature description, obtain the feature descriptor of each feature point, and obtain the feature points with the same name between frame images by matching the descriptors.

3. The multi-source satellite video joint image stabilization method according to claim 2, characterized in that: In step 21, the real and imaginary parts of the image are calculated in the spatial domain by combining the Log-Gabor filter with the Fourier transform. The real and imaginary parts represent the even symmetric wavelet and the odd symmetric wavelet respectively. The image is convolved with the even symmetric wavelet and the odd symmetric wavelet respectively to obtain the response component. The analysis results of all directions and all orientations are considered, and the noise compensation term is introduced to finally obtain the two-dimensional phase model.

4. The multi-source satellite video joint image stabilization method according to claim 3, characterized in that: In step 21, the even-symmetric wavelet is , the odd symmetric wavelet is , the image is , the response component obtained by convolution is and , ,image Amplitude component at scale s and direction o and phase component Can be and Obtained, the noise compensation term is introduced as T, and the obtained two-dimensional phase model is , ,in, is the pixel coordinate, is a weighting function, is the deviation of the phase component, It is a number that prevents the denominator from being 0. The image is processed by the phase model to obtain the phase map of the image. Harris corner point detection is performed on this phase map, and the detected points are used as feature points of the image.

5. The multi-source satellite video joint image stabilization method according to claim 4, characterized in that: In step 22, given an image, first, the amplitude is calculated using the response component obtained in step 21; then, the convolution layers are arranged in order to obtain a convolution sequence, which is a multi-channel convolution map. For each pixel position of the convolution map, an ordered array is obtained, and then the maximum value in the array and its corresponding position channel are calculated, and the position channel is set to the pixel value of the pixel position.

6. The multi-source satellite video joint image stabilization method according to claim 5, characterized in that: In step 22, the image is , the response component obtained in step 21 is and , the calculated amplitude is the amplitude at scale s and direction o. For direction o, all The amplitudes of the scales are summed to obtain the log-Gabor layer , the resulting multi-channel convolution map is ,in, N s is a specific scale value. is the direction number, superscript ω = 1, 2, ..., and the pixel position is , the ordered array is , through the formula Find the maximum value in an array and its corresponding position channel ,Will The value of is set to the pixel position in the feature description map The pixel value of the feature description map refers to the pixel position Corresponding convolution map, for each feature point, select a local image block with J×J pixels centered on the feature, and use a Gaussian function with a standard deviation equal to J / 2 to assign weights to each pixel. If the window position changes, this process avoids sudden changes in the feature description. Then, the local area is divided into 6×6 sub-grids, and a distribution histogram without face elements is constructed for each sub-grid. The feature vector is obtained by connecting all histograms, and the dimension of the feature vector is 6×6×n; in order to obtain invariance to illumination changes, the final normalized feature vector is the feature descriptor of the feature point, where J and n are positive integers.

Images