An image deblurring algorithm integrating carrier posture and motion information

Abstract

An image deblurring algorithm that integrates carrier posture motion information. A blurred image dataset containing carrier posture motion information is constructed, and a small-size image feature matrix is ​​obtained through a multi-level downsampling mechanism. The carrier posture motion information is then vectorized and encoded, and this information is concatenated with the small-size image feature matrix. A multi-level upsampling mechanism is then used to restore the image feature matrix size. To address the shortcomings of neural network blind deblurring, the present invention more comprehensively considers the position information of the global blurred image, improving the influence of carrier posture motion information on the blurred image feature matrix, and providing a new solution for motion information fusion methods for non-blind image deblurring.

Description

Technical Field

[0001] The present invention relates to the field of image deblurring, and in particular to an image deblurring algorithm that integrates carrier posture motion information. Background Art

[0002] Remote sensing platforms such as spacecraft, aircraft, and drones typically move with the platform during in-orbit wide-band imaging of the Earth. The relative motion between the imaging system and the target causes varying degrees of image shift in the focal plane. This can easily lead to image information loss and quality degradation during image generation, transmission, and acquisition, resulting in blurred images that are difficult to understand. Image blurring significantly impacts image quality and also affects image recognition and object identification. Currently, image deblurring widely uses convolutional layers to decompose and reconstruct inputs. These layers take an input and extract features by decomposing the image into a feature matrix. However, feature matrices have low spatial understanding, meaning they cannot truly understand the location of objects in the image. This is because, in order to extract object features, all pixels that are not object features must be discarded. This means discarding all background pixels, which removes all knowledge of the object's location in the image.

[0003] Restoring motion blurred images and providing target motion information can be widely used in astronomical detection, military aviation, road traffic, medical imaging, industrial control and other fields, and has very important practical significance.

[0004] Patent application number CN 108198131 A discloses a "space target jitter and motion blur simulation algorithm based on multi-image fusion." This method fuses multiple frames of images using pixel-by-pixel averaging to produce a simulated image of a space target with jitter and motion blur. However, this method lacks sufficient design considerations for determining image position information. Patent application number CN112634153 A discloses an "image deblurring method based on edge enhancement." While ensuring deblurring quality, it reduces model derivation time and the number of model parameters. However, this method does not incorporate prior information and cannot directly determine image position information. Summary of the Invention

[0005] (1) Purpose of the invention

[0006] The purpose of this invention is to provide an image deblurring algorithm that incorporates carrier pose and motion information. By downsampling the image, the present invention encodes the carrier pose and motion information and concatenates it with the image's small-scale feature matrix. This overlays the carrier pose and motion information while deblurring the image, enabling better on-board processing of image positional relationships. This algorithm, which integrates carrier pose and motion information at small-scale image features, provides a novel approach to motion information fusion for non-blind image deblurring.

[0007] (2) Technical solution

[0008] The technical solution of the present invention is an image deblurring algorithm that integrates carrier posture motion information, including: constructing a blurred image data set containing carrier posture motion information, preprocessing the data, extracting blurred image features through a convolutional neural network, obtaining a small-size feature matrix of the image through a multi-level and multi-downsampling mechanism, position vectorizing the carrier posture motion information, first matrix splicing the encoded position information vector and the small-size feature matrix according to the second dimension, and then matrix splicing according to the first dimension, wherein any vacancies are filled with 0, and finally a multi-level upsampling mechanism is used to restore the clear image.

[0009] A blurred image dataset containing carrier posture motion information is constructed. In this method, a three-axis air-floating platform is used to simulate the carrier posture motion. An optical camera is fixed to the three-axis air-floating platform to shoot a fixed scene, and the motion information of the three-axis air-floating platform is recorded during the shooting. The data is then sorted and randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0010] A convolutional neural network is used to extract local features of a single-channel image, and a multi-level downsampling mechanism is used to obtain a small-size feature matrix. The obtained carrier posture motion information matrix A is concatenated with the image small-size feature matrix B. For example, with n=m=3:

[0011]

[0012] The splicing algorithm is as follows:

[0013] First, concatenate the first row of A with B along the second dimension to get:

[0014]

[0015] Next, transpose the second row in A to get [a 21 a 22 a 23 ] T , then concatenate it with C according to the first dimension and fill the vacancies with 0, so we get:

[0016]

[0017] That is, the image feature matrix containing the carrier's posture and motion information is obtained.

[0018] Since the feature map obtained after the input image is extracted through the convolutional neural network is of small size, in order to restore the image to its original size for further calculation, it is necessary to use a multi-level upsampling mechanism to restore the image matrix. Based on the original image pixels, a suitable interpolation algorithm is used to insert new elements between the pixels to realize the image mapping operation from small resolution to large resolution, so as to complete the final deblurred image feature matrix containing the carrier posture motion information.

[0019] This invention achieves the goal of image deblurring while preserving carrier pose motion information. This method uses a method of encoding carrier pose motion information during downsampling. This method preserves carrier pose motion information while deblurring the image, significantly reducing the likelihood of positional confusion or loss caused by downsampling during image deblurring and improving the certainty of carrier pose motion information.

[0020] (3) Main advantages of the present invention

[0021] The above-described technical solution of the present invention has the following advantages: The present invention aims to deblur remote sensing platform images while simultaneously integrating carrier pose motion information. During the image deblurring process, a small-size image matrix is ​​superimposed on the carrier pose motion information matrix, followed by upsampling. This deblurs the remote sensing platform image while preserving the carrier pose motion information. The present invention integrates an image deblurring algorithm to more comprehensively consider the carrier pose motion information of globally blurred images, resulting in images with carrier pose motion information having greater robustness and improved applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a framework diagram of the implementation process of the present invention; DETAILED DESCRIPTION

[0023] In order to make the technical solutions, advantages and purposes of the present invention more clear, the method flow is explained in combination with specific examples and with reference to the accompanying drawings to further illustrate the technical solutions of the present invention.

[0024] Example 1 of the present invention is an image deblurring algorithm that integrates carrier posture and motion information, see Figure 1 , follow these steps:

[0025] A blurred image dataset containing carrier attitude motion information is constructed. For typical moving carrier attitudes, such as spacecraft, aircraft, and remote sensing platforms like drones, a full-physics simulation system for spacecraft attitude control is used to achieve full-physics simulation of the spacecraft attitude control system. By fully integrating the real hardware components of the attitude control system, the carrier attitude motion is simulated. In this embodiment, a three-axis air-floating platform is used to simulate carrier attitude motion. An optical camera is attached to the three-axis air-floating platform to capture a fixed scene, and the motion information of the three-axis air-floating platform is recorded during the capture. It should be noted that, assuming that the motion between the camera and the captured object is uniform linear motion, the mathematical expression for the uniformly distributed blur kernel motion is as follows:

[0026]

[0027] Among them, θ represents the angle between the direction of relative motion between the target and the device and the imaging screen; m represents the movement size of the target pixel point during the camera shooting process. The data is then sorted and randomly divided into training set, validation set and test set in a ratio of 7:2:1.

[0028] Use convolutional neural network to extract local features of single-channel images. Generally, convolution calculation output is calculated as follows:

[0029]

[0030] Where N is the length of one side of the input N×N image, Pad is the padding number, kernal is the number of single sides of the convolution kernel (usually 3×3), and stride is the moving step size of the kernal.

[0031] The local features of the blurred image are extracted through the convolutional neural network, and the image small-size feature matrix is ​​obtained using a multi-level downsampling mechanism. The carrier posture motion information is vectorized and encoded to obtain the carrier posture motion information encoding matrix. To express, concatenate matrices A and B:

[0032]

[0033] The splicing algorithm is as follows:

[0034] First, concatenate the first row of A with B along the second dimension to get:

[0035]

[0036] Next, transpose the second row in A to get [a 21 a 22 a 23 ] T , then concatenate it with C according to the first dimension and fill the vacancies with 0, so we get:

[0037]

[0038] That is, the image feature matrix containing the carrier's posture and motion information is obtained.

[0039] After the input image is extracted through the convolutional neural network, the resulting feature map is of small size. In order to restore the image to its original size for further calculations, it is necessary to use a multi-level upsampling mechanism to restore the image matrix and realize the operation of mapping the image from small resolution to large resolution to complete the final deblurred image feature matrix containing position information.

[0040] The contents not described in detail in the specification of the present invention belong to the prior art known to those skilled in the art.

Claims

1. An image deblurring algorithm that integrates carrier posture and motion information, characterized by: Construct a fuzzy image dataset containing carrier posture motion information, vectorize the carrier posture motion information, and perform multi-level downsampling on the image data to obtain a small-size feature matrix, which is then fused with the carrier posture motion information. Then, a multi-level upsampling mechanism is used to restore the size of the spliced ​​image feature matrix. The specific steps are as follows: (1) Constructing a fuzzy image dataset containing carrier posture motion information An air-bearing platform is used to simulate the three-axis posture motion of the carrier. An optical camera is fixed to the air-bearing platform to shoot the target scene. The motion information of the air-bearing platform is recorded synchronously. By shooting different scenes several times, a fuzzy image dataset coupled with the carrier posture motion information is constructed. (2) Vectorized encoding of carrier posture motion information Construct a one-dimensional empty list in the pitch and yaw directions, elementize the carrier attitude motion information according to the sampling period, fill it into the empty list, and fill the vacant positions of the generated motion information matrix with 0 to meet the requirements of matching the dimension of the small-size feature matrix of the image. Then, vectorize the motion information list and obtain the carrier attitude motion information matrix: (3) Obtaining a small-size feature matrix of the image based on a multi-level downsampling mechanism The convolutional neural network multi-level downsampling mechanism is used to extract the small-size feature matrix of the blurred image. For an image with a pixel size of S*P, the small-size feature matrix of the image is obtained through the multi-level downsampling mechanism: where b ij (i,j≤n) is a feature point; (4) Fusion of the image's small-size feature matrix and the carrier's posture and motion information matrix Taking n=3, m=3 as an example, the obtained carrier posture motion information matrix A is concatenated with the image small-size feature matrix B: The splicing algorithm is as follows: First, concatenate the first row of A with B along the second dimension to get: Next, transpose the second row in A to get [a 21 a 22 a 23 ] T , then concatenate it with C according to the first dimension and fill the vacancies with 0, so we get: That is, the image feature matrix containing the carrier posture motion information is obtained; (5) Restoring the small-size feature matrix of the spliced ​​image Based on the multi-level upsampling mechanism, the interpolation method is used to enlarge the image. That is, new elements are inserted between the pixels of the original image using a suitable interpolation algorithm to realize the mapping operation of the image from small resolution to large resolution, so as to complete the final image deblurring operation containing the carrier posture motion information.

Images