Unmanned aerial vehicle image restoration method, system and device based on inertial sensor

Abstract

The invention provides an image restoration method based on an inertial sensor under vibration of an unmanned aerial vehicle. The technology is specially used for carrying out optimization processing on images shot by the unmanned aerial vehicle in a vibration environment. The quality of an image damaged by vibration of the unmanned aerial vehicle is recovered and improved by using the inertial sensor. The method comprises the following steps: acquiring angular velocity and acceleration information of the unmanned aerial vehicle in real time by using an inertial sensor within exposure time of an image acquisition device; mapping the movement track of the unmanned aerial vehicle in the three-dimensional space to an image plane according to the angular velocity and acceleration information of the unmanned aerial vehicle to form a two-dimensional image point movement track; generating a corresponding point spread function (PSF) for each image point in the image according to the two-dimensional image point movement track obtained by mapping; and performing restoration processing on an image acquired by the unmanned aerial vehicle by using the generated point spread function so as to eliminate motion blur and improve image quality.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular provides a method, system and device for restoring an image of a drone based on an inertial sensor, aiming at the problem of image blur caused by vibration of the drone during flight. Background Art

[0002] In disaster scenarios, such as fire, earthquake and explosion, the use of multi-rotor unmanned aerial vehicles (UAV) has become an important tool for emergency rescue and disaster assessment. Unfortunately, due to the small inertia of aerial photography UAVs, they are easily affected by the environment and produce special vibrations. In these specific situations, UAVs are often affected by environmental factors such as earthquake vibrations, explosion shock waves and thermal updrafts in fires, which cause the UAVs to vibrate violently during flight, resulting in blurred images, which seriously affects the quality and availability of images.

[0003] To mitigate this effect, drones are usually equipped with advanced flight control systems and anti-vibration gimbals, which allow the drone to move in the opposite direction of the attitude change to adjust the camera attitude, thereby effectively offsetting the impact of the drone's attitude change on the optical axis, ensuring that the camera always maintains an ideal horizontal position and guaranteeing the imaging quality of the aerial camera.

[0004] Despite these precautions, the impact of vibration on UAV imaging cannot be completely avoided due to the complexity of environmental conditions, such as low air flow, body vibration, etc. Therefore, developing effective post-image processing technology to compensate for image blur caused by vibration is crucial to improving the quality of UAV images. Summary of the invention

[0005] In view of this, the present invention provides a method, system and device for restoring an image of a UAV based on an inertial sensor.

[0006] One aspect of the present invention provides a method for restoring an image of a drone based on an inertial sensor, the method comprising: using a data acquisition device, including an inertial measurement unit sensor composed of three accelerometers and three gyroscopes, for obtaining the angular velocity and acceleration information of the image acquisition device in a preset exposure time period when the environment in which the image acquisition device is located is vibrating, wherein the inertial measurement unit sensor is rigidly fixedly connected to the image acquisition device, the X, Y, and Z axes of the inertial measurement unit sensor are parallel to the X, Y, and Z axes of the image acquisition device, respectively, and the angular velocity and acceleration information are the moving angular velocity and acceleration of the image acquisition device when the image acquisition device is affected by vibration in the preset exposure time period; generating a motion trajectory of the drone image acquisition device in three-dimensional space according to the angular velocity and acceleration information; generating a motion trajectory of the image points of the drone image acquisition device on the image plane according to the motion trajectory in three-dimensional space, that is, a point spread function of the image points. The image acquired by the drone image acquisition device is restored according to the point spread function.

[0007] According to an embodiment of the present invention, the above-mentioned generation of the moving trajectory of the image points in the image captured by the above-mentioned UAV image acquisition device according to the above-mentioned angular velocity and acceleration information includes: generating the Euler angles of the above-mentioned UAV image acquisition device and the motion trajectory information in three-dimensional space according to the above-mentioned angular velocity and acceleration information; generating the image point displacement trajectory of the UAV image acquisition device on the image plane, that is, the point spread function, according to the above-mentioned Euler angles and the motion trajectory in three-dimensional space.

[0008] According to an embodiment of the present invention, restoring the image acquired by the image acquisition device according to the point spread function includes: restoring the acquired image using a non-blind image restoration algorithm according to the point spread function.

[0009] Another aspect of the present invention provides an unmanned aerial vehicle image restoration system based on an inertial sensor, the system comprising: an image acquisition device for acquiring images; an inertial measurement unit sensor for acquiring angular velocity and acceleration information of the image acquisition device in a preset exposure time period, wherein the inertial measurement unit sensor is rigidly fixedly connected to the image acquisition device, the X, Y, and Z axes of the inertial measurement unit sensor are parallel to the X, Y, and Z axes of the image acquisition device, and the angular velocity and acceleration are the moving angular velocity and acceleration of the image acquisition device; a processor for generating a moving trajectory of an image point in the image acquired by the image acquisition device according to the angular velocity and acceleration information, generating a point spread function of the image point in the image acquired by the image acquisition device according to the moving trajectory of the image point, and restoring the image acquired by the image acquisition device according to the point spread function.

[0010] According to an embodiment of the present invention, the system further includes: a Raspberry Pi 4B, which is used to connect the image acquisition device, the data acquisition device and the processor, and to acquire and store the angular velocity and acceleration information of the image acquisition device in a preset exposure time period acquired by the data acquisition device in real time, so as to facilitate subsequent transmission to the processor.

[0011] Another aspect of the present invention provides an unmanned aerial vehicle image restoration device based on an inertial sensor, the device comprising: a data acquisition device, for an inertial measurement unit sensor to obtain angular velocity and acceleration information of the image acquisition device in a preset exposure time period, wherein the inertial measurement unit sensor is rigidly fixedly connected to the image acquisition device, the X, Y, and Z axes of the inertial measurement unit sensor are parallel to one of the X, Y, and Z axes of the image acquisition device, and the angular velocity and acceleration information are the moving angular velocity and acceleration of the image acquisition device when the image acquisition device is affected by random vibration in the preset exposure time period; an image acquisition device, for acquiring blurred images; a data processing device, for transmitting the data and images acquired by the data acquisition device and the image acquisition device to a processor, constructing a 3D motion model and converting the 3D motion trajectory into a 2D image plane trajectory, and then generating a point spread function of each image point according to the image point trajectory, and finally restoring the image acquired by the image acquisition device according to the point spread function using a total variation regularization image restoration algorithm to obtain a clear image, and using indicators such as SSIM and PSNR to evaluate the quality of the restored image.

[0012] According to an embodiment of the present invention, by using an inertial measurement unit sensor to obtain the angular velocity and acceleration information of the image acquisition device in a preset exposure time period, the angular velocity and acceleration information of the image acquisition device in the preset exposure time period can be obtained when random vibration exists in the environment where the image acquisition device is located. Further, a movement trajectory of image points in the image acquired by the image acquisition device can be generated according to the angular velocity and acceleration information, and a point spread function of the image points in the image acquired by the image acquisition device can be generated according to the movement trajectory of the image points. Then, the image acquired by the image acquisition device can be restored according to the point spread function, thereby ultimately achieving a significant improvement in the quality of the image acquired by the image acquisition device without increasing the volume and cost of the image acquisition device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0014] Figure 1 shows an overall flow chart according to an embodiment of the present invention;

[0015] Figure 2A flow chart of an image acquisition device according to an embodiment of the present invention is shown;

[0016] Figure 3 A flow chart of a data acquisition device according to an embodiment of the present invention is shown;

[0017] Figure 4 A flow chart of a data processing device according to an embodiment of the present invention is shown;

[0018] Figure 5 The figure shows the comparison between the blurred image restored according to the embodiment of the present invention and the restored image. DETAILED DESCRIPTION

[0019] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0021] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0022] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] Drones are affected by environmental factors such as earthquake vibrations, shock waves from explosions, and thermal updrafts from fires, which can cause severe vibrations in the drone during flight, which in turn causes the camera to produce unstable movements such as shaking or rotation. This dynamic disturbance can cause motion blur in the images captured by the drone, causing distortion of image details and affecting image clarity and quality.

[0024] The suppression methods in the related art mainly include mechanical compensation and optical compensation. However, for drone aerial photography tasks in specific scenarios, there will still be residual random vibrations after mechanical compensation or optical compensation. This residual random vibration becomes an important factor affecting the quality of drone aerial images. Based on this, an embodiment of the present invention provides a drone image restoration method based on an inertial measurement unit sensor.

[0025] An embodiment of the present invention provides an image restoration method based on an inertial measurement unit sensor, the method comprising: using an inertial measurement unit sensor 401, which includes three accelerometers and three gyroscopes, to obtain angular velocity and acceleration information of the image acquisition device 30 in a preset exposure time period when there is vibration in the environment where the image acquisition device 30 is located, wherein the inertial measurement unit sensor is rigidly fixedly connected to the image acquisition device, the X, Y, and Z axes of the inertial measurement unit sensor are respectively parallel to the X, Y, and Z axes of the image acquisition device, and the angular velocity and acceleration information are the moving angular velocity and acceleration of the image acquisition device when the image acquisition device is affected by vibration in the preset exposure time period; generating a moving trajectory of an image point in the image acquired by the image acquisition device according to the angular velocity and acceleration information; generating a point spread function of the image point in the image acquired by the image acquisition device according to the moving trajectory of the image point; and restoring the image acquired by the image acquisition device according to the point spread function.

[0026] According to an embodiment of the present invention, by using the inertial measurement unit sensor 401 to obtain the angular velocity and acceleration information of the image acquisition device 30 in a preset exposure time period, the angular velocity and acceleration information of the image acquisition device in the preset exposure time period can be obtained when there is random vibration in the environment where the image acquisition device is located. Further, the image point movement trajectory of the image points in the image acquired by the image acquisition device can be generated according to the angular velocity and acceleration information, and the point spread function of the image points in the image acquired by the image acquisition device can be generated according to the image point movement trajectory. Then, the image acquired by the image acquisition device is restored according to the point spread function, and finally, the image quality acquired by the image acquisition device is significantly improved without increasing the volume and cost of the image acquisition device.

[0027] Figure 2A flow chart of an image acquisition device 30 according to an embodiment of the present invention is shown;

[0028] like Figure 2 As shown, the image acquisition device 30 generates linear displacement and angular displacement due to vibration, thereby causing abnormal image displacement and generating a blurred image 304. The generated blurred image 304 is transmitted to the data processing device 50 through the connection of the Raspberry Pi 4B 60, waiting for image processing and restoration.

[0029] Figure 3 A flow chart of a data acquisition device 40 according to an embodiment of the present invention is shown;

[0030] like Figure 3 As shown, the data acquisition device 40 uses an inertial measurement unit sensor 401 to obtain the angular velocity and acceleration information of the image acquisition device 30 in a preset exposure time period, wherein the inertial measurement unit sensor is rigidly fixedly connected to the image acquisition device, and the X, Y, and Z axes of the inertial measurement unit sensor are respectively parallel to the X, Y, and Z axes of the image acquisition device, and the angular velocity and acceleration information are the moving angular velocity and acceleration of the image acquisition device when the image acquisition device is affected by vibration in the preset exposure time period;

[0031] According to an embodiment of the present invention, the inertial measurement unit (IMU) sensor 401 is a component for measuring angular vibration and linear vibration information.

[0032] According to an embodiment of the present invention, an inertial measurement unit sensor 401 is used to obtain the angular velocity and acceleration information of the image acquisition device 30 in a preset exposure time period, wherein the inertial measurement unit sensor is rigidly fixedly connected to the image acquisition device, and the X, Y, and Z axes of the inertial measurement unit sensor are respectively parallel to the X, Y, and Z axes of the image acquisition device, so that the angular velocity and acceleration information on the three axes can be used to correct the image acquired by the image acquisition device 30 in three dimensions. Since the image acquired by the image acquisition device is a two-dimensional image, correcting the three dimensions of the two-dimensional image can better restore the image and improve the quality of the restored image.

[0033] According to an embodiment of the present invention, by using the inertial measurement unit sensor 401 to obtain the angular velocity and acceleration information of the image acquisition device 30 in a preset exposure time period, the inertial measurement unit sensor can perform high-precision measurement of the instantaneous angular velocity and acceleration of the image acquisition device when the image acquisition device is affected by abnormal random vibration, and output angular velocity and acceleration information with high accuracy, stable output and high reliability.

[0034] According to an embodiment of the present invention, the obtained angular velocity and acceleration data are transmitted to the data processing device 50 through the Raspberry Pi 4B 60 to wait for the next step of processing.

[0035] Figure 4 A flow chart of a data processing device 50 according to an embodiment of the present invention is shown;

[0036] like Figure 4 As shown, the image processing method includes the following steps: obtaining a blurred image 304 and acceleration and angular velocity information 402 during the exposure time. The blurred image is an image collected by the image acquisition device 30 during a preset exposure time period, and the acceleration and angular velocity information is the angular velocity and acceleration data of the image acquisition device during the preset exposure time period obtained by the inertial measurement unit sensor 401. The processor 501 receives the blurred image and the acceleration and angular velocity information. The processor is responsible for processing these data to prepare for subsequent steps. Constructing a 3D motion model and converting the 3D motion trajectory into a 2D image plane trajectory. This step converts the collected three-dimensional motion data into trajectory information on a two-dimensional plane to provide necessary motion information for the subsequent image restoration process. Generate a point spread function 503. According to the constructed 3D motion model and the 2D image plane trajectory, generate a corresponding point spread function to describe the degree and direction of image blur. Total variation regularization image restoration algorithm. Using the generated point spread function and total variation regularization algorithm, the blurred image is restored to obtain a clear image 505. The quality of the restored image is evaluated using indicators such as SSIM and PSNR. The quality of the restored image is evaluated using indicators such as structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) to verify the effectiveness of the restoration algorithm and the quality of the restored image.

[0037] The moving trajectory of the image points in the image captured by the image acquisition device 30 is generated according to the angular velocity and acceleration information.

[0038] According to an embodiment of the present invention, when affected by random vibration within a preset exposure time period, the actual imaging point position of the target point in the image acquisition device 30 will be offset from the ideal imaging point position, and the distance between the actual imaging point position corresponding to the target point and the ideal imaging point position is discretized into multiple parts, and the discretized image point offset distance is used as the image point movement trajectory of the image point.

[0039] A point spread function of the image points in the image captured by the image capture device 30 is generated according to the image point movement trajectory.

[0040] According to an embodiment of the present invention, the point spread function may characterize the intensity of light energy left on an image point corresponding to the point light source on the focal plane during the movement of the point light source relative to the focal plane.

[0041] The image captured by the image acquisition device 30 is restored according to the point spread function.

[0042] According to an embodiment of the present invention, the image captured by the image acquisition device 30 is restored by using a point spread function in combination with a non-blind image restoration algorithm.

[0043] According to an embodiment of the present invention, the non-blind image restoration algorithm may be, for example, a total variation regularized image restoration algorithm, an algorithm based on a total variation model, or an algorithm based on a sparse representation, etc. The present invention does not limit the specific non-blind image restoration algorithm, and it may be selected according to specific circumstances during actual application.

[0044] According to an embodiment of the present invention, by using the inertial measurement unit sensor 401 to obtain the angular velocity and acceleration information of the image acquisition device 30 in a preset exposure time period, the angular velocity and acceleration information of the image acquisition device in the preset exposure time period can be obtained when there is abnormal random vibration in the environment where the image acquisition device is located. Further, the movement trajectory of the image points in the image acquired by the image acquisition device can be generated according to the angular velocity and acceleration information, and the point spread function of the image points in the image acquired by the image acquisition device can be generated according to the movement trajectory of the image points. Then, the image acquired by the image acquisition device is restored according to the point spread function, and finally the quality of the image acquired by the image acquisition device is significantly improved without increasing the volume and cost of the image acquisition device.

[0045] According to an embodiment of the present invention, the imaging process of the image acquisition device 30 is regarded as pinhole imaging. If the image acquisition device moves during the exposure time, the point light source in the scene will be projected to different positions of the sensor. The final trajectory left on the sensor is the PSF of the point. The projections of these different perspectives are integrated to generate a blurred image 304, which is related to the initial projection or potential clear image 505 through the homography matrix.

[0046]

[0047] Among them, g, f, n in the formula represent the ordered vectors of blurred image, clear image and Gaussian noise respectively, and H t is the 3×3 homography matrix at instant t; [0,T] is the exposure time, and x is the uniform pixel coordinate.

[0048] In discrete form, the blurred image is the weighted sum of N projected transformations of the sharp image during the exposure period, as shown below.

[0049]

[0050] w in the formula kRepresents weight, k=1....N satisfies∑ k w k =1 means in camera pose H k In the case where the sensor measures camera motion, w k It represents the sampling interval, which is equal to 1 / N and is used for uniform sampling of inertial sensors. It can be seen that the blurred image can be represented by the sum of multiple frames of low-resolution images with relative displacement, or it can be determined by the average of the cumulative sum of multiple frames of clear images with relative displacement. Each clear image can be obtained by the initial clear image f(x) through the plane projection transformation homography matrix H k get.

[0051] The homography matrix is ​​a linear mapping, which can be used to project a target point in the three-dimensional (3D) object space to K different positions on the image plane.

[0052] (x k ,y k ,1) T =H k (x0,y0,1) T

[0053] H in the formula k The rotation matrix R is obtained by the two internal parameters of the camera and the inertial sensor measurement k and the translation vector t k It consists of, in xyz convention, the rotation matrix R k Defined as

[0054] R k =R z (γ k )R y (β k )R x (α k )

[0055] where α k ,β k ,γ k It represents the Euler angle obtained by the angular velocity of the camera during exposure measured by the IMU, [R x ,R y ,R z ] represents the rotation matrix of each axis of the camera. Its representation is as follows:

[0056]

[0057] After obtaining the rotation matrix of the camera at each sampling interval, the accelerometer information at each sampling interval is converted to the reference coordinate system to calculate the relative translation of the camera:

[0058]

[0059] where a k =[a kx ,a ky ,a kz ] is the three-axis acceleration of the camera during the exposure period.

[0060] After projecting the world point (X, Y, Z) to K different locations (x k ,y k ), assume that K different poses are sampled by inertial measurements. The motion trajectory is directly generated by the inertial measurements and depends on their accuracy. Ideally, the projected motion on the image plane is equal to the PSF of the camera motion. The PSF of all images is expressed as:

[0061]

[0062] According to an embodiment of the present invention, for example, the image captured by the image acquisition device 30 may be restored based on a point spread function using a total variation regularized image restoration algorithm in a non-blind image restoration algorithm.

[0063] According to an embodiment of the present invention, the image degradation process can be represented by a mathematical model of the following formula.

[0064]

[0065] Among them is a column vector obtained by arranging the observed blurred image in column order; is the column vector obtained by arranging the original clear image in column order; is the fuzzy operator, which is a circulant matrix constructed based on PSF; It is a column vector obtained by arranging the noise matrix in column order.

[0066] According to an embodiment of the present invention, based on the image degradation model in the above formula, the objective function E(u) of the restoration optimization model of the total variation regularized image restoration algorithm is given in the following formula.

[0067]

[0068] Among them, the first item is the fidelity item, which ensures that the distance between the original clear image and the observed image after blurring is small enough; the second item is the regularization item; ||.||2 means finding the vector two norm; Represents the gradient of the image in the horizontal and vertical directions; μ is the regularization term coefficient, which is used to balance the fidelity term and the regularization term.

[0069] The point spread function h obtained above and the image u0 captured by the image acquisition device 30 can be substituted into the above formula, and the objective function E(u) is minimized for solving, so as to obtain the restored image u.

[0070] Figure 5 A blurred image 701 and a restored image 702 according to an embodiment of the present invention are shown.

[0071] Figure 5 The restored image 702 in FIG. 1 is a restored image according to the restoration method provided by the embodiment of the present invention. Figure 5 The image obtained by restoring the blurred image 701 in the figure can be seen from the restored image 702 that the image edges and numbers in the restored image 702 are already very clear and can be well recognized, indicating that the restoration method provided by the embodiment of the present invention can better restore the blurred image caused by random vibration and improve the image quality.

[0072] According to an embodiment of the present invention, the above-mentioned drone image restoration device based on an inertial measurement unit sensor can be used to implement the above-mentioned drone image restoration method based on an inertial measurement unit sensor. For the description of the drone image restoration device based on an inertial measurement unit sensor, reference can be made to the above-mentioned description of the drone image restoration method based on an inertial measurement unit sensor, which will not be repeated here.

[0073] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for restoring drone images based on inertial sensors, characterized in that: The method comprises: By using an inertial measurement unit sensor, when an aerial photography drone is easily affected by the environment and vibrates due to its small inertia, the angular velocity and acceleration information of an image acquisition device carried by the drone during the exposure time is obtained, wherein the inertial measurement unit is rigidly fixedly connected to the drone image acquisition device, and each axis of the XYZ three axes of the inertial measurement unit sensor is parallel to the XYZ three axes of the drone image acquisition device, and the angular velocity and acceleration information reflects the changes in linear displacement and angular displacement of the drone image acquisition device caused by the vibration of the drone during the exposure period. Generate a motion trajectory of the drone image acquisition device in three-dimensional space under vibration according to the angular velocity and acceleration information; A total variation regularized image restoration algorithm based on the inertial sensor to obtain the UAV motion information is used to process the image blur problem caused by the UAV vibration; Generate a motion trajectory of an image point of the UAV image acquisition device on a two-dimensional image plane according to the motion trajectory of the UAV image acquisition device and its internal parameter coefficient; Generate a point spread function of an image point in an image acquired by an image acquisition device of a drone according to the image point motion trajectory; The image captured by the drone image acquisition device under vibration is restored according to the point spread function.

2. According to the method of claim 1, the angular velocity and acceleration information generate a motion trajectory of the drone image acquisition device in three-dimensional space under vibration, comprising: Generate a homography transformation matrix of the image acquisition device on a two-dimensional image plane according to the image acquisition device of the drone and its internal parameter coefficients; The moving trajectory of the image points in the image collected by the drone image acquisition device is generated according to the homography transformation matrix.

3. According to the method of claim 2, the homography transformation matrix generates the moving trajectory of the image points in the image collected by the drone image acquisition device, comprising: Integrating the angular velocity information within the exposure time to generate an angular displacement of the image acquisition device, and calculating the Euler angle information within the exposure time based on the angular displacement information; Calculating the three-dimensional motion trajectory of the image acquisition device within the exposure time using the Euler angle information; The moving trajectory of the image points in the image collected by the UAV image acquisition device is generated according to the three-dimensional motion trajectory and the Euler angle information.

4. According to the method of claim 3, the three-dimensional motion trajectory and Euler angle information generate the image point movement trajectory in the image collected by the drone image acquisition device, comprising: The three-dimensional motion trajectory, Euler angle information, the focal length of the image acquisition device and its internal parameter coefficients are used to generate the homography matrix of the image acquisition device, and the three-dimensional motion trajectory of the image acquisition device is mapped to the image plane using the homography matrix to generate the movement trajectory of the image point on the image.

5. According to the method of claim 1, the image point movement trajectory generates a point spread function of the image point in the image captured by the image acquisition device, comprising: Generating a motion trajectory of the drone image acquisition device in three-dimensional space according to the angular velocity and acceleration information; Generate a homography transformation matrix of the image acquisition device on a two-dimensional image plane according to the image acquisition device of the drone and its internal parameter coefficients; The moving trajectory of the image points in the image collected by the unmanned aerial vehicle image acquisition device, that is, the point spread function of the image points, is generated according to the homography transformation matrix.

6. According to the method of claim 1, the total variation regularized image restoration algorithm for obtaining the UAV motion information by the inertial sensor comprises: The gradient and of the original image are introduced as regularization terms into the image restoration optimization model, and the objective function of the image restoration optimization model is as follows: Among them, in the above objective function, the left side of the plus sign is the fidelity term, the right side of the plus sign is the regularization term, μ represents the regularization term coefficient, which is used to control the weight ratio between the fidelity term and the regularization term, || ||2 represents the vector bi-norm; min represents the minimum value of the fidelity term.

7. An unmanned aerial vehicle image restoration system based on inertial sensors, characterized in that: The system comprises: An unmanned aerial vehicle image acquisition device, used for acquiring images; An inertial measurement unit sensor, comprising three accelerometers and three gyroscopes, for obtaining angular velocity and acceleration information of the image acquisition device in a preset exposure time period when the environment in which the image acquisition device is located is subject to vibration, wherein the inertial measurement unit sensor is rigidly fixedly connected to the image acquisition device, the X, Y, and Z axes of the inertial measurement unit sensor are respectively parallel to the X, Y, and Z axes of the image acquisition device, and the angular velocity and acceleration information are the moving angular velocity and acceleration of the image acquisition device when the image acquisition device is affected by vibration during the preset exposure time period; The processor is used to generate a moving trajectory of image points of the image captured by the image acquisition device according to the angular velocity and acceleration information, generate a point spread function of the image points in the image captured by the image acquisition device according to the moving trajectory of the image points, and perform restoration based on the captured image.

8. The system according to claim 7, characterized in that The system further comprises: Raspberry Pi 4B is used to connect to the inertial measurement unit sensor, collect and store the angular velocity and acceleration information of the image acquisition device during the exposure period obtained by the inertial measurement unit sensor into the internal flash, and transmit it to the processor after the drone aerial photography mission is completed.

9. An unmanned aerial vehicle image restoration device based on an inertial sensor, characterized in that: The device comprises: An angular velocity and acceleration information acquisition module, used for an inertial measurement unit sensor to acquire angular velocity information of the unmanned image acquisition device during exposure when the environment in which the unmanned image acquisition device is located has abnormal vibration, wherein the inertial measurement unit sensor is rigidly fixedly connected to the unmanned image acquisition device, and the X, Y, and Z axes of the inertial measurement unit sensor are respectively parallel to the X, Y, and Z axes of the image acquisition device, and the angular velocity and acceleration information are the moving angular velocity and acceleration of the image acquisition device when the image acquisition device is affected by vibration during the preset exposure time period; An image point movement trajectory generating module, used to generate a movement trajectory of an image point in an image captured by the drone image acquisition device according to the angular velocity and acceleration information; A point spread function generation module, used to generate a point spread function of an image point in the image collected by the drone image acquisition device according to the image point movement trajectory; The image restoration module is used to restore the image collected by the UAV image acquisition device according to the point spread function using the total variation regularized image restoration algorithm in the non-blind image restoration algorithm.

Images