Image processing methods, apparatus, aircraft, and computer-readable storage media

Abstract

This application discloses an image processing method, apparatus, aircraft, and computer-readable storage medium. The method includes: acquiring a first RS image and a second RS image; acquiring first IMU data corresponding to the first RS image and second IMU data corresponding to the second RS image based on the IMU unit; performing spatiotemporal alignment of the rows of the first RS image based on the first IMU data and the second IMU data, and compensating the spatiotemporally aligned RS image to obtain a third RS image; and determining a target sharp image corresponding to the first RS image based on the third RS image and the second RS image. This application can reduce the RS rolling shutter effect and motion blur effect of normal RS images by using RS images with the lowest exposure time, thereby improving the sharpness of RS images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, aircraft, and computer-readable storage medium. Background Art

[0002] Traditional cameras are mainly divided into GS (Global Shutter) cameras and RS (Rolling Shutter) cameras. GS cameras capture images where all pixels are exposed simultaneously, directly reflecting rich scene information. RS cameras use a line-by-line exposure method, exposing at fixed time intervals from the first line to the last (or from the last line to the first), with each line having an equal exposure time. Therefore, the exposure range of each line is different. When objects move quickly, this can easily cause deformation, resulting in a degraded visual effect; this phenomenon is also known as the rolling shutter effect.

[0003] The jelly effect limits the application of RS cameras in visual positioning and mapping and visual navigation, making them suitable only for low-speed, stable scenarios (such as park robots and indoor cleaning robots) and difficult to apply in large, high-speed scenarios (such as large aircraft, high-speed autopilots, and large agricultural machinery). Summary of the Invention

[0004] The main objective of this application is to provide an image processing method, apparatus, aircraft, and computer-readable storage medium, aiming to solve the technical problem that RS cameras are difficult to apply to high-speed scenes due to the rolling shutter effect.

[0005] To achieve the above objectives, this application provides an image processing method, comprising:

[0006] Acquire a first RS image captured by the rolling shutter RS ​​camera before the minimum exposure time is set, and a second RS image captured by the RS camera after the minimum exposure time is set, wherein the RS camera includes an IMU unit and an ISP unit, and the ISP unit periodically sets the minimum exposure time corresponding to the RS camera;

[0007] Based on the IMU unit, first IMU data corresponding to the first RS image and second IMU data corresponding to the second RS image are obtained;

[0008] Based on the first IMU data and the second IMU data, the rows of the first RS image are spatiotemporally aligned, and the spatiotemporally aligned RS image is compensated to obtain the third RS image;

[0009] Based on the third RS image and the second RS image, a clear target image corresponding to the first RS image is determined.

[0010] Further, the step of performing spatiotemporal alignment of the rows of the first RS image based on the first IMU data and the second IMU data includes:

[0011] Based on the first IMU data and the second IMU data, the target pose transformation parameters are determined;

[0012] Based on the target pose transformation parameters, the rows of the first RS image are spatiotemporally aligned to obtain a spatiotemporally aligned RS image.

[0013] Further, the step of determining the target pose transformation parameters based on the first IMU data and the second IMU data includes:

[0014] Obtain the pose transformation parameters between the first IMU data and the second IMU data;

[0015] The pose transformation parameters are interpolated to obtain the target pose transformation parameters.

[0016] Furthermore, the step of compensating the spatiotemporally aligned RS image to obtain a third RS image includes:

[0017] Obtain the camera pose data corresponding to the first RS image;

[0018] Based on the camera pose data, the spatiotemporally aligned RS image is compensated to obtain the third RS image.

[0019] Furthermore, the step of compensating the spatiotemporally aligned RS image based on the camera pose data to obtain the third RS image includes:

[0020] Obtain the exposure time of each row in the first RS image;

[0021] Based on the camera pose data and the exposure time, determine the camera rotation increment between each row and the starting row in the first RS image;

[0022] Based on the camera rotation increment, the spatiotemporally aligned RS image is compensated to obtain the third RS image.

[0023] Further, the step of determining the target clear image corresponding to the first RS image based on the third RS image and the second RS image includes:

[0024] Based on the second RS image and the third RS image, determine the fuzzy kernel function;

[0025] Based on the blur kernel function and the third RS image, the target clear image is determined.

[0026] Further, the step of determining the blur kernel function based on the second RS image and the third RS image includes:

[0027] Perform a Fast Fourier Transform on the second RS image to obtain the first transform result;

[0028] Perform a Fast Fourier Transform on the third RS image to obtain the second transform result;

[0029] Based on the first transformation result and the second transformation result, the fuzzy kernel function is determined.

[0030] Further, the step of determining the fuzzy kernel function based on the first transformation result and the second transformation result includes:

[0031] Based on the first transformation result and the second transformation result, determine the Fourier transform result corresponding to the fuzzy kernel function;

[0032] The fuzzy kernel function is obtained by performing an inverse Fourier transform on the Fourier transform result corresponding to the fuzzy kernel function.

[0033] Further, the step of determining the target clear image based on the blur kernel function and the third RS image includes:

[0034] Based on the blur kernel function, the third RS image is spatially convolved or deconvolved to obtain the clear image of the target.

[0035] Furthermore, to achieve the above objectives, this application also provides an aircraft equipped with a rolling shutter RS ​​camera. The RS camera includes an IMU unit and an ISP unit, wherein the ISP unit periodically sets the minimum exposure time corresponding to the RS camera. The aircraft includes:

[0036] The image acquisition module is used to acquire a first RS image captured by the rolling shutter RS ​​camera before the minimum exposure time is set, and a second RS image captured by the RS camera after the minimum exposure time is set;

[0037] The data acquisition module is used to acquire, based on the IMU unit, the first IMU data corresponding to the first RS image and the second IMU data corresponding to the second RS image;

[0038] The alignment compensation module is used to perform spatiotemporal alignment of the rows of the first RS image based on the first IMU data and the second IMU data, and to compensate the spatiotemporally aligned RS image to obtain a third RS image.

[0039] The determining module is used to determine the target clear image corresponding to the first RS image based on the third RS image and the second RS image.

[0040] In addition, to achieve the above objectives, this application also provides an image processing apparatus, which includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the steps of the aforementioned image processing method.

[0041] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the steps of the aforementioned image processing method.

[0042] This application acquires a first RS image captured by a rolling shutter RS ​​camera before setting the minimum exposure time, and a second RS image captured by the RS camera after setting the minimum exposure time. Then, based on the IMU unit, it acquires first IMU data corresponding to the first RS image and second IMU data corresponding to the second RS image. Next, based on the first and second IMU data, it performs spatiotemporal alignment on the rows of the first RS image and compensates for the spatiotemporally aligned RS image to obtain a third RS image. Finally, based on the third RS image and the second RS image, it determines the target clear image corresponding to the first RS image. This method can reduce the RS rolling shutter effect and motion blur effect of normal RS images by using RS images with the minimum exposure time, thereby improving the clarity of RS images.

[0043] Meanwhile, this image processing method can be applied to airborne vision systems of large aircraft and large mechanical equipment with high-intensity vibrations. By correcting the original image, this image processing method can be applied to visual navigation and other visual applications. In other words, this image processing method is not limited to visual navigation. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of the image processing device in the hardware operating environment involved in the embodiments of this application;

[0045] Figure 2 This is a flowchart illustrating the first embodiment of the image processing method of this application;

[0046] Figure 3 This is a scene diagram illustrating an embodiment of the image processing method of this application;

[0047] Figure 4 This is a schematic diagram of the functional modules of an embodiment of the aircraft of this application.

[0048] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0050] like Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of an image processing device in the hardware operating environment involved in the embodiments of this application.

[0051] The image processing device in this application embodiment can be an aircraft or vehicle, such as a large aircraft, a high-speed autonomous vehicle, or large agricultural machinery. Figure 1 As shown, the image processing device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0052] Optionally, the image processing device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. These sensors may include, for example, light sensors, motion sensors, and other sensors. Of course, the image processing device may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated upon here.

[0053] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the image processing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an image processing program.

[0055] exist Figure 1 In the image processing apparatus shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user terminal) and communicate data with the client; and the processor 1001 can be used to call the image processing program stored in the memory 1005.

[0056] In this embodiment, the image processing apparatus includes: a memory 1005, a processor 1001, and an image processing program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the image processing program stored in the memory 1005, it executes the steps of the image processing methods in the following embodiments.

[0057] This application also provides an image processing method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the image processing method of this application.

[0058] The image processing method includes:

[0059] Step S101: Acquire a first RS image captured by the rolling shutter RS ​​camera before the minimum exposure time is set, and a second RS image captured by the RS camera after the minimum exposure time is set. The RS camera includes an IMU unit and an ISP unit, and the ISP unit sets the minimum exposure time corresponding to the RS camera at regular intervals.

[0060] The image processing device or aircraft is equipped with an RS camera and a main control module. The RS camera includes an IMU (Inertial Measurement Unit) unit, an ISP (Image Signal Processing) unit, and a camera. The parameters of the IMU unit and the ISP unit can be set through the main control module. The ISP unit sets the minimum exposure time corresponding to the RS camera at regular intervals. The minimum exposure time can be 5-10ms, and the timed setting of the minimum exposure time can be 1s.

[0061] After detecting that the minimum exposure time corresponding to the RS camera is set by the ISP unit, a second RS image captured by the RS camera after the minimum exposure time is set can be acquired. For example, when the exposure of the second RS image is completed, the second RS image is acquired, that is, the second RS image is an RS image obtained by exposure using the minimum exposure time. The first RS image captured by the rolling shutter RS ​​camera before the minimum exposure time is also acquired. Since the rolling effect and motion blur effect of the RS camera are greatly reduced at the minimum exposure time, the rolling effect and motion blur effect of the second RS image are much smaller than those of the first RS image.

[0062] It should be noted that when the second RS image exposure is complete, the RS camera's exposure time can be set to the normal exposure time (the exposure time corresponding to the first RS image). The second RS image can be any RS image obtained using the normal exposure time between the current setting of the minimum exposure time and the previous setting of the minimum exposure time.

[0063] Step S102: Based on the IMU unit, acquire the first IMU data corresponding to the first RS image and the second IMU data corresponding to the second RS image;

[0064] After acquiring the first RS image and the second RS image, the first IMU data corresponding to the first RS image and the second IMU data corresponding to the second RS image are acquired based on the IMU unit. Specifically, the first IMU data is acquired in the IMU unit according to the exposure time of each row in the first RS image, and the second IMU data is acquired in the IMU unit according to the exposure time of each row in the second RS image.

[0065] Step S103: Based on the first IMU data and the second IMU data, perform spatiotemporal alignment on the rows of the first RS image, and compensate the spatiotemporally aligned RS image to obtain the third RS image;

[0066] After acquiring the first IMU data and the second IMU data, the rows of the first RS image are spatiotemporally aligned based on the first IMU data and the second IMU data to obtain the spatiotemporally aligned RS image. Specifically, the target pose transformation parameters are first determined based on the first IMU data and the second IMU data, that is, the target pose transformation parameters are obtained based on the IMU data corresponding to each row in the first RS image and the IMU data corresponding to each row in the second RS image. Then, the rows of the first RS image are spatiotemporally aligned based on the target pose transformation parameters. Specifically, the spatiotemporal alignment algorithm in related technologies can be used to spatiotemporally align the rows of the first RS image to obtain the spatiotemporally aligned RS image.

[0067] After obtaining the spatiotemporally aligned RS image, compensation is performed on the spatiotemporally aligned RS image to obtain the third RS image. Specifically, the camera pose data corresponding to the first RS image can be obtained, that is, the RS camera pose data at the exposure time of each row in the first RS image. Based on the camera pose data, the spatiotemporally aligned RS image is compensated row by row to obtain the third RS image, so as to eliminate the RS rolling shutter effect of the first RS image through spatiotemporal alignment and compensation.

[0068] Step S104: Based on the third RS image and the second RS image, determine the target clear image corresponding to the first RS image.

[0069] After obtaining the third RS image, the target clear image corresponding to the first RS image is determined based on the third RS image and the second RS image. Specifically, a blur kernel function can be determined through the third RS image and the second RS image, and the third RS image is processed by the blur kernel function to obtain the target clear image, thereby reducing the motion blur effect and improving the clarity of the target clear image.

[0070] By acquiring a first RS image captured by a rolling shutter RS ​​camera before setting the minimum exposure time, and a second RS image captured by the RS camera after setting the minimum exposure time, the system then acquires first IMU data corresponding to the first RS image and second IMU data corresponding to the second RS image based on the IMU unit. Next, based on the first and second IMU data, the rows of the first RS image are spatiotemporally aligned, and the spatiotemporally aligned RS image is compensated to obtain a third RS image. Finally, based on the third RS image and the second RS image, a target clear image corresponding to the first RS image is determined. This method can reduce the RS rolling shutter effect and motion blur effect of normal RS images by using RS images with the minimum exposure time, thereby improving the clarity of RS images.

[0071] Meanwhile, this image processing method can be applied to airborne vision systems of large aircraft and large mechanical equipment with high-intensity vibrations. By correcting the original image, this image processing method can be applied to visual navigation and other visual applications. In other words, this image processing method is not limited to visual navigation.

[0072] Based on the first embodiment, a second embodiment of the image processing method of this application is proposed, wherein step S103 includes:

[0073] Step S201: Determine the target pose transformation parameters based on the first IMU data and the second IMU data;

[0074] Step S202: Based on the target pose transformation parameters, perform spatiotemporal alignment on the rows of the first RS image to obtain a spatiotemporally aligned RS image.

[0075] After acquiring the first IMU data and the second IMU data, the target pose transformation parameters are determined based on the first IMU data and the second IMU data. Specifically, the target pose transformation parameters are determined according to the first IMU data and the second IMU data, that is, the target pose transformation parameters are obtained according to the IMU data corresponding to each row in the first RS image and the IMU data corresponding to each row in the second RS image. Further, in a possible implementation, this step S201 includes:

[0076] Step S201: Obtain pose transformation parameters between the first IMU data and the second IMU data;

[0077] Step S201: Perform an interpolation operation on the pose transformation parameters to obtain the target pose transformation parameters.

[0078] It should be noted that the RS image is calculated at 30Hz and 1080 lines, which is 32,400 lines per second, while the IMU data is calculated at a maximum of 1000Hz. Therefore, the IMU sampling frequency cannot reach the level of each line of the RS image, so interpolation processing is required.

[0079] After acquiring the first IMU data and the second IMU data, the pose transformation parameters between the first IMU data and the second IMU data are obtained. Based on the IMU data corresponding to each row in the first RS image and the IMU data corresponding to each row in the second RS image, the pose transformation parameters are obtained. The parameter of each row in the pose transformation parameters can be the difference between the IMU data corresponding to each row in the first IMU data and the IMU data corresponding to each row in the second IMU data.

[0080] After obtaining the pose transformation parameters, interpolation is performed on the pose transformation parameters to obtain the target pose transformation parameters. Specifically, the B-spline method (model) can be used to interpolate the pose transformation parameters.

[0081] After obtaining the target pose transformation parameters, the rows of the first RS image are spatiotemporally aligned based on the target pose transformation parameters. Specifically, the spatiotemporal alignment algorithm in related technologies can be used to spatiotemporally align the rows of the first RS image to obtain the spatiotemporally aligned RS image.

[0082] The target pose transformation parameters are determined based on the first IMU data and the second IMU data. Then, based on the target pose transformation parameters, the rows of the first RS image are spatiotemporally aligned to obtain a spatiotemporally aligned RS image. The first RS image can be accurately spatiotemporally aligned using the pose transformation parameters between the first IMU data and the second IMU data to reduce the RS rolling shutter effect of the image.

[0083] Based on the first embodiment, a third embodiment of the image processing method of this application is proposed, wherein step S103 includes:

[0084] Step S301: Obtain camera pose data corresponding to the first RS image;

[0085] Step S302: Based on the camera pose data, compensate the spatiotemporally aligned RS image to obtain the third RS image.

[0086] After obtaining the spatiotemporally aligned RS image, the camera pose data corresponding to the first RS image is obtained. The camera pose data includes the pose data of the RS camera corresponding to the exposure time of each row in the first RS image.

[0087] After obtaining the camera pose data, the spatiotemporally aligned RS image is compensated based on the camera pose data, that is, the spatiotemporally aligned RS image is compensated row by row according to the camera pose data to obtain the third RS image. Further, in a possible implementation, step S302 includes:

[0088] Step S3021: Obtain the exposure time of each row in the first RS image;

[0089] Step S3022: Based on the camera pose data and the exposure time, determine the camera rotation increment between each row and the starting row in the first RS image;

[0090] Step S3023: Based on the camera rotation increment, compensate the spatiotemporally aligned RS image to obtain the third RS image.

[0091] After obtaining the camera pose data, the exposure time of each row in the first RS image is obtained. Based on the camera pose data and the exposure time, the camera rotation increment between each row in the first RS image and the starting row is determined. Specifically, the pose data (starting pose data) of the starting row of the first RS image in the camera pose data is first obtained. The pose data of each row in the first RS image in the camera pose data is transformed to the coordinate system corresponding to the starting row. The difference between the pose data of each row after transformation and the starting pose data is used as the camera rotation increment between each row in the first RS image and the starting row.

[0092] After obtaining the camera rotation increment, the spatiotemporally aligned RS image is compensated based on the camera rotation increment. Specifically, the camera rotation increment can be back-projected onto the spatiotemporally aligned RS image to obtain the third RS image.

[0093] The camera pose data corresponding to the first RS image is obtained; then, based on the camera pose data, the spatiotemporally aligned RS image is compensated to obtain the third RS image. The RS rolling shutter effect of the image is reduced by compensating the spatiotemporally aligned RS image according to the camera pose data.

[0094] Based on the above embodiments, a fourth embodiment of the image processing method of this application is proposed, wherein step S104 includes:

[0095] Step S401: Determine the blur kernel function based on the second RS image and the third RS image;

[0096] Step S402: Based on the blur kernel function and the third RS image, determine the target clear image.

[0097] After obtaining the third RS image, the blur kernel function is determined using the third RS image and the second RS image. Specifically, in one possible implementation, step S401 includes:

[0098] Step S4011: Perform a fast Fourier transform on the second RS image to obtain the first transform result;

[0099] Step S4012: Perform a fast Fourier transform on the third RS image to obtain a second transform result;

[0100] Step S4013: Determine the fuzzy kernel function based on the first transformation result and the second transformation result.

[0101] After obtaining the third RS image, perform a Fast Fourier Transform on the second RS image to obtain the first transform result. At the same time, perform a Fast Fourier Transform on the third RS image to obtain the second transform result.

[0102] After obtaining the first transformation result and the second transformation result, the fuzzy kernel function is determined based on the first transformation result and the second transformation result to accurately obtain the fuzzy kernel function. Specifically, in one possible implementation, step S4013 includes:

[0103] Step a: Based on the first transformation result and the second transformation result, determine the Fourier transform result corresponding to the fuzzy kernel function;

[0104] Step b: Perform an inverse Fourier transform on the Fourier transform result corresponding to the fuzzy kernel function to obtain the fuzzy kernel function.

[0105] After obtaining the first and second transformation results, the Fourier transform result corresponding to the fuzzy kernel function is determined based on the first and second transformation results. The fuzzy kernel function is PSF (point spread function). Figure 3 The clear target image is the second RS image, and the actual captured image is the third RS image. The second RS image * the blur kernel function = the third RS image. Therefore, dividing the second transformation result by the first transformation result will give the Fourier transform result corresponding to the blur kernel function.

[0106] After obtaining the Fourier transform result corresponding to the fuzzy kernel function, performing an inverse Fourier transform on the Fourier transform result corresponding to the fuzzy kernel function will yield the fuzzy kernel function, thus allowing for accurate acquisition of the fuzzy kernel function.

[0107] After obtaining the blur kernel function, the target clear image is determined based on the blur kernel function and the third RS image. Specifically, in one possible implementation, step S402 includes:

[0108] Step c: Perform spatial convolution or deconvolution on the third RS image based on the blur kernel function to obtain the clear target image.

[0109] After obtaining the blur kernel function, spatial convolution or deconvolution can be performed on the third RS image using the blur kernel function to accurately obtain a clear image of the target. Specifically, Wiener filtering parameters can be set according to the blur kernel function. By processing the third RS image with the parameters set by Wiener filtering, a clear image of the target can be obtained, thereby reducing the motion blur effect and improving the clarity of the clear image of the target.

[0110] By determining a blur kernel function based on the second RS image and the third RS image, and then determining the target sharp image based on the blur kernel function and the third RS image, the target sharp image can be obtained by processing the RS image using the blur kernel function, thereby reducing the motion blur effect of the RS image and improving the sharpness of the RS image.

[0111] In addition, this application also proposes an aircraft, referring to Figure 4 The aircraft is equipped with a rolling shutter RS ​​camera, which includes an IMU unit and an ISP unit. The ISP unit periodically sets the minimum exposure time corresponding to the RS camera. The aircraft includes:

[0112] The image acquisition module 10 is used to acquire a first RS image captured by the rolling shutter RS ​​camera before the minimum exposure time is set, and a second RS image captured by the RS camera after the minimum exposure time is set.

[0113] Data acquisition module 20 is used to acquire first IMU data corresponding to the first RS image and second IMU data corresponding to the second RS image based on the IMU unit;

[0114] The alignment compensation module 30 is used to perform spatiotemporal alignment of the rows of the first RS image based on the first IMU data and the second IMU data, and to compensate the spatiotemporally aligned RS image to obtain a third RS image.

[0115] The determining module 40 is used to determine the target clear image corresponding to the first RS image based on the third RS image and the second RS image.

[0116] The methods executed by the above-mentioned program units can be referred to in the various embodiments of the image processing method of this application, and will not be repeated here.

[0117] Furthermore, embodiments of this application also propose a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0118] Furthermore, embodiments of this application also propose a computer program product that includes an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0119] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0120] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0122] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Acquire a first RS image captured by the rolling shutter RS ​​camera before the minimum exposure time is set, and a second RS image captured by the RS camera after the minimum exposure time is set, wherein the RS camera includes an IMU unit and an ISP unit, and the ISP unit periodically sets the minimum exposure time corresponding to the RS camera; Based on the IMU unit, first IMU data corresponding to the first RS image and second IMU data corresponding to the second RS image are obtained; Based on the first IMU data and the second IMU data, the rows of the first RS image are spatiotemporally aligned, and the spatiotemporally aligned RS image is compensated to obtain the third RS image; Based on the third RS image and the second RS image, determine the target clear image corresponding to the first RS image; The step of determining the target clear image corresponding to the first RS image based on the third RS image and the second RS image includes: Perform a Fast Fourier Transform on the second RS image to obtain a first transformation result; perform a Fast Fourier Transform on the third RS image to obtain a second transformation result; based on the first transformation result and the second transformation result, determine the Fourier Transform result corresponding to the fuzzy kernel function; perform an Inverse Fourier Transform on the Fourier Transform result corresponding to the fuzzy kernel function to obtain the fuzzy kernel function. Based on the blur kernel function and the third RS image, the target clear image is determined.

2. The image processing method as described in claim 1, characterized in that, The step of performing spatiotemporal alignment of the rows of the first RS image based on the first IMU data and the second IMU data includes: Based on the first IMU data and the second IMU data, the target pose transformation parameters are determined; Based on the target pose transformation parameters, the rows of the first RS image are spatiotemporally aligned to obtain a spatiotemporally aligned RS image.

3. The image processing method as described in claim 2, characterized in that, The step of determining the target pose transformation parameters based on the first IMU data and the second IMU data includes: Obtain the pose transformation parameters between the first IMU data and the second IMU data; The pose transformation parameters are interpolated to obtain the target pose transformation parameters.

4. The image processing method as described in claim 1, characterized in that, The step of compensating the spatiotemporally aligned RS image to obtain a third RS image includes: Obtain the camera pose data corresponding to the first RS image; Based on the camera pose data, the spatiotemporally aligned RS image is compensated to obtain the third RS image.

5. The image processing method as described in claim 4, characterized in that, The step of compensating the spatiotemporally aligned RS image based on the camera pose data to obtain the third RS image includes: Obtain the exposure time of each row in the first RS image; Based on the camera pose data and the exposure time, determine the camera rotation increment between each row and the starting row in the first RS image; Based on the camera rotation increment, the spatiotemporally aligned RS image is compensated to obtain the third RS image.

6. The image processing method according to any one of claims 1 to 5, characterized in that, The step of determining the target clear image based on the blur kernel function and the third RS image includes: Based on the blur kernel function, the third RS image is spatially convolved or deconvolved to obtain the clear image of the target.

7. An aircraft, characterized in that, The aircraft is equipped with a rolling shutter RS ​​camera, which includes an IMU unit and an ISP unit. The ISP unit periodically sets the minimum exposure time corresponding to the RS camera. The aircraft includes: The image acquisition module is used to acquire a first RS image captured by the rolling shutter RS ​​camera before the minimum exposure time is set, and a second RS image captured by the RS camera after the minimum exposure time is set; The data acquisition module is used to acquire, based on the IMU unit, the first IMU data corresponding to the first RS image and the second IMU data corresponding to the second RS image; The alignment compensation module is used to perform spatiotemporal alignment of the rows of the first RS image based on the first IMU data and the second IMU data, and to compensate the spatiotemporally aligned RS image to obtain a third RS image. The determining module is used to determine the target clear image corresponding to the first RS image based on the third RS image and the second RS image; The determining module is specifically used for: Perform a Fast Fourier Transform on the second RS image to obtain a first transformation result; perform a Fast Fourier Transform on the third RS image to obtain a second transformation result; based on the first transformation result and the second transformation result, determine the Fourier Transform result corresponding to the fuzzy kernel function; perform an Inverse Fourier Transform on the Fourier Transform result corresponding to the fuzzy kernel function to obtain the fuzzy kernel function. Based on the blur kernel function and the third RS image, the target clear image is determined.

8. The aircraft as claimed in claim 7, characterized in that, The alignment compensation module is also used for: Based on the first IMU data and the second IMU data, the target pose transformation parameters are determined; Based on the target pose transformation parameters, the rows of the first RS image are spatiotemporally aligned to obtain a spatiotemporally aligned RS image.

9. The aircraft as claimed in claim 8, characterized in that, The alignment compensation module is also used for: Obtain the pose transformation parameters between the first IMU data and the second IMU data; The pose transformation parameters are interpolated to obtain the target pose transformation parameters.

10. The aircraft as claimed in claim 7, characterized in that, The alignment compensation module is also used for: Obtain the camera pose data corresponding to the first RS image; Based on the camera pose data, the spatiotemporally aligned RS image is compensated to obtain the third RS image.

11. The aircraft as claimed in claim 10, characterized in that, The alignment compensation module is also used for: Obtain the exposure time of each row in the first RS image; Based on the camera pose data and the exposure time, determine the camera rotation increment between each row and the starting row in the first RS image; Based on the camera rotation increment, the spatiotemporally aligned RS image is compensated to obtain the third RS image.

12. An image processing apparatus, characterized in that, The image processing apparatus includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the steps of the image processing method as described in any one of claims 1 to 6.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image processing program, which, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 6.

Images