Image processing for observing a celestial body, in particular earth

Abstract

The present invention relates to a method for image processing, comprising: - super-resolution reconstruction of a high-resolution image of a celestial body (102), such as Earth, from low-resolution images (IBRI...IBRN) taken by an on-board imager (106) in a vehicle, such as an aircraft or a space vehicle, designed to move at a distance from the celestial body (102). This method further comprises deblurring the high-resolution image by means of a model which was previously trained by machine learning in order to cancel out an intrinsic blur of the on-board imager (106) and a motion blur resulting from movement of the on-board imager (106) during an acquisition time for acquiring low-resolution images (IBRI -..IBRN).

Description

Description TITLE: IMAGE PROCESSING FOR OBSERVATIONS OF A CELESTIAL BODY, IN PARTICULAR THE EARTH Technical field of the invention

[0001] The present invention relates to a method for processing images of observations of a celestial body, a computer program for implementing this method, a method for observing a celestial body, and a celestial body observation installation. Technological background

[0002] The article "Fast and accurate multi-frame super-resolution of satellite images" by ANGER et al. (2020) describes an image processing method, of the type comprising: - a super-resolution reconstruction of a high-resolution image of a celestial body, such as the Earth, from low-resolution images taken by an imager on board a vehicle, such as an aircraft or a space vehicle, designed to move at a distance from the celestial body.

[0003] The drawback of the method described in this article is that it uses the minimization of a blur image reconstruction function, to which regularization terms are added to avoid noise amplification. This method requires tuning the regularization constants, which, in practice, imposes a trade-off between the strength of the deblurring and the ability to reduce noise. In particular, the deblurring will be of poor quality if the images are very noisy.

[0004] It may therefore be desirable to provide an alternative image processing method that overcomes at least some of the aforementioned problems and constraints. Summary of the invention

[0005] Therefore, an image processing method of the aforementioned type is proposed, characterized in that it further comprises: deblurring of the high-resolution image using a model previously trained by machine learning to cancel an intrinsic blur of the embedded imager and a trailing blur resulting from a movement of the onboard imager during a low-resolution image acquisition time, prior training of the model comprising: - for each of several images: • a simulation of a burst shooting to obtain, from the image considered, successive simulated images all overlapping so as to present a common part, forming a reference image, • a blurring of the successive simulated images to simulate the intrinsic blur of the onboard imager and the trailing blur resulting from the movement of the onboard imager relative to the celestial body during the acquisition time, and • a reconstruction of a high-resolution blurred image by super-resolution from the simulated and blurred images; and - a supervised training of the model from the reference images and the blurred images.

[0006] The invention may further include one or more of the following optional features, in any technically feasible combination.

[0007] Optionally, the low-resolution images are multispectral so that the reconstruction provides several high-resolution monospectral images in respective spectral bands, and each high-resolution monospectral image is deblurred using a respective model, dedicated to the spectral band of that high-resolution monospectral image.

[0008] Optionally, the model is also pre-trained by machine learning to denoise the high-resolution image.

[0009] Optionally, the model also includes a convolutional neural network.

[0010] Optionally, the model is also designed to provide a deblurred image of the same resolution as the high-resolution image input to the model.

[0011] Optionally, the model's preliminary training also includes: - for each of several images: • a simulation of burst shooting to obtain, from the image in question, successive simulated images overlapping all in such a way as to present a common part, forming a reference image, • a blurring of successive simulated images to simulate the intrinsic blur of the onboard imager and the trailing blur resulting from the displacement of the onboard imager relative to the celestial body during the acquisition time, and • a reconstruction of a high-resolution blurred image by super-resolution from the simulated and blurred images; and - a supervised training of the model from the reference images and the blurred images.

[0012] Optionally, the model's pre-training also includes, for each image, noise generation of the simulated and blurred images.

[0013] Optionally, the noise and / or intrinsic blur also depends on the spectral band to which the model is dedicated.

[0014] Optionally, except for the first simulated image, each of the other simulated images is shifted from the previous one by a respective shift in the image, comprising a constant shift and a random shift.

[0015] A machine learning training method is also proposed for a model designed to deblur a high-resolution image of a celestial body, such as the Earth, reconstructed by super-resolution from low-resolution images taken by an imager mounted on a vehicle, such as an aircraft or a spacecraft, designed to move at a distance from the celestial body. The deblurring aims to eliminate an intrinsic blur of the mounted imager and a motion blur resulting from the movement of the mounted imager during the acquisition of the low-resolution images. The method comprises: - for each of several images: • a simulation of a burst of shots to obtain, from the image in question, successive simulated images all overlapping so as to present a common part, forming a reference image.• a blurring of successive simulated images to simulate the intrinsic blur of the onboard imager and the resulting motion blur, displacement of the onboard imager relative to the celestial body during an acquisition time, and • a reconstruction of a high-resolution blurred image by super-resolution from the simulated and blurred images; and - a supervised training of the model from the reference images and the blurred images.

[0016] Also proposed is a computer program downloadable from a communication network and / or recorded on a computer-readable medium, characterized in that it includes instructions for executing the steps of a process according to the invention, when said program is executed on a computer.

[0017] A method for observing a celestial body is also proposed, comprising: - taking successive low-resolution images of the celestial body by an onboard imager; and - image processing according to a method according to the invention.

[0018] Also proposed is an observation system for a celestial body, such as the Earth, comprising: - a vehicle, such as an aircraft or a spacecraft, designed to move at a distance from the celestial body and carrying an imager designed to successively capture overlapping low-resolution images; and - an image processing device designed to implement an image processing method according to the invention. Brief description of the figures

[0019] The invention will be better understood with the aid of the following description, given solely by way of example and made with reference to the accompanying drawings in which: - Figure 1 is a simplified view of an installation according to the invention for observing a celestial body, such as the Earth, - Figure 2 is a functional view of an image processing device of the installation of Figure 1, - Figure 3 is a block diagram of a method according to the invention for training a machine learning model of the training device of Figure 2, - Figure 4 illustrates certain steps of the training method of Figure 3, - Figure 5 is a block diagram of a method according to the invention for observing a celestial body, implemented by the installation of Figure 1, - Figure 6 illustrates a step of the method of Figure 3, for extracting simulated images from a high-resolution original image, - Figure 7 is a functional diagram of a computer system that can serve as an image processing device for the installation of Figure 1, and - Figure 8 is a functional diagram of a computer system that can serve as a training device for one or more models used in the image processing device of Figure 2. Detailed description of the invention

[0020] With reference to Figure 1, an example of an installation 100 according to the invention, for the observation of a celestial body 102, for example the Earth as in the illustrated example, will now be described.

[0021] The installation 100 includes, first of all, a vehicle 104 designed to move at a distance from the celestial body 102. The vehicle 104 is, for example, an aircraft (airplane, drone, etc.) or a spacecraft, such as a satellite as in the illustrated example. The vehicle 104 is located, for example, at an altitude of more than 1 km in the case of an aircraft, and at an altitude of more than 500 km in the case of a satellite.

[0022] Vehicle 104 includes an onboard imager 106 designed to take successive low-resolution IBR1 ^ IBRn ^ IBRN (N ! 2) images of the celestial body 102. These low-resolution images I BR1 ^ I BRNhave, for example, a resolution of nxm pixels and are taken quickly enough, in so-called "burst on area" mode (from the English "push frame"), taking into account the movement of the vehicle 104 relative to the celestial body 102, so that the low-resolution images I BR1 ^ I BRN se all overlap. Thus, there is a common part between all the low resolution IBR1 ^ IBRN images, representing the same area P of the celestial body 102.

[0023] Furthermore, these low-resolution images I BR1 ^I BRN can be multispectral, that is to say that each low-resolution image I BRn is a combination of images I BRn 1 ^ I BRn s ^ I BRn S in S different spectral bands.

[0024] Furthermore, the onboard imager 106 exhibits intrinsic blurring represented by a modulation transfer function MTF, as well as an acquisition time to capture each low-resolution image I BRn .

[0025] Installation 100 also includes an image processing device 108, designed to process low-resolution images. BR1 ^I BRN images taken by the onboard imager 106.

[0026] The image processing unit 108 is, for example, mounted with the imager in the vehicle 104, or remotely from the vehicle, for example on the celestial body 102 as in the illustrated example. The installation 100 then comprises two wireless communication devices 110, 112 designed to exchange data, one being mounted in the vehicle 104 and the other being connected to the image processing unit 108.

[0027] With reference to Figure 2, an example of the implementation of the image processing device 108 will now be described.

[0028] The image processing device 108 includes first of all a reconstruction module 202 designed to reconstruct by super-resolution a high resolution IHR image, from the low resolution IBR1 ^ IBRN images taken by the on-board imager 106. For example, the high resolution image has a resolution of NxM pixels, with NxM > nxm, or even N > n and M > m.

[0029] Indeed, the onboard imager 106 moves relative to the celestial body 102, mainly due to the movement of the vehicle 104 relative to the celestial body 102 and vibrations of the onboard imager 106. The reconstruction module 202 makes it possible to control these untimely movements without requiring inertial data, line-of-sight data or image time-stamping data.

[0030] When low-resolution imagesBR1 ^ I BRN are multispectral, the high-resolution image I HR is also multispectral, that is to say a combination of high-resolution images HR 1 ^ I HR s ^ I HR S in the different spectral bands.

[0031] To implement super-resolution reconstruction, the 202 reconstruction module is designed, for example, as follows. The 202 reconstruction module is designed to first perform a statistical analysis on each pair of low-resolution images. BR1 ^ I BRN successive, to determine a homographic transformation linking the two low-resolution images of this pair. The reconstruction module 202 is designed to then superimpose the low-resolution images I BR1 ^ I BRNfrom the determined homographic transforms. The 202 reconstruction module is designed to then resample and merge the low-resolution images. BR1 ^ I BRN superimposed to obtain the high-resolution image I HR The fusion process includes, for example, a summation of low-resolution images. BR1 ^ I BRN resampled. An example of reconstruction by super-resolution is described, for example, in the article by ANGER et al. mentioned previously.

[0032] The image processing device 108 further includes a separation module 204 designed to separate high-resolution monospectral images I HR 1 ^ IHR S of the high-resolution IHR image.

[0033] The image processing device 108 further comprises, for each spectral band s, a model M spreviously trained by machine learning to deblur the high-resolution monospectral IHR image s of the spectral band considered. In particular, each model M s is pre-trained to cancel the intrinsic blur of the onboard imager 106 and the motion blur resulting from the movement of the onboard imager 106 during the acquisition time. In addition, the pre-training of the model M s can allow the latter to denoise the high-resolution monospectral IHR image s For example, each M model s is designed to provide a deblurred image of the same resolution as the high-resolution IHR image s Entry point. Each M model s is, for example, a convolutional neural network.

[0034] The image processing device 108 may further include a contrast module 206 designed to improve the contrast of each of the high-resolution monospectral images IHR 1 ^ I HR S .

[0035] With reference to Figure 3 and Figure 4, an example of a 300 training method for each model M s will now be described.

[0036] During step 302, clear IMG images are obtained for the spectral band s of the M model s At least some of these IMG images can be used by For example, they may have been taken by imagers in operation with a higher resolution than the onboard imager 106. At least some of these IMG images may come from generic image banks, that is, representing any scene. The IMG images used for the spectral band under consideration may not have been taken in that spectral band. For example, an image taken in the visible range in grayscale can be used to train a model dedicated to infrared. In this case, the grayscale levels are considered infrared levels.

[0037] The following steps 304 to 314 are then implemented for each IMG image.

[0038] During an optional step 304, the current IMG image is modified to simulate illumination.

[0039] During step 306, a burst shot is simulated from the current IMG image.

[0040] For this, portions of the same size n S xm S are extracted from the current IMG image to obtain successive simulated images IS1, IS2, IS3, as if the onboard imager 106 were flying over the IMG image and successively taking these images IS1, IS2, IS3. Except for the first simulated image IS1, each of the other simulated images IS2, IS3 is thus offset from the previous one by a respective offset ms1, ms2 in the IMG image, comprising a constant offset D and a random offset A1, A2: ms1 = D + A1, ms2 = D + A2. The constant offset D simulates the displacement of the onboard imager 106 relative to the celestial body 102, while the random offset A1, A2 simulates the vibrations of the onboard imager 106. The simulated images IS1, IS2, IS3 all overlap in such a way as to present a common part, forming an IR reference image.

[0041] With reference to Figure 6, the extraction of the simulated images IS1, IS2, IS3 is illustrated in the case of an IMG image obtained from an aerial photograph taken with a very high-resolution imager. Other types of IMG images (landscape, animals, etc.) can also be used.

[0042] Returning to Figure 3, during step 308, each of the simulated images I S1 , I S2 , I S3 is resized to the nxm resolution of the onboard imager 106.

[0043] During step 310, each of the simulated images I S1 , I S2 , I S3 is blurred.

[0044] To achieve this, two blurring techniques are implemented, in any order, or even simultaneously.

[0045] The first blurring method is designed to simulate the intrinsic blur of the onboard imager 106 based on its modulation transfer function (MTF). This function is preferably specific to the onboard imager 106 and not generic. For example, it is measured on the onboard imager 106 or calculated from a simulation of the onboard imager 106. Preferably, the modulation transfer function (MTF) depends on the spectral band considered.

[0046] The second blurring is designed to simulate the trailing blur resulting from the movement of the onboard imager 106 relative to the celestial body 102, during the acquisition time of the onboard imager 106.

[0047] Ces deux floutages sont par exemple réalisés de la manière suivante : !"#$% = & '()*+(),è-$% & '!("é avec : & the convolution operation, and !"#$% respectively the image before and after blurring, and'()*+(),è-$% and '!("é respectively the convolution kernel of the intrinsic blur of the embedded imager 106 and of the motion blur.

[0048] The convolution kernel '()*+(),è-$%can for example be a sampled d^Airy task.

[0049] The convolution kernel ' !("é For example, in the case of a horizontal blur of length 2 pixels: '!("é

[0050] During step 312, each of the simulated images I S1 , I S2 , I S3 and blurred is noisy, that is to say that the value of at least some pixels is modified. Preferably, the added noise depends on the spectral band considered.

[0051] During a step 314, a high-resolution blurry image IF is reconstructed by super-resolution from the simulated low-resolution images IS1, IS2, IS3, preferably in the same way as is done in the reconstruction module 202 of the image processing device 108. In particular, in Figure 4, the determined homographic transformations are noted m1, m2.

[0052] Thus, the preceding steps 302 to 314 allow us to obtain IF blurred images and associated IR reference images, forming a training corpus for the M model s .

[0053] During step 316, the model M s is trained in a supervised manner. For this, parameters of the model M s are updated so that this M model s output provided, for each blurred image I F provided as input, an output image resembling as closely as possible the reference image I R associate.

[0054] Preferably, high-resolution blurry images F are generated on the fly, rather than beforehand. This limits the computer memory required. Thus, the IMG images are distributed in batches, for example, four IMG images per batch. The blurry images I F are obtained for the IMG images of a first batch and the M model sis trained from the reference images I R and blurry images F for this first batch. Then, this is repeated for the next batch.

[0055] Furthermore, a single IMG image can be used to obtain several pairs of reference images. R and blurry I F by changing parameters in steps 302 to 314 each time. These parameters include, for example, one or more of the following: illumination (step 304), random offsets (step 306), a motion blur length (step 310), and noise parameters (step 312). These parameters are, for example, randomly drawn from predefined intervals.

[0056] With reference to Figure 5, an example of an observation method 500 of the celestial body 102 will now be described.

[0057] During a step 502, the onboard imager 106 takes successive low-resolution IBR1 ^ IBRN images of the celestial body 102.

[0058] During a step 504, the low resolution IBR1 ^ IBRN images are transmitted to the image processing device 108.

[0059] In step 506, the reconstruction module 202 reconstructs a high-resolution IHR image using super-resolution from the low-resolution IBR1 ^ IBRN images. Preferably, this step 506 is performed without any additional information other than the low-resolution I images. BR1 ^ I BRN .

[0060] When low-resolution images BR1 ^ I BRN are multispectral, the high-resolution IHR image is also multispectral, that is to say, composed of several high-resolution monospectral images. HR s .

[0061] During step 508, each high-resolution monospectral IHR image s is deblurred using the M model s for the spectral band s.

[0062] During an optional step 510, the contrast module 206 enhances the contrast of each of the high-resolution monospectral images I HR S , in order to improve their interpretation by a human operator. These high-resolution monospectral images I HR S can then be displayed on a display device (not shown) of installation 100, such as a screen, in order to be viewed by the operator.

[0063] With reference to Figure 7, the image processing device 108 is, for example, a computer system comprising a data processing unit 702 (such as a microprocessor) and a main memory 704 (such as RAM, from the English "Random Access Memory") accessible by the processing unit 702. The computer system further comprises, for example, a network interface and / or a computer-readable medium, such as a local medium (such as a local hard disk 706) or a remote medium (such as a remote hard disk accessible via the network interface through a communication network) or a removable medium (such as a USB flash drive, from the English "Universal Serial Bus", or a CD, from the English "Compact Disc" or a DVD, from the English "Digital Versatile Disc") readable by means of an appropriate reader of the computer system (such as a USB port or a CD and / or DVD disc drive).A computer program 708 containing instructions for the processing unit 702 is stored on the medium 706 and / or downloadable via the network interface. This computer program 708 is, for example, intended to be loaded into the main memory 704, so that the processing unit 702 executes its instructions to implement the image processing procedure combining the preceding steps 506, 508, and 510. For example, modules 202, 204, 206, and models M. 1 ^ M S are software modules of the computer program 708.

[0064] Alternatively, all or part of these modules could be implemented as hardware modules, i.e. as an electronic circuit, for example micro-wired, not involving a computer program.

[0065] With reference to Figure 8, an example of device 800 for the implementation of process 300 will now be described.

[0066] In this example, device 800 is a computer system comprising a data processing unit 802 (such as a microprocessor) and a main memory 804 (such as RAM, short for Random Access Memory). Memory") accessible by the processing unit 802. The computer system further includes, for example, a network interface and / or computer-readable media, such as local media (such as a local hard drive 806) or remote media (such as a remote hard drive accessible via the network interface through a communication network) or removable media (such as a USB flash drive, or a CD, or a DVD, or a Digital Versatile Disc) readable by means of a suitable reader of the computer system (such as a USB port or a CD and / or DVD disc drive). A computer program 808 containing instructions for the processing unit 802 is stored on the media 806 and / or downloadable via the network interface.This computer program 808, for example, is intended to be loaded into main memory 804, so that the processing unit 802 can execute its instructions to implement process 300.

[0067] Alternatively, all or part of the steps of process 300 could be implemented by hardware modules, i.e. in the form of an electronic circuit, for example micro-wired, not involving a computer program.

[0068] In conclusion, it should be noted that the invention is not limited to the embodiments described above. Indeed, it will be apparent to those skilled in the art that various modifications can be made to the embodiments described above, in light of the information just provided.

[0069] In the detailed presentation of the invention given above, the terms used shall not be interpreted as limiting the invention to the embodiments set forth in this description, but shall be interpreted as including all equivalents which can be foreseen by a person skilled in the art by applying their general knowledge to the implementation of the teaching which has just been disclosed to them.

Claims

Claims [1] Image processing method, comprising: - a super-resolution reconstruction (506) of a high-resolution image (HRI) s ) of a celestial body (102), such as the Earth, from low-resolution images (IBR1^IBRN) taken by an onboard imager (106) in a vehicle, such as an aircraft or a spacecraft, designed to move at a distance from the celestial body (102); characterized in that it further comprises: - a deblurring (510) of the high-resolution image (I HR s ) by means of a model (M s ) previously trained by machine learning to cancel an intrinsic blur of the onboard imager (106) and a trailing blur resulting from a movement of the onboard imager (106) during a low-resolution image acquisition time (I BR1 ^I BRN ) ; and in that the prior training of the model (M s) includes: - for each of several images (IMG): ^ a simulation (306) of a burst shot to obtain, from the considered image (IMG), simulated images (I S1 , I S2 , I S3 successive overlapping layers, all presenting a common part, forming a reference image (I R ), ^ a blurring (310) of the simulated images (I S1 , I S2 , I S3 ) successive to simulate the intrinsic blur of the onboard imager (106) and the trailing blur resulting from the displacement of the onboard imager (106) relative to the celestial body (102) during the acquisition time, and a reconstruction (314) of a high-resolution blurred image (I F ) by super-resolution from the simulated images (I S1 , I S2 , I S3 ) and blurred; and - a training (316) of the model (M s ) in a supervised manner using reference images (I R ) and blurry images (IF ). [2] Method according to claim 1, wherein the low-resolution images (I BR1 ^I BRN ) are multispectral, so the reconstruction provides several high-resolution monospectral images (IHR) s ) in respective spectral bands, and in which each high-resolution monospectral image (I HR s ) is defatting (510) by means of a model (M s ) respectively, dedicated to the spectral band of this high-resolution monospectral image (HRI) s ). [3] Method according to claim 1 or 2, wherein the model (M s ) is also pre-trained by machine learning to denoise the high-resolution image (I HR s ). [4] A method according to any one of claims 1 to 3, wherein the model (M s) comprises a convolutional neural network. [5] Method according to any one of claims 1 to 4, wherein the model (M s ) is designed to provide a deblurred image of the same resolution as the high-resolution image (I HR s ) as input to the model (M s ). [6] Method according to any one of claims 1 to 5, wherein the prior training of the model (M s ) further includes, for each of the images (IMG), a noise (312) of the simulated images (I S1 , I S2 , I S3 ) and blurred. [7] A method according to claim 6, wherein the noise (312) and / or intrinsic blurring depends on the spectral band to which the model (M s ) is dedicated. [8] Method according to any one of claims 1 to 7, wherein, except for the first simulated image (I S1 ), each of the other simulated images (I S2, IS3) is offset from the previous one by a respective offset (ms1, ms2) in the image (IMG), comprising a constant offset and a random offset. [9] Method (300) of machine learning training a model (M s ), so that the model (M s ) is designed to deblur a high-resolution image (IHR) s) of a celestial body (102), such as the Earth, reconstructed by super-resolution from low-resolution images (IBR1^IBRN) taken by an on-board imager (106) in a vehicle, such as an aircraft or a space vehicle, designed to move at a distance from the celestial body (102), the deblurring aimed at canceling an intrinsic blur of the on-board imager (106) and a trailing blur resulting from a movement of the on-board imager (106) during an acquisition time of the low-resolution images (IBR1^IBRN), the process comprising: - for each of several images (IMG): ^ a simulation (306) of a burst shooting to obtain, from the image (IMG) considered, successive simulated images (IS1, IS2, IS3) all overlapping so as to present a common part, forming a reference image (IR), ^ a blurring (310) of successive simulated images (IS1, IS2, IS3) to simulate the intrinsic blur of the onboard imager (106) and the trailing blur resulting from the displacement of the onboard imager (106) relative to the celestial body (102) during an acquisition time, and ^ a reconstruction (314) of a high-resolution blurred image (IF) by super-resolution from the simulated and blurred images (IS1, IS2, IS3); and - a training (316) of the model (M s) in a supervised manner from reference images (IR) and blurred images (IF). [10] Computer program (708) downloadable from a communication network and / or stored on a computer-readable medium, characterized in that it includes instructions for executing the steps of a method according to any one of claims 1 to 8, when said program is executed on a computer. [11] Computer program (808) downloadable from a communication network and / or stored on a computer-readable medium, characterized in that it includes instructions for executing the steps of a method according to claim 9, when said program is executed on a computer. [12] Method for observing (500) a celestial body (102), comprising: - a low-resolution image acquisition (502) (I BR1 ^ I BRN) successive images of the celestial body (102) by an on-board imager (106); and - image processing according to a method according to any one of claims 1 to 8. [13] Installation (100) for observing a celestial body (102), such as the Earth, comprising: - a vehicle (104), such as an aircraft or a space vehicle, designed to move at a distance from the celestial body and carrying an imager (106) designed to successively capture overlapping low-resolution images; and - an image processing device (108) designed to implement a method according to any one of claims 1 to 8.

Images