Image processing of observations of a celestial body, particularly the Earth

Abstract

The present invention relates to an image processing method comprising: - a super-resolution reconstruction of a high-resolution image of a celestial body (102), such as the Earth, from low-resolution images (IBR1…IBRN) acquired by an onboard imager (106) in a vehicle, such as an aircraft or a spacecraft, designed to move at a distance from the celestial body (102). This method further comprises deblurring the high-resolution image using a model previously trained by machine learning to eliminate intrinsic blur of the onboard imager (106) and motion blur resulting from the movement of the onboard imager (106) during the acquisition of the low-resolution images (IBR1…IBRN). Figure for the abstract: Fig. 1

Description

Title of the invention: IMAGE PROCESSING FOR OBSERVATIONS OF A CELESTIAL BODY, IN PARTICULAR THE EARTH. Technical field of the invention

[0001] The present invention relates to a method for processing images of observation of a celestial body, a computer program for implementing this method, a method for observing a celestial body and an installation for observing a celestial body. Technological background

[0002] The article “Fast and accurate multi-frame super-resolution of satellite images” by ANGER et al. (2020) describes an image processing method of the type comprising: - a super-resolution reconstruction of a high-resolution image of a celestial body, such as the Earth, from low-resolution images taken by an imager on board a vehicle, such as an aircraft or a space vehicle, designed to move at a distance from the celestial body.

[0003] The drawback of the method described in this article is that it uses the minimization of a blur image reconstruction function to which regularization terms are added to avoid noise amplification. This method requires tuning the regularization constants which, in practice, imposes a trade-off between the strength of the deblurring and the ability to reduce noise. In particular, the deblurring will be of low quality if the images are very noisy.

[0004] It may therefore be desirable to provide an alternative image processing method that makes it possible to overcome at least some of the aforementioned problems and constraints. Summary of the invention

[0005] An image processing method of the aforementioned type is therefore proposed, characterized in that it further comprises: deblurring of the high-resolution image by means of a model previously trained by machine learning to cancel an intrinsic blur of the embedded imager and a trailing blur resulting from a movement of the embedded imager during a low-resolution image acquisition time.

[0006] The invention may further include one or more of the following optional features, according to any technically possible combination.

[0007] Optionally, the low-resolution images are multispectral so that the reconstruction provides several high-resolution monospectral images in respective spectral bands, and each high-resolution monospectral image is deblurred by means of a respective model, dedicated to the spectral band of this high-resolution monospectral image.

[0008] Optionally, the model is also pre-trained by machine learning to denoise the high-resolution image.

[0009] Optionally also includes a convolutional neural network.

[0010] Optionally, the model is also designed to provide a deblurred image of the same resolution as the high-resolution image input to the model.

[0011] Optionally, the prior training of the model also includes: - for each of several images: • a simulation of burst shooting to obtain, from the image in question, successive simulated images all overlapping in such a way as to present a common part, forming a reference image, • a blurring of successive simulated images to simulate the intrinsic blur of the onboard imager and the trailing blur resulting from the movement of the onboard imager relative to the celestial body during the acquisition time, and • a reconstruction of a high-resolution blurred image using super-resolution from the simulated and blurred images; and - a supervised training of the model using reference images and blurred images.

[0012] Optionally, the prior training of the model also includes, for each of the images, noise from the simulated and blurred images.

[0013] Optionally, the noise and / or intrinsic blur also depends on the spectral band to which the model is dedicated.

[0014] Optionally also, except for the first simulated image, each of the other simulated images is offset from the previous one by a respective offset in the image, comprising a constant offset and a random offset.

[0015] A method for training a model by machine learning is also proposed, such that the model is designed to deblur a high-resolution image of a celestial body, such as the Earth, reconstructed by super-resolution from low-resolution images taken by an imager mounted on a vehicle, such as an aircraft or a spacecraft, designed to move at a distance from the celestial body. The deblurring aims to eliminate an intrinsic blur of the mounted imager and a trailing blur resulting from the movement of the mounted imager during the acquisition of the low-resolution images. The method comprises: - for each of several images: • a simulation of burst shooting to obtain, from the image in question, successive simulated images all overlapping in such a way as to present a common part, forming a reference image, • a blurring of successive simulated images to simulate the intrinsic blur of the onboard imager and the motion blur resulting from the movement of the onboard imager relative to the celestial body during an acquisition time, and • a reconstruction of a high-resolution blurred image using super-resolution from the simulated and blurred images; and - a supervised training of the model using reference images and blurred images.

[0016] Also proposed is a computer program downloadable from a communication network and / or recorded on a computer-readable medium, characterized in that it includes instructions for the execution of the steps of a process according to the invention, when said program is executed on a computer.

[0017] A method for observing a celestial body is also proposed, comprising: - the acquisition of successive low-resolution images of the celestial body by an onboard imager; and - image processing according to a method according to the invention.

[0018] An installation for observing a celestial body, such as the Earth, is also proposed, comprising: - a vehicle, such as an aircraft or a spacecraft, designed to move at a distance from the celestial body and carrying an imager designed to successively capture overlapping low-resolution images; and - an image processing device designed to implement an image processing method according to the invention. Brief description of the figures

[0019] The invention will be better understood with the aid of the following description, given solely by way of example and made with reference to the accompanying drawings in which: - Fig. 1 is a simplified view of an installation according to the invention for observing a celestial body, such as the Earth, - [Fig.2] is a functional view of an image processing device from the installation of [Fig.1], - [Fig.3] is a block diagram of a method according to the invention for training a machine learning model of the training device of [Fig.2], - [Fig.4] illustrates certain steps of the training process of [Fig.3], - [Fig.5] is a block diagram of a method according to the invention for observing a celestial body, implemented by the installation of [Fig.1], - Figure [Fig. 6] illustrates a step in the process of [Fig. 3], for extracting simulated images from a high-resolution original image, - [Fig. 7] is a functional diagram of a computer system that can serve as an image processing device for the installation of [Fig. 1], and - [Fig.8] is a functional diagram of a computer system that can serve as a training device for one or more models used in the image processing device of [Fig.2]. Detailed description of the invention

[0020] With reference to [Fig.1], an example of an installation 100 according to the invention, for the observation of a celestial body 102, for example the Earth as in the illustrated example, will now be described.

[0021] The installation 100 first comprises a vehicle 104 designed to move at a distance from the celestial body 102. The vehicle 104 is, for example, an aircraft (airplane, drone, etc.) or a spacecraft, such as a satellite as in the illustrated example. The vehicle 104 is located, for example, at an altitude of more than 1 km in the case of an aircraft, and at an altitude of more than 500 km in the case of a satellite.

[0022] The vehicle 104 includes an onboard imager 106 designed to take successive low-resolution IBRi ... IBRn ... Ibrn (N > 2) images of the celestial body 102. These low-resolution IBRi ... IBRn images have, for example, a resolution of nxm pixels and are taken sufficiently quickly, in so-called "push frame" mode, taking into account the movement of the vehicle 104 relative to the celestial body 102, so that the low-resolution IBRi ... Ibrn images all overlap. Thus, there is a common area between all the low-resolution IBRi ... IBRn images, representing the same area P of the celestial body 102.

[0023] Furthermore, these low-resolution IBR i.. .IBRN images can be multispectral, that is to say that each low-resolution IBRn image is a combination of IBR ...Ibr ns ... IBr nS images in S different spectral bands.

[0024] Furthermore, the on-board imager 106 exhibits an intrinsic blur represented by a modulation transfer function MTF, as well as an acquisition time to take each low-resolution image IBRn.

[0025] The installation 100 further includes an image processing device 108, designed to process the low resolution IBR1.. .IBrn images taken by the on-board imager 106.

[0026] The image processing device 108 is, for example, mounted with the imager in the vehicle 104, or remotely from the vehicle, for example on the celestial body 102 as in the illustrated example. The installation 100 then comprises two wireless communication devices 110, 112 designed for exchanging data, one being mounted in the vehicle 104 and the other being connected to the image processing device 108.

[0027] With reference to [Fig.2], an example of an embodiment of the image processing device 108 will now be described.

[0028] The image processing device 108 first includes a reconstruction module 202 designed to reconstruct by super-resolution a high-resolution IHR image, from the low-resolution IBR i ... IBrn images taken by the on-board imager 106. For example, the high-resolution image has a resolution of NxM pixels, with NxM > nxm, or even N > n and M > m.

[0029] Indeed, the onboard imager 106 moves relative to the celestial body 102, mainly due to the movement of the vehicle 104 relative to the celestial body 102 and vibrations of the onboard imager 106. The reconstruction module 202 makes it possible to control these untimely movements without requiring inertial data, line-of-sight data or image time-stamping data.

[0030] When the low-resolution IBR i ... IBRN images are multispectral, the high-resolution IHR image is also multispectral, i.e. a combination of high-resolution IH R* ... Ih rs • • • lu RS images in the S different spectral bands.

[0031] To implement super-resolution reconstruction, the reconstruction module 202 is, for example, designed as follows. The reconstruction module 202 is designed to first perform a statistical analysis on each pair of successive low-resolution IBRi ... IBRN images to determine a homographic transformation linking the two low-resolution images of that pair. The reconstruction module 202 is then designed to superimpose the low-resolution IBRi ... IBRN images using the determined homographic transforms. The reconstruction module 202 is then designed to resample and then merge the superimposed low-resolution IBRi ... IBRN images to obtain the high-resolution IHR image. The merging process includes, for example, a summation of the resampled low-resolution IBRi ... IBRN images. An example of super-resolution reconstruction is described, for example, in the article by Anger et al.mentioned previously.

[0032] The image processing device 108 further includes a separation module 204 designed to separate the high-resolution monospectral images Ihr1... IhRS from the high-resolution image IHR.

[0033] The image processing device 108 further comprises, for each spectral band s, a model Ms pre-trained by machine learning to deblur the high-resolution monospectral IHRS image of the spectral band s under consideration. In particular, each model Ms is pre-trained to cancel the intrinsic blur of the onboard imager 106 and the motion blur resulting from the movement of the onboard imager 106 during the acquisition time. In addition, the pre-training of the model Ms may enable it to denoise the high-resolution monospectral IHRS image. For example, each model Ms is designed to provide a deblurred image of the same resolution as the input high-resolution IHRS image. Each model Ms is, for example, a convolutional neural network.

[0034] The image processing device 108 may further include a contrast module 206 designed to improve the contrast of each of the high-resolution monospectral IHR* ... IHrS- images

[0035] With reference to [Fig.3] and [Fig.4], an example of a 300 training method for each model Ms will now be described.

[0036] In a step 302, sharp IMG images are obtained for the spectral band s of the model Ms. At least some of these IMG images may, for example, have been taken by imagers in operation with a higher resolution than the onboard imager 106. At least some of these IMG images may come from generic image banks, i.e., representing any scene. The IMG images used for the spectral band s under consideration may not have been taken in that spectral band s. For example, an image taken in the visible range in grayscale may be used to train a model dedicated to infrared. In this case, the grayscale levels are considered as infrared levels.

[0037] The following steps 304 to 314 are then implemented for each IMG image.

[0038] During an optional step 304, the current IMG image is modified to simulate illumination.

[0039] During a step 306, a burst shot is simulated from the current IMG image.

[0040] To achieve this, portions of the same size nsxms are extracted from the current IMG image to obtain successive simulated images ISi, IS2, Iss, as if the onboard imager 106 were flying over the IMG image and successively taking these images ISi, IS2, Iss. Except for the first simulated image ISi, each of the other simulated images IS2, Iss is thus offset from the previous one by a respective offset msl, ms2 in the IMG image, comprising a constant offset D and a random offset Al, A2: msl = D + Al, ms2 = D + A2. The constant offset D simulates the movement of the onboard imager 106 relative to the celestial body 102, while the random offset A1, A2 simulates the vibrations of the onboard imager 106. The simulated images ISi, IS2, Iss all overlap so as to present a common part, forming an IR reference image.

[0041] With reference to [Fig. 6], the extraction of the simulated images ISi, IS2, Iss is illustrated. in the case of an IMG image from an aerial photograph taken with a very high-resolution imager. Other types of IMG images (landscape, animals, etc.) can also be used.

[0042] Returning to [Fig.3], during a step 308, each of the simulated images ISi, L2, IS3 is resized to the resolution nxm of the on-board imager 106.

[0043] During a step 310, each of the simulated images ISi, IS2, Iss is blurred.

[0044] For this purpose, two blurrings are implemented, in any order, or even simultaneously.

[0045] The first blurring is designed to simulate the intrinsic blur of the onboard imager 106 based on its modulation transfer function (MTF). This function is preferably specific to the onboard imager 106 and not generic. For example, it is measured on the onboard imager 106 or calculated from a simulation of the onboard imager 106. Preferably, the modulation transfer function (MTF) depends on the spectral band considered.

[0046] The second blurring is designed to simulate the trailing blur resulting from the movement of the onboard imager 106 relative to the celestial body 102, during the acquisition time of the onboard imager 106.

[0047] These two blurrings are, for example, carried out in the following manner: Ifloue — ^^intrinsic ' Kfilé^^C : * the convolution operation, I and 1fioue respectively the image before and after blurring, and ^intrinsic and panning respectively the convolution kernel of the intrinsic blur of the embedded imager 106 and of the panning blur.

[0048] The Kintrinsic convolution kernel "Can Pæ" example be a sampled Airy task.

[0049] The convolution kernel Kest, for example, in the case of motion blur horizontal of length 2 pixels: -0 0 0' KfÜé = 1 3 j. 3 1 3 .0 0 0.

[0050] During a step 312, each of the simulated images IS1, IS2, Iss and blurred is noisy, i.e. the value of at least some pixels is modified. Preferably, the added noise depends on the spectral band considered.

[0051] During a step 314, a high-resolution blurry image IF is reconstructed by super-resolution from the simulated low-resolution images ISb Is2, Iss, preferably in the same way as is carried out in the reconstruction module 202 of the image processing device 108. In particular, in [Fig.4], the determined homographic transformations are denoted ml, m2.

[0052] Thus, the preceding steps 302 to 314 make it possible to obtain IF blurred images and associated IR reference images, forming a training corpus for the Ms model.

[0053] During a step 316, the model Ms is trained in a supervised manner. For this purpose, parameters of the model Ms are updated so that this model Ms provides as output, for each blurred IF image supplied as input, an output image resembling as closely as possible the associated IR reference image.

[0054] Preferably, the high-resolution IF blur images are generated on the fly, rather than beforehand. This limits the computer memory required. Thus, the IMG images are distributed in batches, for example, four IMG images per batch. The IF blur images are obtained for the IMG images of a first batch, and the Ms model is trained from the IR reference images and the IF blur images for this first batch. Then, this is repeated for the next batch.

[0055] Furthermore, a single IMG image can be used to obtain several pairs of IR and IF blur reference images, by changing parameters in steps 302 to 314 each time. These parameters include, for example, one or more of the following: illumination (step 304), random offsets (step 306), a motion blur length (step 310), and noise parameters (step 312). These parameters are, for example, randomly drawn from predefined intervals.

[0056] With reference to [Fig.5], an example of a method for observing 500 of the celestial body 102 will now be described.

[0057] During a step 502, the on-board imager 106 takes successive low-resolution IBR i ... IBrn images of the celestial body 102.

[0058] During a step 504, the low-resolution IBR i ... IBR N images are transmitted to the image processing device 108.

[0059] During a step 506, the reconstruction module 202 reconstructs a high-resolution IHR image by superresolution from the low-resolution IBR i ... IBRN images. Preferably, this step 506 is carried out without any additional information other than the low-resolution IBRi ... IBRN images.

[0060] When the low-resolution IBRi ... IBRN images are multispectral, the high-resolution IHR image is also multispectral, i.e., composed of several high-resolution monospectral IHRS images.

[0061] During a step 508, each high-resolution monospectral IHRS image is deblurred using the Ms model for the spectral band s.

[0062] In an optional step 510, the contrast module 206 enhances the contrast of each of the high-resolution monospectral IHRS images to improve their readability by a human operator. These high-resolution monospectral IHRS images can then be displayed on a display device (not shown) of the installation 100, such as a screen, for viewing by the operator.

[0063] With reference to [Fig.7], the image processing device 108 is, for example, a computer system comprising a data processing unit 702 (such as a microprocessor) and a main memory 704 (such as RAM, from the English "Random Access Memory") accessible by the processing unit 702. The computer system further comprises, for example, a network interface and / or a computer-readable medium, such as, for example, a local medium (such as a local hard disk 706) or a remote medium (such as a remote hard disk accessible via the network interface through a communication network) or a removable medium (such as a USB flash drive, from the English "Universal Serial Bus", or a CD, from the English "Compact Disc" or a DVD, from the English "Digital Versatile Disc") readable by means of an appropriate reader of the computer system (such as a USB port or a CD and / or DVD disc drive).A computer program 708 containing instructions for the processing unit 702 is stored on the storage medium 706 and / or downloadable via the network interface. This computer program 708 is, for example, intended to be loaded into the main memory 704, so that the processing unit 702 executes its instructions to implement the image processing procedure combining the preceding steps 506, 508, 510. For example, modules 202, 204, 206 and models M1 ... Ms are software modules of the computer program 708.

[0064] Alternatively, all or part of these modules could be implemented as hardware modules, i.e. in the form of an electronic circuit, for example micro-wired, not involving a computer program.

[0065] With reference to [Fig.8], an example of device 800 for implementing process 300 will now be described.

[0066] In this example, the device 800 is a computer system comprising a data processing unit 802 (such as a microprocessor) and a main memory 804 (such as RAM, or Random Access Memory) accessible by the processing unit 802. The computer system further comprises, for example, a network interface and / or a computer-readable medium, such as a local medium (such as a local hard drive 806) or a remote medium (such as a remote hard drive accessible via the network interface through a communication network) or removable media (such as a USB flash drive, a CD, or a DVD) readable by means of a suitable drive in the computer system (such as a USB port or a CD and / or DVD drive). A computer program 808 containing instructions for the processing unit 802 is stored on the medium 806 and / or downloadable via the network interface. This computer program 808 is, for example, intended to be loaded into main memory 804 so that the processing unit 802 can execute its instructions to implement process 300.

[0067] Alternatively, all or part of the steps of the process 300 could be implemented by hardware modules, i.e. in the form of an electronic circuit, for example micro-wired, not involving a computer program.

[0068] In conclusion, it should be noted that the invention is not limited to the embodiments described above. It will indeed be apparent to those skilled in the art that various modifications can be made to the embodiments described above, in light of the information just disclosed to them.

[0069] In the detailed presentation of the invention given above, the terms used shall not be interpreted as limiting the invention to the embodiments set forth in this description, but shall be interpreted as including all equivalents which can be foreseen by a person skilled in the art by applying their general knowledge to the implementation of the teaching which has just been disclosed to them.

Claims

Demands

1. An image processing method comprising: - a super-resolution reconstruction (506) of a high-resolution image (IHrs) of a celestial body (102), such as the Earth, from low-resolution images (IBR1...IBRN) acquired by an imager mounted on a vehicle (106) in a vehicle, such as an aircraft or a spacecraft, designed to move at a distance from the celestial body (102); characterized in that it further comprises: - deblurring (510) the high-resolution image (IHrs) by means of a model (Ms) previously trained by machine learning to cancel an intrinsic blur of the mounted imager (106) and a trailing blur resulting from the movement of the mounted imager (106) during the acquisition time of the low-resolution images (IBri...Ibrn

2. )• Method according to claim 1, wherein the low-resolution images (IBri...IBrn) are multispectral such that the reconstruction provides several high-resolution monospectral images (IHrs) in respective spectral bands, and wherein each high-resolution monospectral image (IHrs) is deblurred (510) by means of a respective model (Ms), dedicated to the spectral band of that high-resolution monospectral image (IHRS).

3. Method according to claim 1 or 2, wherein the model (Ms) is further pre-trained by machine learning to denoise the high-resolution image (IHRS).

4. A method according to any one of claims 1 to 3, wherein the model (Ms) comprises a convolutional neural network.

5. A method according to any one of claims 1 to 4, wherein the model (Ms) is designed to provide a deblurred image of the same resolution as the high-resolution image (IHrs) input to the model (Ms).

6. A method according to any one of claims 1 to 5, wherein the pre-training of the model (Ms) comprises: - for each of several images (IMG): • a simulation (306) of a burst shooting to obtain, from the image (IMG) considered, successive simulated images (ISi, Is2, Iss) all overlapping so as to present a common part, forming a reference image (Ir), • a blurring (310) of the successive simulated images (ISi, Is2, Iss) to simulate the intrinsic blur of the onboard imager (106) and the trailing blur resulting from the movement of the onboard imager (106) relative to the celestial body (102) during the acquisition time, and • a reconstruction (314) of a high-resolution blurred image (IF) by super-resolution from the simulated and blurred images (IS1, IS2, Iss); and - a training (316) of the model (Ms) in a supervised manner from the reference images (IR) and the blurred images (IF).

7. Method according to claim 6, wherein the pre-training of the model (Ms) further comprises, for each of the images (IMG), a noise (312) of the simulated (ISi, Is2, Iss) and blurred images.

8. Method according to claims 2 and 7 taken together, wherein the noise (312) and / or intrinsic blur depends on the spectral band to which the model (Ms) is dedicated.

9. A method according to any one of claims 6 to 8, wherein, except for the first simulated image (ISi), each of the other simulated images (IS2, Iss) is offset from the previous one by a respective offset (msl, ms2) in the image (IMG), comprising a constant offset and a random offset.

10. A method (300) for machine learning a model (Ms), such that the model (Ms) is designed to deblur a high-resolution image (IHrs) of a celestial body (102), such as the Earth, reconstructed by super-resolution from low-resolution images (IBRi...IBRn) taken by an onboard imager (106) in a vehicle, such as an aircraft or a spacecraft, designed to moving at a distance from the celestial body (102), deblurring aimed at canceling an intrinsic blur of the onboard imager (106) and a trailing blur resulting from a movement of the onboard imager (106) during a low-resolution image acquisition time (IBRi...IBRN), the method comprising: - for each of several images (IMG): • a simulation (306) of a burst shooting to obtain, from the image (IMG) considered, successive simulated images (IS1, IS2, Iss) all overlapping so as to present a common part, forming a reference image (Ir), • a blurring (310) of the successive simulated images (ISb Is2, Iss) to simulate the intrinsic blur of the onboard imager (106) and the trailing blur resulting from the displacement of the onboard imager (106) relative to the celestial body (102) during an acquisition time, and • a reconstruction (314) of a high-resolution blurred image (IF) by super-resolution from the simulated and blurred images (ISb Is2, Iss); and - a training (316) of the model (Ms) in a supervised manner from the reference images (IR) and the blurred images (IF).

11. Computer program (708) downloadable from a communication network and / or stored on a computer-readable medium, characterized in that it includes instructions for carrying out the steps of a process according to any one of claims 1 to 9, when said program is executed on a computer.

12. Computer program (808) downloadable from a communication network and / or stored on a computer-readable medium, characterized in that it includes instructions for carrying out the steps of a process according to claim 10, when said program is executed on a computer.

13. Method for observing (500) a celestial body (102), comprising:

14. - the acquisition (502) of successive low-resolution images (IBR i ... IBR N) of the celestial body (102) by an onboard imager (106); and - an image processing method according to any one of claims 1 to 9. Installation (100) for observing a celestial body (102), such as the Earth, comprising: - a vehicle (104), such as an aircraft or a spacecraft, designed to move at a distance from the celestial body and carrying an imager (106) designed to successively capture overlapping low-resolution images; and - an image processing device (108) designed to implement a method according to any one of claims 1 to 9.

Images