Method for controlling a drone using pre-recorded images of a target

Abstract

A method for controlling an autonomous vehicle (1) comprising at least one electronic control unit (2) and a first optronic device (5) connected to the electronic control unit to control the vehicle based on the position of a target (T) in initial images provided by the optronic device. The method comprises the steps of: capturing at least one second image (Ib) of the target with a second optronic device (15) that is separate from the vehicle and has a higher capture performance than the first optronic device; from said at least one second image, forming reference images (Ir) of the target at different close approaches and loading the reference images into the memory of the electronic control unit; having the electronic control unit search for the reference images in the initial images to steer the vehicle. ABSTRACT FIGURE: Fig. 1

Description

Title of the invention: Method for controlling a drone from pre-recorded images of a target

[0001] The present invention relates to the field of at least partially autonomous vehicles, in particular for navigation. These vehicles will be called drones in the remainder of the description.

[0002] BACKGROUND OF THE INVENTION

[0003] A drone is an aerial, spatial, nautical, terrestrial or amphibious vehicle which generally comprises a motorization to move the drone on a trajectory, at least one guidance member to modify the trajectory of the drone, on-board sensors and an electronic control unit executing a piloting program corresponding to a mission that the drone must accomplish. The mission always includes the movement of the drone from a starting point to a target or arrival point and for example the transport of passenger(s) or merchandise(s), surveillance by collecting information such as images or electromagnetic signals, jamming of electromagnetic signals, firing of a munition (the munition can be constituted by the drone itself)...

[0004] The control unit is connected to the guidance member and to the on-board sensors, as well as possibly to the motorization, and the program is arranged to control the guidance member and the motorization according to the information provided by the on-board sensors.

[0005] The sensors on board the most complex (and most expensive) drones include positioning sensors (for example an inertial unit and / or a satellite positioning signal receiver or GNSS) and environmental sensors allowing the electronic control unit to perceive its environment, such as an optronic image capture sensor, a radar, a lidar, a sonar, etc.

[0006] On the other hand, the least expensive drones only carry a limited number of sensors and more generally a single optronic image capture device. When the drone is not purely and simply remote-controlled from images sent to a remote piloting station (this is then referred to as remote guidance), the optronic image capture device is arranged to provide images to the electronic control unit which is itself programmed to search for a target in the images provided and to control the guidance member as a function of the position of the target in said images provided (this is generally referred to as autoguidance by target tracking or "tracking"). The electronic control unit executes for this purpose a image processing computer program using the images provided by the optronic image capture device. This assumes that the optronic image capture device has capture performance such that it provides images with a sufficient resolution / target size ratio for the image processing program to detect the target in the images. This also depends on the distance between the drone and the target.

[0007] SUBJECT OF THE INVENTION

[0008] The invention aims in particular to control by autoguiding a drone provided with an image capture device having limited performance. Summary of the invention

[0009] For this purpose, the invention provides a method for controlling an autonomous vehicle comprising at least one electronic control unit and a first optronic device connected to the electronic control unit and arranged to provide first images to the electronic control unit which is programmed to search for a target in the first images and to steer the vehicle according to the position of the target in the first images. The method comprises the steps of: - capturing at least a second image of the target with a second optronic device which is separate from the vehicle and which has an image generation performance greater than an image generation performance of the first optronic device; - from said at least one second image, form reference images of the target at different approaches and load the reference images into the memory of the electronic control unit; - have the electronic control unit search for reference images in the first images to steer the vehicle.

[0010] Thus, the method of the invention is implemented by means of two optronic image generation devices, namely a first optronic device on board the vehicle and a second optronic device which is not on board the vehicle. The second optronic device is for example a telescope, binoculars, a viewfinder, etc. and the image generation performance of the second optronic device is superior to that of the first optronic device. The better image generation performance of the second optronic device concerns the capture itself of the image and / or subsequent processing and may for example result from the presence in the second optronic device: of a focal length adjustment device, and / or of a higher resolution sensor, and / or of a computing unit sufficiently powerful to implement super-resolution processing.The vehicle control is achieved by exploiting the performance of the second . optronic device (the most efficient) to form reference images that can be used by the vehicle control unit in combination with the images provided by the first optronic device (the least efficient). It is therefore not necessary to carry a very efficient and therefore expensive optronic image capture device on the vehicle. Furthermore, it should be noted that the method works even if the target is not visible in the reference images corresponding to the lowest approximations (in fact, the size of the target may be less than one pixel in said images which actually shows the target's environment): the vehicle's electronic control unit searches for the reference images in the first images whether the target is detectable or not in the reference images.

[0011] Other characteristics and advantages of the invention will emerge from reading the following description of a particular and non-limiting embodiment of the invention. Brief description of the drawings

[0012] Reference will be made to the accompanying drawings, among which:

[0013] [Fig.l] [Fig.l] is a schematic representation illustrating the principle of the method according to the invention;

[0014] [Fig.2] [Fig.2] is a schematic view of a drone that can be used for the implementation work of invention;

[0015] [Fig.3] [Fig.3] is a schematic view of a remote optronic image capture device that can be used to implement the invention;

[0016] [Fig.4] [Fig.4] is a schematic representation illustrating an implementation of the method according to the invention;

[0017] [Fig.5a] [Fig.5a] is a representation of a first reference image for implementing the invention;

[0018] [Fig.5b] [Fig.5b] is a representation of a second reference image for implementing the invention;

[0019] [Fig.5c] [Fig.5c] is a representation of a third reference image for implementing the invention;

[0020] [Fig.5d] [Fig.5d] is a representation of a fourth reference image for implementing the invention;

[0021] [Fig.6] [Fig.6] is a flowchart illustrating the implementation of the method according to the invention;

[0022] [Fig.7] [Fig.7] is a schematic representation illustrating one of the processes possible ways of developing reference images. DETAILED DESCRIPTION OF THE INVENTION

[0023] With reference to [Fig.l], the invention is described here in relation to a drone 1, here aerial, and a remote station 11.

[0024] Also referring to [Fig.2], the drone 1 comprises a control unit 2 connected to: a memory 3, a transmitter / receiver 4, a first optronic image capture device 5, a motorization 6 and a steering organ 7 (or guidance organ).

[0025] The control unit 2 is an electronic unit comprising for example one or more processors and the memory 3 contains an operating program (commonly called OS) and one or more computer programs comprising instructions arranged to implement the method of the invention. The control unit 2 is for example produced in the form of one or more integrated circuits, an FPGA or other. The control unit 2 is powered by an on-board electrical network comprising at least one battery. The memory 3 is accessible by the control unit 2 which is arranged to execute the programs contained in the memory 3.

[0026] The transmitter / receiver 4 is a radio transmitter / receiver which is arranged to communicate via a communication channel with the remote station 11 and which preferably incorporates an electronic unit for encrypting and decrypting the transmitted data. The communication channel may be a direct communication channel, a satellite communication channel passing through a communication satellite or any other communication channel (for example relayed by an aircraft, a vehicle or a land station, a vehicle or a maritime station, etc.). The transmitter / receiver 4 is not intended here to be used during the flight of the drone 1 but only to transfer data before the flight. Alternatively, it is therefore possible to establish a wired connection between the control unit 2 and the remote station 11 to allow an exchange of data.

[0027] The optronic device 5 conventionally comprises a sensor which operates in the visible and infrared ranges and which is oriented to capture images Is of the environment in front of the drone 1.

[0028] The control unit 2 is connected to other sensors and in particular here: - status sensors of the various components / equipment of the drone 1; - an inertial unit comprising, in a manner known per se, acceler- rometers for detecting a specific force along the axes of a measuring mark and angular sensors of the orientation of the measuring mark relative to a reference mark.

[0029] The motorization 6 is arranged to propel the drone along an aerial trajectory and here comprises electric motors driving propellers (for example, the drone 1 here is of the quadricopter type).

[0030] The governing body 7 here comprises an electronic device making it possible to control the rotation speed of the electric motors driving the propellers independently of each other to modify the trajectory of the drone 1.

[0031] The control unit 2 is programmed, in a manner known per se, to control the motorization 6 and the steering organ 7 along an aerial trajectory while maintaining the drone 1 in an appropriate attitude using the data from the inertial unit.

[0032] The control unit 2 is also programmed, in a manner known per se, to accomplish a mission consisting, for example, of taking the drone from a starting point D to an arrival point A and firing a munition at a target T located at the arrival point A.

[0033] The control unit 2 is here programmed to implement a self-guiding method consisting of searching for a target in images Is continuously supplied by the optronic device 5 and to control the steering member 6 as a function of the position of the target T in the images Is. The target is searched for from reference images Ir which are recorded in the memory 3.

[0034] The reference images Ir are obtained from basic images Ib provided by the remote station IL

[0035] With reference to [Fig.3], the remote station 11 comprises a control unit 2 connected to: a memory 13, a transmitter / receiver 14 and an optronic image capture device 15.

[0036] The control unit 12 is an electronic unit comprising for example one or more processors and the memory 13 contains an operating program (commonly called OS) and one or more computer programs comprising instructions arranged to implement the method of the invention. The control unit 12 is for example produced in the form of one or more integrated circuits, an FPGA or other. The memory 13 is accessible by the control unit 12 which is arranged to execute the programs contained in the memory 13.

[0037] The transmitter / receiver 14 is a radio transmitter / receiver which is arranged to communicate via a communication channel with the drone 1 and which preferably incorporates an electronic unit for encrypting and decrypting the transmitted data. The communication channel may be a direct communication channel, a satellite communication channel passing through a communication satellite or any other communication channel (for example relayed by an aircraft, a vehicle or a ground station, a vehicle or a maritime station, etc.). Alternatively, it is possible to establish a wired connection between the control units 2 and 12 to allow an exchange of data.

[0038] The optronic device 15 is here a pair of binoculars and conventionally comprises a sensor which operates in the visible and infrared ranges and has performances of image generation higher than those of the optronic device 5. In particular, the optronic device 15 comprises an optic with adjustable focal length (optical zoom) while the optronic device 5 has a fixed focal length and the sensor of the optronic device 15 has a maximum resolution higher than a maximum resolution of the sensor of the optronic device 5.

[0039] During a preliminary mission preparation phase, prior to the execution of the mission by the drone 1, an operator captures the basic images Ib by means of the image capture device 15 of the remote station 11 and the control unit 12 produces the reference images Ir from the basic images Ib (see in particular figures 1 and 4).

[0040] Three processes for developing reference images Ir, which may be combined, are considered here.

[0041] In the first process, the operator points the optronic device 15 at the target T and captures a high-resolution base image Ib of the target T using a small focal length so as to have a wide viewing angle. This base image Ib is recorded in the memory 13 and the control unit 12 will develop a succession of n reference images Ir by performing for each of these a sub-sampling and a resizing of the base image Ib. Thus, in the first process of developing the reference images Ir, a single base image Ib gives rise to several reference images Ir.

[0042] In the second process, the operator points the optronic device 15 at the target T and captures a number of m basic images Ib by adjusting the optics of the optronic device 15 to a different focal length each time. For example, the operator captures a first basic image Ib#l with a large focal length (so that the target T appears in the basic image Ib#l close to the optronic device 15), then a basic image Ib#2 with a slightly smaller focal length (so that the target T appears in the basic image Ib#2 a little further from the optronic device 15 than in the basic image Ib#l), then a basic image Ib#3 with an even smaller focal length (so that the target T appears in the basic image Ib#3 a little further from the optronic device 15 than in the basic image Ib#2) and so on until the last basic image Ib#M captured with the shortest focal length.The m basic images Ib are recorded in the memory 13 and the control unit 12 will, after processing or not, develop n reference images Ir (m being equal to n in the present case but this is not obligatory) by making each of the basic images Ib a reference image Ir. The number of reference images Ir required depends on the distance separating the drone 1 and the target T and on the capture performance (in particular the maximum resolution) of the optronic device 5. In the most basic implementation of the second process, each image of . base Ib gives rise to a single reference image Ir. Nevertheless, it is possible to carry out, on one or more of the base images Ib, a processing allowing several reference images Ir to be obtained for a single base image Ib.

[0043] In the third process illustrated more particularly in [Fig.7], the operator points the optronic device 15 at the target T and captures k basic images Ib of the target T in such a way that the basic images Ib are partially superimposable. These basic images Ib are recorded in the memory 13 and the control unit 12 will carry out super-resolution processing on these images to obtain a reference image Ir#l. Thus, in the third process of developing the reference images Ir, one or more basic images Ib give rise to one or more reference images Ir.

[0044] In the described embodiment, as an illustrative example, the first two processes are combined (see [Fig.4]) using the base images Ib#l, Ib#2, ... Ib#M captured according to the second process: the base image Ib#l alone is used to develop the reference image Ir#l by cutting and subsampling; the base image Ib#l and the base image Ib#Ml are used to develop the reference image Ir#2 by subsampling, resizing and combining; the base image Ib#Ml is used as is to develop the reference image Ir#Nl, and the base image Ib#M is used as is to develop the reference image Ir#N.

[0045] At the end of the process of developing the reference images Ir, the control unit 2 classifies the reference images Ir according to the size of the target T in the reference image Ir (this is also referred to as an ordering in ascending order of the resolution of the target in the reference image Ir or by apparent approximation or magnification of the target in the reference image Ir). As an example, figures 5a to 5d show four reference images referenced from Irl to IR4. The reference image Irl is the reference image in which the target appears closest to the observer while the reference image Ir4 is the reference image in which the target appears furthest from the observer. It will also be noted that, in images Ir3 and Ir4, the target is barely visible while it can be clearly identified in images Irl and Ir2.The n reference images Ir thus ordered are transmitted to the drone 1 via the transmitters / receivers 14, 4 and recorded in the memory 3 of the drone 1.

[0046] Furthermore, during this same preliminary phase, the operations to be carried out during the mission (here firing a munition at the target T) are programmed in the control unit 2.

[0047] Once this is done, the operator commands the control unit 2 to start the drone 1 ( [Fig.6]).

[0048] The control unit 2 continuously activates the optronic device 5, the motorization 6 and the steering organ 7 so that the drone 1 rises vertically and pivots so that the optronic device 5 scans its environment. The tracking program executed by the control unit 2 selects the first reference image Ir4 to make it the current reference image (step 101) and searches, in the images Is provided by the optronic device 5, the reference image Ir4 and its position in the images Is (locking, step 102). Once this has been found, the control unit 2 controls the steering organ 7 in such a way that the drone 1 heads towards the target T by ensuring that the reference image Ir4 remains in the field of the images Is provided and for example in the center of the images Is provided (control of the steering of the steering organ 7 on the position of the target in the image Is, step 106).

[0049] The search for the reference images in the images Is is carried out in a conventional manner and comprises a calculation of a correlation score between each reference image Ir and the images Is (step 103). As the drone 1 approaches the target T, the correlation score between the images Is and the current reference image (Ir4) and the correlation score between the images Is and the next reference image (Ir3) are compared with each other and the reference image having the best correlation score is selected as the current reference image (this is of interest in the case where the reference images have been classified as indicated above).Alternatively, at each instant, the correlation scores between the images Is and each reference image can be calculated and compared, and the one with the best correlation score can be selected as the current reference image (this is useful in the case where the reference images have not been classified). The control unit 2 then selects the reference image Ir3 as the current image and searches for it in the images Is: the correlation score will then start to increase again, and so on with each reference image. In the implementation mode described, the tracking program: . - continuously calculates the correlation score (current Ref correlation) between the reference image (Ir4 for example) and the Is images and the correlation score (next Ref correlation) between the next reference image (Ir3 in this example) and the Is images (step 103); - compares these two scores to each other (step 104); - if the correlation score (current Ref correlation) between the reference image (Ir4) and the Is images is greater than the correlation score (next Ref correlation) between the next reference image (Ir3) and the Is images, the servo-control is maintained on the position of the reference image Ir4 in the Is images; - if the correlation score (current Ref correlation) between the reference image (Ir4) is less than the correlation score (next Ref correlation) between the next reference image (Ir3) and the Is images, the next reference image (Ir3) becomes the current reference image (step 105) and the servoing is carried out according to the position of the reference image Ir3 in the Is images.

[0050] The target T is considered to be reached as a function of an apparent size of the target T in the first images Is. According to a first implementation assuming knowledge of the actual size of the target T and the characteristics of the optronic device 5, the distance from the drone 1 to the target T can be determined by stadimetry as a function of the apparent size of the target T in the images Is. According to a second implementation, the temporal evolution of the apparent size of the target in the images Is makes it possible to calculate a time before impact at constant speed. Other implementations are obviously possible, for example using: a statistical model linking apparent sizes of targets and distances to the target; a rangefinder; a pressure sensor, etc.When target T is considered to have been reached (step 107), the action to be performed is carried out and for example: the ammunition is armed and fired; drone 1 is brought to the racetrack around target T to take images of it... .

[0051] It will be noted that the method works even if the target is not visible in the reference images corresponding to the smallest approximations (in fact, the size of the target may be less than one pixel in said images which actually shows the environment of the target): the electronic control unit of the drone 1 searches for the reference images Ir in the first images Is whether the target T is detectable or not in the reference images Ir (in [Fig.l], we see that the target is not visible in the reference image Ir#N).

[0052] Of course, the invention is not limited to the embodiment described but encompasses any variant falling within the scope of the invention as defined by the claims.

[0053] In particular, the drone may have a structure different from that described.

[0054] The drone may include other sensors replacing or adding to those mentioned previously: - a lidar, - a sonar, - a receiver of satellite positioning signals (or GNSS receiver) usable for example on a portion of the route in the absence of jamming, - a radar to detect the possible presence of an obstacle in front of the drone 1, - an optronic sensor in the infrared range or in the visible range, - an electromagnetic signal sensor, - a threat detector and for example a missile or missile departure detector shot,

[0055] The drone may not include an inertial unit, whether for its stabilization or its navigation, and operate solely by any type of visual servoing. When the line of sight of the optronic system is exactly in the axis of advance of the drone, the visual servoing aims to orient the drone so that the target is imaged in the center of its field of observation. But in the case of a drone requiring an inclination to advance (as is the case with the majority of quadcopters) with a camera in the axis of the body of the drone, it will be necessary to seek to position the target in the upper field of the image.

[0056] The optronic device 5 may comprise a sensor which operates in the visible and / or infrared ranges and which is oriented / orientable to capture images of the environment in front of the drone 1 or all around it.

[0057] The motorization 6 is here arranged to propel the drone along an aerial trajectory and can comprise a propeller of the reactor, turbojet, ramjet, turboprop type, etc.

[0058] The steering organ 7 may comprise steerable motorized fins or steerable propellers or any other controllable mechanism making it possible to modify the trajectory of the drone.

[0059] During the preliminary phase, it is possible to program waypoints (these waypoints being considered as intermediate targets), and other types of operations to be carried out during the mission such as dropping off a package, remaining under observation around the target, activating a communication jammer, relaying communications, etc.

[0060] Visual servoing may only be used on a portion of the drone's path. For example, the drone 1 could be remotely controlled on an initial portion of the path and then be self-guided by visual servoing on a final portion of the path.

[0061] The drone can be any land, water, air, amphibious, space vehicle, etc., at least partially autonomous. This vehicle can be designed to transport one or more passengers, a payload (surveillance equipment, ammunition, etc.), merchandise, etc. The vehicle can also be a missile or a prowling munition, etc.

[0062] In the embodiment described, it is provided that, to begin the flight of the drone 1, the control unit 2 continuously activates the optronic device 5, the motorization 6 and the steering organ 7 so that the drone 1 rises vertically and pivots so that the optronic device 5 scans its environment. Alternatively, the operator can orient the drone in the right direction or enter a heading value if the drone has a magnetic compass that the electronic control unit 2 can use to pilot the drone 1.

[0063] The reference images can be obtained by one or more of the processes described and each process can also be modified. For example, only a cutting or only a sub-sampling of the base image can be carried out. The reference images can also be obtained by different processes and for example by averaging (in a weighted or unweighted manner) base images.

[0064] The optronic device 15 may comprise an optic with adjustable focal length (optical zoom) and / or a sensor having a maximum resolution greater than a maximum resolution of the sensor of the optronic device 5 and / or a powerful electronic computing unit for processing the basic images, for example in super-resolution.

[0065] The optronic device 15 can be a portable device or one mounted on a vehicle such as a land vehicle, a boat, an airplane, a satellite, another drone, etc. It can be a telescope, binoculars, a viewfinder, a pointing device, an optronic ball, etc.

Claims

Claims

1. Method for controlling an autonomous vehicle (1) comprising at least one electronic control unit (2) and a first optronic device (5) connected to the electronic control unit and arranged to provide first images (Is) to the electronic control unit which is programmed to search for a target (T) in the first images and to steer the vehicle according to the position of the target in the first images, characterized in that the method comprises the steps of: - capturing at least one second image (Ib) of the target with a second optronic device (15) which is separate from the vehicle and which has an image generation performance greater than an image generation performance of the first optronic device; - from said at least one second image, forming reference images (Ir) of the target at different approaches and loading the reference images into the memory of the electronic control unit;- have the electronic control unit search for the reference images (Ir) in the first images (Is) to steer the vehicle.;

2. Method according to claim 1, in which the second capturing device (15) comprises an optic with adjustable focal length and several second images (Ib) are captured at different focal lengths each corresponding to an apparent magnification of the target.

3. A method according to claim 1 or 2, wherein the second image (Ib) is captured at a predetermined resolution greater than a maximum resolution of the first capture device (5) and at least one of the reference images (Ir) is extracted from the second image by cutting and / or reducing the resolution.

4. Method according to any one of the preceding claims, in which at least one of the reference images (Ir) results from a super-resolution processing carried out on the second image (Ib).

5. A method according to any preceding claim, in in which the reference images (Ir) are classified according to the approach to the target (T) in the reference images.

6. Method according to any one of the preceding claims, wherein the number of reference images (Ir) depends on a distance separating the vehicle (1) and the target (T) and on the capture performance of the first optronic device (5).

7. Method according to any one of the preceding claims, in which the capture of each second image (Ib) is carried out from a starting point (D) of the vehicle (1).

8. Method according to any one of the preceding claims, in which the search for the reference images (Ir) in the first images (Is) comprises a calculation of a correlation score between each reference image and at least one of the first images.

9. Method according to claim 8, wherein the first control unit (2): - continuously calculates the correlation score (current Ref correlation) between the current reference image and the first images (Is) and a correlation score (next Ref correlation) between at least one next reference image and the first images (step 103); - compares these two scores with each other (step 104); - if the correlation score (current Ref correlation) between the current reference image (Ir4) and the first images is lower than the correlation score (next Ref correlation) between the next reference image (Ir3) and the first images, the next reference image becomes the current reference image.

10. A method according to any preceding claim, wherein the target (T) is considered to be reached based on an apparent size of the target in the first images (Is).

Images