System and method for controlling movable object to track target

Abstract

Systems and methods are provided that may support target tracking. A controller may obtain a feature model of a target, wherein the feature model represents an image characteristic of the target. Further, the controller may extract one or more features from one or more images captured by an imaging device carried by a movable object, and may apply the feature model to the one or more features to determine a similarity.

Description

Technical Field

[0001] The disclosed embodiments relate generally to enabling target following, and more particularly, but not exclusively, to smooth target following. Background Art

[0002] Aircraft such as unmanned aerial vehicles (UAVs) can be used to perform surveillance, reconnaissance and exploration missions for various applications. Due to the high popularity of UAVs and the high demand of the UAV market, there is a need to add more attractive functions to UAVs. For example, it is desirable to enable UAVs to automatically track objects. However, despite recent achievements in object tracking, object tracking is still challenging in computer vision (e.g., objects may experience deformation, occlusion and even become invisible over a long period of time). This is a general aspect that embodiments of the present invention are intended to address. Summary of the invention

[0003] Systems and methods are described herein that can support target tracking. A controller can obtain a feature model of a target, wherein the feature model represents image characteristics of the target. In addition, the controller can extract one or more features from one or more images captured by an imaging device carried by a movable object, and can apply the feature model to the one or more features to determine similarity.

[0004] Also described herein are systems and methods that can support target tracking. A controller can apply a feature model of a target to an image captured for tracking a target to generate a tracking output. In addition, the controller can determine that a tracking failure has occurred based on the tracking output, and can examine one or more subsequent images to detect the target.

[0005] Systems and methods that can support target tracking are also described herein. A controller can acquire a target from one or more images captured by an imaging device carried by a movable object. In addition, the controller can obtain a relative distance between the movable object and the target, and can generate one or more control signals to track the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A movable object environment according to various embodiments of the present invention is shown.

[0007] Figure 2 An exemplary carrier in a movable object environment according to an embodiment is shown.

[0008] Figure 3 It is demonstrated that different embodiments according to the present invention support target tracking in a movable object environment.

[0009] Figure 4An exemplary target tracking system in a movable object environment according to various embodiments of the present invention is presented.

[0010] Figure 5 It is demonstrated that different embodiments according to the present invention support target tracking in a movable object environment.

[0011] Figure 6 Initializing target tracking in a movable object environment according to various embodiments of the present invention is demonstrated.

[0012] Figure 7 Tracking of a target in a movable object environment according to various embodiments of the present invention is demonstrated.

[0013] Figure 8 The perspective transformation relationship between two images captured for tracking a target in a movable object environment according to various embodiments of the present invention is demonstrated.

[0014] Fig. 9 The use of inverse perspective transforms to support tracking in a movable object environment is demonstrated according to various embodiments of the present invention.

[0015] Fig.10 A flowchart of tracking a target in a movable object environment according to various embodiments of the present invention is shown.

[0016] Fig.11 It is demonstrated that different embodiments according to the present invention support target tracking and re-detection in a movable object environment.

[0017] Fig.12 The use of a positioning device to assist in target tracking in a movable object environment according to various embodiments of the present invention is demonstrated.

[0018] Fig.13 A flow chart supporting target tracking and re-detection in a movable object environment according to different embodiments of the present invention is shown.

[0019] Fig.14 Tracking of a target based on range measurements in a movable object environment according to various embodiments of the present invention is demonstrated.

[0020] Fig.15 It is demonstrated that relative target distances are measured in a movable object environment according to various embodiments of the present invention.

[0021] Fig.16 A flow chart of tracking a target based on distance measurement in a movable object environment according to various embodiments of the present invention is shown. DETAILED DESCRIPTION

[0022] The present invention is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like reference numerals indicate similar elements. It should be noted that references to "one" or "an" or "some" embodiments in the present disclosure do not necessarily refer to the same embodiments, and such references mean at least one embodiment.

[0023] In the following description of the present invention, an unmanned aerial vehicle (UAV) is used as an example of a movable object. It will be clear to those skilled in the art that other types of movable objects may be used without limitation.

[0024] According to various embodiments of the invention, the system can automatically track and detect a target over an extended period of time, for example using a mobile object such as an unmanned aerial vehicle (UAV) that has limited resources (e.g., in terms of both computing power and power resources). Additionally, the system can provide the ability to relocate the target once it is lost.

[0025] Figure 1 The movable object environment according to different embodiments of the present invention is shown. Figure 1 , a movable object 118 in movable object environment 100 may include a carrier 102 and a payload 104. Although movable object 118 may be depicted as an aircraft, this depiction is not intended to be limiting, and any suitable type of movable object may be used. Those skilled in the art will recognize that any embodiments described herein in the context of an aircraft system may be applied to any suitable movable object (e.g., a UAV). In some cases, payload 104 may be provided on movable object 118 without carrier 102.

[0026] According to various embodiments of the invention, movable object 118 may include one or more movement mechanisms 106 (eg, propulsion mechanisms), sensing systems 108 , and communication systems 110 .

[0027] The mobile mechanism 106 may include one or more of a rotor, a propeller, a blade, an engine, a motor, a wheel, an axle, a magnet, a nozzle, an animal, or a human. For example, the movable object may have one or more propulsion mechanisms. The mobile mechanisms 106 may all be of the same type. Alternatively, the mobile mechanisms 106 may be different types of mobile mechanisms. The mobile mechanism 106 may be mounted on the movable object 118 (or vice versa) using any suitable means, such as a support element (e.g., a drive shaft). The mobile mechanism 106 may be mounted on any suitable portion of the movable object 118, such as, mounted on its top, bottom, front, rear, side, or a suitable combination.

[0028] In some embodiments, the mobile mechanism 106 can enable the movable object 118 to take off vertically from the ground or land vertically on the ground without the movable object 118 performing any horizontal movement (e.g., without taxiing along a runway). Optionally, the mobile mechanism 106 can be operable to allow the movable object 118 to hover in the air at a specified position and / or orientation. One or more of the mobile mechanisms 106 can be controlled independently of other mobile mechanisms. Alternatively, the mobile mechanism 106 can be configured to be controlled simultaneously. For example, the movable object 118 can have multiple horizontally oriented rotors, which can provide lift and / or thrust to the movable object. The multiple horizontally oriented rotors can be actuated to enable the movable object 118 to take off vertically, land vertically, and hover. In some embodiments, one or more of the horizontally oriented rotors can rotate in a clockwise direction, while one or more of the horizontal rotors can rotate in a counterclockwise direction. For example, the number of clockwise rotors can be equal to the number of counterclockwise rotors. The rotation rate of each of the horizontally oriented rotors may be independently varied to control the lift and / or thrust generated by each rotor and thereby adjust the spatial disposition, velocity, and / or acceleration of movable object 118 (e.g., with respect to up to three degrees of translation and up to three degrees of rotation).

[0029] The sensing system 108 may include one or more sensors that can sense the spatial disposition, velocity, and / or acceleration of the movable object 118 (e.g., with respect to different degrees of translation and different degrees of rotation). The one or more sensors may include any sensor, including a GPS sensor, a motion sensor, an inertial sensor, a proximity sensor, or an image sensor. The sensed data provided by the sensing system 108 may be used to control the spatial disposition, velocity, and / or orientation of the movable object 118 (e.g., using a suitable processing unit and / or control module). Alternatively, the sensing system 108 may be used to provide data related to the environment surrounding the movable object, such as weather conditions, proximity to potential obstacles, location of geographic features, location of man-made structures, and the like.

[0030] The communication system 110 enables communication with a terminal 112 having a communication system 114 via a wireless signal 116. The communication systems 110, 114 may include any number of transmitters, receivers and / or transceivers suitable for wireless communication. The communication may be a one-way communication, so that data can only be transmitted in one direction. For example, a one-way communication may only involve the movable object 118 transmitting data to the terminal 112, or vice versa. Data may be transmitted from one or more transmitters of the communication system 110 to one or more receivers of the communication system 112, or vice versa. Alternatively, the communication may be a two-way communication, so that data can be transmitted in two directions between the movable object 118 and the terminal 112. The two-way communication may involve transmitting data from one or more transmitters of the communication system 110 to one or more receivers of the communication system 114, and vice versa.

[0031] In some embodiments, terminal 112 may provide control data to and receive information from one or more of movable object 118, carrier 102, and payload 104 (e.g., position and / or motion information of the movable object, carrier, or payload; data sensed by the payload, such as image data captured by a payload camera; and data generated by image data captured by a payload camera). In some cases, the control data from the terminal may include instructions for the relative position, movement, actuation, or control of the movable object, carrier, and / or payload. For example, the control data may result in a modification of the position and / or orientation of the movable object (e.g., by controlling movement mechanism 106) or the movement of the payload relative to the movable object (e.g., by controlling carrier 102). Control data from the terminal can result in control of the payload, such as controlling the operation of a camera or other image capture device (e.g., taking still or dynamic photos, zooming in or out, turning on or off, switching imaging modes, changing image resolution, zooming, changing depth of field, changing exposure time, changing viewing angle or field of view).

[0032] In some cases, the communication from the movable object, carrier and / or payload may include information from one or more sensors (e.g., of the sensing system 108 or the payload 104) and / or data generated based on the sensed information. The communication may include sensing information from one or more different types of sensors (e.g., GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors). Such information may be about the position (e.g., location, orientation), movement, or acceleration of the movable object, carrier, and / or payload. Such information from the payload may include data captured by the payload or the sensed state of the payload. The control data transmitted by the terminal 112 may be configured to control the state of one or more of the movable object 118, the carrier 102, or the payload 104. Alternatively or in combination, the carrier 102 and the payload 104 may each further include a communication module configured to communicate with the terminal 112 so that the terminal can independently communicate with and control each of the movable object 118, the carrier 102, and the payload 104.

[0033] In some embodiments, the movable object 118 may be configured to communicate with another remote device in addition to or in place of the terminal 112. The terminal 112 may also be configured to communicate with another remote device as well as the movable object 118. For example, the movable object 118 and / or the terminal 112 may communicate with another movable object or a carrier or payload of another movable object. Where desired, the remote device may be a second terminal or other computing device (e.g., a computer, laptop, tablet, smartphone, or other mobile device). The remote device may be configured to transmit data to the movable object 118, receive data from the movable object 118, transmit data to the terminal 112, and / or receive data from the terminal 112. Optionally, the remote device may be connected to the Internet or other telecommunications network so that data received from the movable object 118 and / or the terminal 112 can be uploaded to a website or server.

[0034] Figure 2 An exemplary carrier in a movable object environment according to an embodiment is shown. Carrier 200 can be used to couple a payload 202 (such as an image capture device) to a movable object (such as a UAV).

[0035] The carrier 200 may be configured to allow the payload 202 to rotate relative to the movable object about one or more axes, such as three axes: an X-axis or pitch axis, a Z-axis or roll axis, and a Y-axis or yaw axis. For example, the carrier 200 may be configured to allow the payload 202 to rotate about only one, two, or three of the axes. The axes may be orthogonal to each other or not. The range of rotation about any of the axes may or may not be limited and may vary for each of the axes. The axes of rotation may or may not intersect with each other. For example, orthogonal axes may intersect with each other. They may intersect or not intersect at the payload 202. Alternatively, they may not intersect.

[0036] The carrier 200 may include a frame assembly 211 including one or more frame members. For example, the frame members may be configured to couple with and support a payload 202 (eg, an image capture device).

[0037] In some embodiments, the carrier 201 may include one or more carrier sensors 213 that may be used to determine the state of the carrier 201 or the payload 202 carried by the carrier 201. The state information may include spatial disposition (e.g., position, orientation, or attitude), velocity (e.g., linear velocity or angular velocity), acceleration (e.g., linear acceleration or angular acceleration), and / or other information related to the carrier, its components, and / or the payload 202. In some embodiments, the state information obtained or calculated from the sensor data may be used as feedback data to control the rotation of components of the carrier (e.g., frame members). Examples of such carrier sensors may include motion sensors (e.g., accelerometers), rotation sensors (e.g., gyroscopes), inertial sensors, and the like.

[0038] The carrier sensors 213 may be coupled to any one or more suitable portions of the carrier (e.g., frame members and / or actuator members), and may or may not be movable relative to the UAV. Additionally or alternatively, at least some of the carrier sensors may be directly coupled to the payload 202 carried by the carrier 201.

[0039] The carrier sensor 213 may be coupled to some or all of the actuator members of the carrier. For example, three carrier sensors may be coupled to the actuator members 212 of the three-axis carrier, respectively, and configured to measure the drive of the corresponding actuator members 212 of the three-axis carrier. Such sensors may include potentiometers or other similar sensors. In an embodiment, a sensor (e.g., a potentiometer) may be inserted into the motor shaft of the motor to facilitate measuring the relative position of the motor rotor and the motor stator, thereby measuring the relative position of the rotor and the stator and generating a position signal representing the relative position. In an embodiment, each sensor coupled to the actuator is configured to provide a position signal of the corresponding actuator member measured by it. For example, a first potentiometer may be used to generate a first position signal of a first actuator member, a second potentiometer may be used to generate a second position signal of a second actuator member, and a third potentiometer may be used to generate a third position signal of a third actuator member. In some embodiments, the carrier sensor 213 may also be coupled to some or all of the frame members of the carrier. The sensor may be capable of transmitting information related to the position and / or orientation of one or more frame members and / or an image capture device of the carrier. The sensor data may be used to determine the position and / or orientation of the image capture device relative to the movable object and / or the reference frame.

[0040] The carrier sensor 213 can provide position and / or orientation data that can be transmitted to one or more controllers (not shown) on the carrier or movable object. The sensor data can be used in a feedback-based control scheme. The control scheme can be used to control the drive of one or more actuator members (such as one or more motors). One or more controllers (which can be located on the carrier or on the movable object carrying the carrier) can generate control signals for driving the actuator members. In some cases, the control signal can be generated based on data received from the carrier sensor indicating the spatial deployment of the carrier or the payload 202 carried by the carrier 201. As previously described herein, the carrier sensor can be located on the carrier or the payload 202. The control signal generated by the controller can be received by different actuator drivers. Based on the control signal, different actuator drivers can control the drive of different actuator members, for example, to achieve the rotation of one or more components of the carrier. The actuator driver can include hardware and / or software components that are suitable for controlling the drive of the corresponding actuator member and receiving the position signal from the corresponding sensor (e.g., potentiometer). The control signal can be transmitted to the actuator driver simultaneously to generate the simultaneous drive of the actuator member. Alternatively, the control signals may be transmitted sequentially or to only one of the actuator drivers.Advantageously, the control scheme may be used to provide feedback control of the actuator member driving the carrier, thereby enabling more precise and accurate rotation of the carrier component.

[0041] In some cases, carrier 201 can be indirectly connected to UAV through one or more damping elements. Damping elements can be configured to reduce or eliminate the movement of loads (e.g., payload, carrier, or both) caused by the movement of movable objects (e.g., UAV). Damping elements can include any element suitable for damping the motion of the connected load, such as active damping elements, passive damping elements, or hybrid damping elements with active and passive damping characteristics. The motion damped by the damping element provided herein can include one or more of vibration, oscillation, shaking, or impact. This type of motion can be derived from the motion of the movable object transferred to the load. For example, motion can include vibrations caused by the operation of the propulsion system and / or other components of the UAV.

[0042] The damping element can provide motion damping by isolating the load from sources of undesired motion (e.g., vibration isolation) by counteracting or reducing the amount of motion imparted to the load. The damping element can reduce the magnitude (e.g., amplitude) of motion that the load would otherwise experience. The motion damping applied by the damping element can be used to stabilize the load, thereby improving the quality of images captured by the load (e.g., an image capture device) and reducing the computational complexity of the image stitching step required to generate a panoramic image based on the captured images.

[0043] The damping element described herein can be formed by any suitable material or material combination, including solid, liquid, or gaseous materials. The material used for the damping element can be compressible and / or deformable. For example, the damping element can be made of sponge, foam, rubber, gel, etc. For example, the damping element can include a rubber ball that is substantially spherical in shape. The damping element can have any suitable shape, such as substantially spherical, rectangular, cylindrical, etc. Alternatively or in addition, the damping element can include a piezoelectric material or a shape memory material. The damping element can include one or more mechanical elements, such as springs, pistons, hydraulic parts, pneumatic parts, buffers, shock absorbers, isolation devices, etc. The characteristics of the damping element can be selected so as to provide a predetermined amount of motion damping. In some cases, the damping element can have viscoelastic properties. The characteristics of the damping element can be isotropic or anisotropic. For example, the damping element can provide motion damping that is equal along all directions of motion. On the contrary, the damping element can only provide motion damping along a subset of the directions of motion (e.g., along a single direction of motion). For example, the damping element may provide damping primarily along the Y (yaw) axis. Thus, the damping element shown may be configured to reduce vertical motion.

[0044] Although different embodiments may be depicted as utilizing a single type of damping element (e.g., a rubber ball), it should be understood that any combination of suitable types of damping elements may be used. For example, one or more damping elements of any suitable type or types may be used to couple the carrier to the movable object. The damping elements may have the same or different features or characteristics, such as stiffness, viscoelasticity, and the like. Each damping element may be coupled to a different portion of the load or only to a portion of the load. For example, the damping element may be located near a contact point or connection point or surface between the load and the movable object. In some cases, the load may be embedded in or enclosed by one or more damping elements.

[0045] Figure 3 It is demonstrated that target tracking is supported in a movable object environment according to various embodiments of the present invention. Figure 3 As shown, target tracking system 300 includes movable object 310 and control terminal 311. System 300 can be used to follow, such as track one or more targets 306. Although movable object 310 is depicted as an unmanned aerial vehicle (UAV), this description is not intended to be limiting, and any suitable type of movable object as described herein can be used. Those skilled in the art will appreciate that any embodiments described herein in the context of an aircraft system can be applied to any suitable movable object.

[0046] In some embodiments, movable object 310 may include carrier 301 and payload 302. Carrier 301 may allow payload 302 to move relative to movable object 310. For example, carrier 301 (e.g., a pan / tilt) may allow payload 302 to rotate about one or more axes. Alternatively or in addition, carrier 301 may allow payload 302 to move linearly along one or more axes. The axes of rotational or translational movement may or may not be orthogonal to each other.

[0047] In some embodiments, payload 302 may be rigidly coupled or connected to movable object 310 such that payload 302 remains substantially stationary relative to movable object 310. For example, carrier 301 connecting movable object 310 and payload 302 may not allow payload 301 to move relative to movable object 310. Alternatively, payload 104 may be coupled directly to movable object 101 without the need for a carrier.

[0048] In some embodiments, payload 302 may include one or more sensors for surveying or tracking one or more targets 306. Examples of such payloads may include image capture devices or imaging devices (e.g., cameras or video cameras, infrared imaging devices, ultraviolet imaging devices, etc.), audio capture devices (e.g., parabolic microphones), infrared imaging devices, etc. Any suitable one or more sensors may be incorporated into payload 302 to capture any visual, audio, electromagnetic signals, or any other desired signals. The sensors may provide static sensing data (e.g., photos) or dynamic sensing data (e.g., videos). The sensors may continuously capture sensing data in real time or at a high frequency.

[0049] In various embodiments, the target 306 tracked by the movable object 101 may include any natural or man-made object or structure, such as a geographical landscape (e.g., a mountain, vegetation, valley, lake, or river), a building, a vehicle (e.g., an aircraft, a ship, a car, a truck, a bus, a van, or a motorcycle). The target 306 may also include a living subject, such as a person or an animal. The target 306 may be mobile or stationary relative to any suitable reference system. The reference system may be a relatively fixed reference system (e.g., the surrounding environment or the earth). Alternatively, the reference system may be a moving reference system (e.g., a moving vehicle). In various embodiments, the target 306 may include a passive target or an active target. An active target may be configured to transmit information related to the target to the movable object, such as the GPS position of the target. Information may be transmitted to the movable object by wireless communication from the communication unit of the active target to the communication unit of the movable object. Examples of active targets may include friendly vehicles, buildings, the military, and the like. Passive targets are not configured to transmit information related to the target. Examples of passive targets may include neutral or hostile vehicles, buildings, troops, and so forth.

[0050] In some embodiments, the control terminal 311 can be configured to provide control data, or data that can be used by the controller 304 on the movable object 310 to generate control data. The control data can be used to directly or indirectly control multiple aspects of the movable object 310. In some embodiments, the control data may include navigation commands for controlling navigation parameters of the movable object (such as the position, speed, orientation, or attitude of the movable object 310). For example, the control data can be used to control the flight of the UAV. The control data can affect the operation of one or more propulsion units that may affect the flight of the UAV. In other cases, the control data may include commands for controlling independent components of the movable object 310.

[0051] In some embodiments, the control data may include information for controlling the operation of the carrier 301. For example, the control data may be used to control an actuation mechanism of the carrier 301 to angularly move and / or linearly move the payload 302 relative to the movable object 310. As another example, the control data may be used to control the movement of the carrier 301 without the payload. As another example, the control data may be used to adjust one or more operating parameters of the payload 302, such as taking a still or dynamic photo, zooming in or out, turning on or off, switching an imaging mode, changing image resolution, zooming, changing depth of field, changing exposure time, changing lens speed, changing viewing angle or field of view, and the like. In other embodiments, the control data may be used to control a sensing system (not shown), a communication system (not shown), and the like of the movable object 310.

[0052] In some embodiments, the control data from the control terminal 311 may include target information, which may be used by the controller 304 on the movable object 310 to generate control data. In some cases, the target information may include characteristics of a particular target, such as an initial position (e.g., coordinates) and / or size of the target in one or more images captured by an imaging device carried by the movable object 310. Additionally or alternatively, the target information may include target type information, such as characteristics of the type or category of the target, including color, texture, pattern, size, shape, dimensions, and the like. The target information may include a data representation of an image of the target. This may include an image of a target within a field of view. The field of view may be defined or covered by an image captured by an imaging device.

[0053] The target information may also include expected target information. The expected target information indicates the characteristics of the tracked target that may appear in the image captured by the imaging device. The expected target information can be used to adjust the movable object, the carrier and / or the imaging device so that the tracked target maintains its appearance in one or more images according to the expected target information. For example, the target can be tracked so as to maintain the expected position and / or size in one or more images captured by the imaging device. For example, the expected position of the tracked target can be located near the center of the image or off-center. The expected size of the tracked target can be about a certain number of pixels. The expected target information may be the same or different from the initial target information. In different embodiments, the expected target information may be provided by or not by the control terminal. For example, the expected target information may be hard-coded in the control logic executed by the processing unit on the movable object, stored in the local and / or remote data storage of the movable object, or obtained from other suitable sources.

[0054] In some embodiments, target information (including specific target information and target type information) can be generated at least in part at a user input at the control terminal 311. In addition or alternatively, the target information can be generated based on data from other sources. For example, the target type information can be obtained based on previous images and / or data extracted from local or remote data storage. The image can be previously captured by an imaging device or other device connected to the movable object 310. The image can be computer generated. Such target type information can be selected by the user and / or automatically provided to the movable object by default.

[0055] The target information can be used by the movable object 310 to track one or more targets 306. Tracking and any other related data processing can be performed at least in part by one or more processors on the movable object 310. In some embodiments, the target information can be used by the movable object to identify the target 306 to be tracked. Such target recognition can be performed based on initial target information, which includes specific characteristics of a specific target (e.g., initial coordinates of the target within the image captured by the movable object), or general characteristics of a certain type of target (e.g., color and / or texture of one or more targets to be tracked). In some cases, target recognition can involve any suitable image recognition and / or matching algorithm. In some embodiments, target recognition includes comparing two or more images to determine, extract and / or match features contained therein.

[0056] Once the target is identified, the expected target information can be used to detect the expected characteristics relative to the target, such as deviations from the expected position and / or size. In some embodiments, the current target characteristics or information can be determined based on one or more images captured by the movable object. The current target information can be compared with the expected target information provided by the control terminal to determine the deviation relative to it. The change in the position of the target can be detected by comparing the coordinates of the target in the image (e.g., the coordinates of the center point of the target) with the coordinates of the expected target position. The change in the size of the target can be detected by comparing the size of the area covered by the target (e.g., in pixels) with the expected target size. In some embodiments, the change in size can be detected by detecting the orientation, boundary or other characteristics of the target.

[0057] Based at least in part on the detected deviation, a control signal may be generated (e.g., by one or more processors on the movable object) that causes an adjustment that substantially corrects the detected deviation. Thus, the adjustment may be used to substantially maintain one or more desired target characteristics (e.g., target position and / or size) within the image captured by the movable object. In some embodiments, the adjustment may be performed substantially in real time while the movable object is executing a user-provided navigation command (e.g., hovering or moving) and / or a predetermined navigation path. The adjustment may also be performed substantially in real time while the imaging device is capturing one or more images. In some embodiments, the adjustment may be generated based on other information, such as sensed data acquired by one or more sensors on the movable object (e.g., a proximity sensor or a GPS sensor). For example, the position information of the tracked target may be obtained by a proximity sensor and / or provided by the target itself (e.g., a GPS position). In addition to the detected deviation, such position information may also be used to generate the adjustment.

[0058] The adjustments may be with respect to the movable object, the carrier, and / or the payload (e.g., the imaging device). For example, the adjustments may cause the movable object and / or the payload (e.g., the imaging device) to change its position, attitude, orientation, angular velocity and / or linear velocity, angular velocity and / or linear velocity, etc. The adjustments may cause the carrier to operate to move the payload (e.g., the imaging device) relative to the movable object, such as around or along one, two, three or more axes. In addition, the adjustments may include adjustments to the zoom, focus, or other operating parameters of the payload (e.g., the imaging device) itself (e.g., zoom in / out).

[0059] In some embodiments, adjustments can be made based at least in part on the type of deviation detected. For example, a deviation from an expected target position may require the movable object and / or payload (e.g., via a carrier) to rotate about one, two, or three axes of rotation. As another example, a deviation from an expected target size may require the movable object to translate along an appropriate axis and / or change the zoom of the imaging device (e.g., zoom in or out).

[0060] In various embodiments, the adjustment to substantially correct the deviation from the expected target information may be achieved by controlling one or more controllable objects, such as a movable object, a carrier, an imaging device, or any combination thereof, via a control signal. In some embodiments, the controllable object may be selected to implement the adjustment, and the corresponding control signal may be generated based at least in part on the configuration or setting of the controllable object. For example, if the imaging device is rigidly coupled to the movable object and is therefore not allowed to move relative to the movable object, the adjustment involving rotation around two axes (e.g., yaw and pitch axes) may be achieved simply by the corresponding rotation of the movable object around the two axes. The same may occur when the imaging device is directly coupled to the movable object, or when the imaging device is coupled to the movable object via a carrier that does not allow relative movement between the imaging device and the movable object. If the carrier allows the imaging device to rotate around at least one axis relative to the movable object, the same dual-axis adjustment may be achieved by combining the adjustments made to both the movable object and the carrier. In this case, the carrier may be controlled to implement the rotation around one or both of the two axes required for the adjustment, and the movable object may be controlled to implement the rotation around one or both of the two axes. For example, the carrier may include a single-axis gimbal that allows the imaging device to rotate about one of the two axes required for adjustment, while rotation about the remaining axis is achieved by the movable object. Alternatively, if the carrier allows the imaging device to rotate about two or more axes relative to the movable object, the same two-axis adjustment can be achieved by the carrier alone. For example, the carrier may include a two-axis or three-axis gimbal.

[0061] As another example, adjustment for changes in the size of the correction target can be accomplished by controlling the zooming in / out of the imaging device (e.g., if the imaging device supports the required zoom level), by controlling the movement of the movable object (e.g., so as to get closer to or farther away from the target), or by a combination of zooming in / out of the imaging device and movement of the movable object. A processor on the movable object can make a decision as to which object or combination of objects to adjust. For example, if the imaging device does not support the zoom level required to maintain the required size of the target within the image, the movable object can be controlled to move instead of or in addition to adjusting the zoom of the imaging device.

[0062] In some embodiments, the adjustment can be implemented by considering other constraints. For example, in the case where the navigation path of the movable object is predetermined, the adjustment can be implemented by the carrier and / or the imaging device without affecting the movement of the movable object. For example, in the case where the remote user actively controls the navigation of the movable object through the control terminal, or in the case where the movable object is navigating according to a pre-stored navigation path (e.g., autonomously or semi-autonomously), the navigation path of the movable object can be predetermined.

[0063] Examples of other constraints may include maximum and / or minimum limits on the rotation angle, angular velocity and / or linear velocity, operating parameters, and the like of the movable object, carrier, and / or payload (e.g., imaging device). Such maximum and / or minimum thresholds may be used to limit the range of adjustment. For example, the angular velocity of the movable object and / or imaging device around a certain axis may be capped by the maximum angular velocity allowed for the movable object, carrier, and / or payload (e.g., imaging device). As another example, the linear velocity of the movable object and / or carrier may be capped by the maximum linear velocity allowed for the movable object, carrier, and / or payload (e.g., imaging device). As yet another example, the adjustment of the focal length of the imaging device may be limited by the maximum and / or minimum focal length of a particular imaging device. In some embodiments, such limits may be predetermined and depend on the specific configuration of the movable object, carrier, and / or payload (e.g., imaging device). In some cases, such configurations may be configurable (e.g., configurable by a manufacturer, an administrator, or a user).

[0064] In some embodiments, the movable object 310 may be configured to provide data and the control terminal 311 may be configured to receive the data, such as sensed data acquired by the sensor 303 on the movable object 310, and tracking data or information indicating characteristics of one or more targets tracked by the movable object 310. Examples of sensed data may include image data acquired by an imaging device carried by the movable object 310 or other data acquired by other sensors. For example, real-time or near real-time video may be streamed from the movable object 310 and / or the payload 302 (e.g., the imaging device) to the control terminal 311. The sensed data may also include data acquired by a global positioning system (GPS) sensor, a motion sensor, an inertial sensor, a proximity sensor, or other sensors. Examples of tracking information may include relative or absolute coordinates and / or size of a target within one or more image frames received from the movable object, changes in a target between consecutive image frames, GPS coordinates or other location information of a target, and the like. In some embodiments, the tracking information may be used by the control terminal 311 to display the target being tracked (e.g., by a graphical tracking indicator, such as a wireframe around the target). In various embodiments, the data received by the control terminal may include raw data (eg, raw sensed data acquired by a sensor) and / or processed data (eg, tracking information processed by one or more processors on the movable object).

[0065] In some embodiments, the control terminal 311 may be located at a location remote or away from the movable object 310, the carrier 301, and / or the payload 302. The control terminal 311 may be disposed on or attached to a support platform. Alternatively, the control terminal 311 may be a handheld or wearable device. For example, the control terminal 311 may include a smartphone, a tablet computer, a laptop computer, a computer, glasses, gloves, a helmet, a microphone, or a suitable combination thereof. Moreover, the control terminal 311 may support the operation of one or more applications 305 (such as a mobile APP), which may include image viewing 315.

[0066] The control terminal 311 can be configured to display data received from the movable object 310 through a display. The displayed data may include sensory data, such as images (e.g., still images and videos) acquired by an imaging device carried by the movable object 310. The displayed data may also include tracking information displayed separately from the image data or superimposed on the image data. For example, the display can be configured to display an image in which a target is indicated or highlighted by a tracking indicator (such as a wireframe, circle, or any other geometric shape surrounding the target being tracked). In some embodiments, the image and tracking indicators are displayed substantially in real time when the image data and tracking information are received from the movable object and / or when the image data is acquired. In other embodiments, the display may be provided after a certain delay.

[0067] The control terminal 311 may be configured to receive user input through an input device. The input device may include a joystick, a keyboard, a mouse, a stylus, a microphone, an image or motion sensor, an inertial sensor, and the like. Any suitable user input may be used to interact with the terminal, such as manually typed commands, voice control, gesture control, or position control (e.g., by movement, positioning, or tilting of the terminal). For example, the control terminal 311 may be configured to allow a user to change the orientation or posture of the control terminal by manipulating a joystick, to interact with a graphical user interface using a keyboard, mouse, finger, or stylus, or to control the state of a movable object, carrier, payload, or any component thereof by using any other suitable method.

[0068] The control terminal 311 may also be configured to allow the user to enter target information using any suitable method. In some embodiments, the control terminal 311 may be configured to enable the user to directly select a target from one or more images (e.g., videos or snapshots) being displayed. For example, the user may select a target by directly touching the screen with a finger or stylus or selecting using a mouse or joystick. The user may draw around the target, touch the target in the image, or otherwise select the target. Computer vision techniques or other techniques may be used to determine the boundaries of the target. In other aspects, user input may define the boundaries of the target. One or more targets may be selected at a time. In some embodiments, the selected target is displayed with a selection indicator that indicates that the user has selected the target to be tracked. In some other embodiments, the control terminal may be configured to allow the user to enter or select target type information, such as color, texture, shape, size, or other characteristics associated with the desired target. For example, the user may enter target type information, select such information using a graphical user interface, or use any other suitable method. In some other embodiments, the target information may be obtained from a source other than the user, such as a remote or local data storage, other computing devices operatively connected to the control terminal or otherwise communicating with it, and the like.

[0069] In some embodiments, the control terminal allows the user to select between a manual tracking mode and an automatic tracking mode. When the manual tracking mode is selected, the user can specify a specific target to be tracked. For example, the user can manually select a target from an image being displayed by the control terminal. The specific target information (e.g., coordinates and / or size) associated with the selected target is then provided to the movable object as the initial target information of the target. On the other hand, when the automatic tracking mode is selected, the user does not specify a specific target to be tracked. Instead, the user can specify descriptive information related to the type of target to be tracked, for example, through a user interface provided by the control terminal. The movable object can then use the initial target information or target type information of the specific target to automatically identify the target to be tracked and then track the identified target.

[0070] Here, providing specific target information (e.g., initial target information) requires more control of tracking of the target by the user and less automated processing or calculation (e.g., image or target recognition) by the processing system on the movable object. On the other hand, providing target type information requires less control of the tracking process by the user, but more calculations are performed by the onboard processing system. The control of the tracking process appropriately assigned between the user and the onboard processing system can be adjusted according to various factors, such as the environment of the movable object, the speed or altitude of the movable object, user preferences, and the computing resources (e.g., CPU or memory) available on and / or outside the movable object. For example, when the movable object is navigating in a relatively complex environment (e.g., with numerous buildings or obstacles or indoors), relatively more control can be assigned to the user compared to when the movable object is navigating in a relatively simple environment (e.g., wide open space or outdoors). As another example, when the movable object is at a lower altitude, more control can be assigned to the user compared to when the movable object is at a higher altitude. As yet another example, if the movable object is equipped with a high-speed processor adapted to perform complex calculations relatively quickly, more control may be delegated to the movable object.In some embodiments, the allocation of control over the tracking process between the user and the movable object may be dynamically adjusted based on the factors described herein.

[0071] User input may be used at least in part to generate control data such as described herein. Control data may be generated by a control terminal, a movable object, a third device, or any combination thereof. For example, a user's manipulation of a joystick or a control terminal or interaction with a graphical user interface may be converted into a predetermined control command for changing the state or parameters of a movable object, a carrier, or a payload. As another example, a user's selection of a target within an image being displayed by a control terminal may be used to generate initial and / or expected target information for tracking purposes, such as the initial and / or expected position and / or size of a target. Alternatively or in addition, control data may be generated based on information obtained from non-user sources, such as remote or local data storage, other computing devices operatively connected to a control terminal, and the like.

[0072] Figure 4 An exemplary target tracking system in a movable object environment according to various embodiments of the present invention is shown. Figure 4 As shown, movable object 400 includes a controller 410 that can receive different types of information, such as image information, from a camera 402 carried by a carrier 401 , as well as other sensors 403 onboard.

[0073] The controller 410 may perform an initialization operation 411 based on the image information received from the camera 402. Optionally, the controller 410 may use a dedicated or general detector 413 to detect a target (ie, an object to be tracked) in the initialization image. Thus, the controller 410 may acquire the target and set up a corresponding tracking strategy.

[0074] In addition, the movable object 400 can use the tracker 412 to track the target. In the event that the tracker 412 loses the target, the system can use the detector 413 to re-detect the lost target in order to continue the tracking process. In addition, the controller 410 can perform additional operations such as position estimation 414 and navigation operations 415 based on the information received from the sensor 403.

[0075] Figure 5 Support for target tracking in a movable object environment 500 is shown according to various embodiments of the present invention.

[0076] At step 501, the movable object may begin tracking a target. At step 502, the movable object may capture one or more images, such as video, in real time. For example, the movable object may utilize a camera carried by a gimbal or other image sensor on the movable object (e.g., a UAV).

[0077] At step 503, the movable object may perform an initialization operation to acquire a target. For example, the movable object may acquire a target from an initialization image and obtain a feature model of the target. In addition, the movable object may continue to perform the initialization operation until the target has been successfully acquired.

[0078] At step 504, the movable object may perform a tracking process. For example, the movable object may employ a vision-based tracker to track a target.

[0079] At step 505, the system may check whether a tracking failure occurs. At step 506, when a tracking failure occurs, the system may perform a target detection operation. Then at step 507, the system may check whether a target is detected. The system may repeatedly perform the target detection operation until the target is detected again.

[0080] In another aspect, at step 508, the system can estimate the target position to continue tracking the target as long as tracking is successful (i.e., including the case where the target is re-detected). At step 509, with the estimated target position, the system can perform mobile control operations, such as flight control of the UAV, which allows the movable object to capture images for continuous tracking.

[0081] Figure 66 shows initializing target tracking in a movable object environment 600 according to various embodiments of the present invention. Figure 6 As shown, movable object 603 can capture image 601, which can be transmitted to application 602 (eg, located on a ground terminal). Application 602 can use image view 604 to display image 601, which includes one or more features, such as objects AC 611-613.

[0082] According to different embodiments of the present invention, the movable object 603 can use different types of object detectors to detect the target (i.e., the object desired by the user), for example, after receiving user input, such as a point on the target and / or a target category to be tracked (e.g., a human).

[0083] For example, the movable object 603 may use an object proposal method in the initialization phase. Figure 6 As shown, the system allows a user to select a point 605 on an object of interest, such as object B 612 in image 601. Once the selected point 605 is received from the user, the system may use a bounding box 606 to define and propose the object 612, which may have an irregular shape.

[0084] In addition, the system may propose multiple object candidates, for example using different bounding boxes. In this case, the user is allowed to make a decision on which object candidate (i.e., bounding box) is desired. The system may then generate a feature model 610 based on the selected object proposal. Figure 6 As shown, feature model 610 may represent image characteristics of the patch of pixels within bounding box 606. After receiving a selection of a desired object candidate (or bounding box) from the ground station, movable object 603 may begin tracking the target based on feature model 610 of the target.

[0085] According to different embodiments of the present invention, feature model 610 can be constructed based on checking common objects, such as human body, car and human face. In this case, feature model 704 can include different discrete objects trained offline. Alternatively, feature model 704 can be constructed based on the characteristics of the analyzed object, such as edge / contour and color information. For example, feature model 610 can be generated using different methods, such as optical flow and / or correlation filtering algorithms. Moreover, feature model 610 can be represented in spatial domain and / or frequency domain.

[0086] Using this initialization process, the data to be transmitted from the ground station (i.e., application 602) to the movable object 603 (e.g., UAV) is limited because only the positions of the selected points 605 may be needed to initialize the tracking process. Therefore, the transmission delay in the initialization step can be minimized. On the other hand, the movable object can continuously transmit image or video data to the application 602 for user interaction because the data link from the movable object 603 (e.g., UAV) to the ground station (i.e., application 602) tends to have a wider bandwidth and higher speed than the data link from the movable object 603 to the ground station (i.e., application 602).

[0087] Figure 7 Tracking a target in a movable object environment 700 according to various embodiments of the present invention is shown. Figure 7 As shown, the movable object 710 may include a carrier 701 carrying an imaging device 702, such as a camera. The imaging device 702 may capture an image 703 of a target 706. In addition, the movable object 710 may include a controller 705, which may maintain a feature model 704 for tracking the target 706 and generate control signals for controlling the movable object 710.

[0088] According to various embodiments of the present invention, the system may track a target by following features that represent the target 706. For example, the system may determine the similarity between different features, such as features AC 711-713 in the image 703 and the feature model 704. The similarity may be calculated as a result value (or score) of a function of each feature in the image 703. Based on the calculated scores, the system may determine which feature represents the target 706. Alternatively, the system may directly compare each feature in the image 703 with the feature model to determine whether the feature represents the target 706.

[0089] The system can periodically determine whether the similarity between the feature and the feature model is maintained during the tracking process, for example by checking whether the result value (or score) of the function remains above a previously determined threshold. The system can consider the target lost when the value is below a previously determined threshold. The system can then check each subsequent image and look for the target. The check can be based on the original feature model or the last updated feature model, and can be performed across different scales and locations in each subsequent image.

[0090] Therefore, the system can maintain tracking accuracy, which is beneficial for long-term target tracking, because small errors may accumulate and make the entire tracking system unstable. Moreover, the system can perform failure detection and target re-detection, which is also beneficial for long-term target tracking in terms of robustness and practicality. For example, once the target reappears after being occluded for a while, the system can maintain tracking of the target.

[0091] Figure 8 The perspective transformation relationship between two images captured for tracking a target in a movable object environment according to various embodiments of the present invention is demonstrated.

[0092] In such Figure 8 In the example shown, a movable object 801 in a movable object environment 800 includes a carrier 802 carrying an imaging device such as a camera 803. The movable object 801 can track a target 804 while the target 804 is moving (or remains stationary).

[0093] For illustration purposes only, at time T1, camera 803 may capture image 805 of object 804 in an image plane having coordinates xy. Next, at time T2, camera 803 may capture image 815 of object 804 (which may or may not be at the same location) in an image plane having coordinates x'-y'.

[0094] like Figure 8 As shown, there may be a perspective transformation 820 relationship between the image coordinates xy in the image 805 and the image coordinates x′−y′ in the image 815 . The perspective transformation 820 is determined by the change in the relative position between the camera 893 and the target 804 .

[0095] Fig. 9 The use of inverse perspective transforms to support tracking in a movable object environment 900 is shown according to various embodiments of the present invention. Fig. 9 As shown, there may be a perspective transformation 910 between image 901, which may be an initial image, and image 902 captured at a later time. Perspective projection 910 may change the geometry of target 911 as it appears in image 902, which may increase the difficulty of applying various image-based tracking algorithms.

[0096] According to various embodiments of the present invention, a stabilization mechanism such as a gimbal can stabilize the imaging process when the UAV moves. Such a stabilization mechanism can mitigate the impact of camera movement on tracking performance. However, especially for objects appearing at the edge of the image, vision-based tracking algorithms, such as the Histogram of Directed Gradients (HOG) method based on gradient features, may still require complex models to handle changes in the top view.

[0097] According to various embodiments of the present invention, the system may compensate for the perspective transformation between the image 901 and the image 902 captured for tracking the target. For example, the system may apply an inverse perspective transformation 920 to the captured image 902 to generate a compensated image 904. The system may obtain the inverse perspective transformation 920 based on the perspective transformation 910.

[0098] Next, the system can perform tracking of the target by applying the feature model to the compensated image 904. Here, the tracking information (such as a bounding box) obtained from the compensated image 904 can be converted back to the captured image 902, which is displayed to the user in the image view on the control terminal. Here, the conversion can be performed using a perspective transform 910.

[0099] Alternatively, the system may apply an inverse perspective transform 920 to the feature model before applying the compensated feature model to the captured image 902 .

[0100] like Fig. 9 As shown, image correction can be achieved by applying a perspective transformation that takes into account the change in the camera's pitch angle. Given the pitch angle and the camera intrinsic matrix, the perspective matrix can be calculated as

[0101]

[0102] where R is the rotation matrix,

[0103]

[0104] and K is the intrinsic matrix,

[0105]

[0106] Fig.10 FIG. 2 shows a flow chart of tracking a target in a movable object environment according to different embodiments of the present invention. Fig.10 As shown, at step 1001, the system may obtain a feature model of a target, wherein the feature model represents image characteristics of the target. Furthermore, at step 1002, the system may extract one or more features from one or more images captured by an imaging device carried by the movable object. Furthermore, at step 1003, the system may apply the feature model to the one or more features to determine similarity.

[0107] Fig.11 It is shown that target tracking and re-detection are supported in a movable object environment 1100 according to various embodiments of the present invention. Fig.11 As shown, the system can use a vision-based tracker to perform target tracking based on the captured image 1101.

[0108] According to different embodiments of the present invention, the vision-based tracker can utilize different tracking algorithms, such as optical flow algorithms and / or correlation filtering algorithms. In addition, the system can automatically track and detect the target over a longer period of time. In addition, tracking can be performed by utilizing a search window, which is a local search range within which the tracking algorithm can find the optimal position / scale of the target, i.e., tracking can be performed locally rather than globally for the entire image.

[0109] like Fig.11 As shown, a vision-based tracker can generate tracking output 1103 when tracking a target by applying a feature model 1102 to an image 1101, the image 1101 including different features, such as features AC 1111-1113. For example, the tracker can follow feature B with a bounding box 1106 in a search window 1110. In addition, based on the tracking output 1103, the system can perform failure detection 1104 and target re-detection 1105 operations.

[0110] Tracking output 1103 may be a feature response map generated for a search window 1110 in image 1101. Each point in the feature response map may represent the correlation or similarity between one or more features extracted from a block of pixels (i.e., a sliding window) in search window 1110 and a feature model 1102 of the target.

[0111] For example, the system can move the sliding window around the search window 1110 to obtain a complete feature response map in the spatial domain. Alternatively, the system can obtain a feature response map in the frequency domain, for example, using a correlation filtering method, without actually moving the sliding window over the search window 1110 in the image 1101.

[0112] For example, the system can combine target detection with a correlation framework by utilizing both a correlation filter-based tracking algorithm and a fast object proposal algorithm. The correlation-based object tracking algorithm is both fast and efficient, which is beneficial for movable objects, such as unmanned aerial vehicles (UAVs), because movable objects usually have limited computing power and power resources. In addition, the system can re-detect the target once it is lost. Moreover, the system can use a single camera to calculate the position of the tracked object in flight in the presence of continuity and stability, such as using a correlation filter-based tracking algorithm to estimate the object size.

[0113] Using the correlation framework, object tracking, which can be performed in the spatial domain and / or the frequency domain, can involve multiple steps, such as a translation estimation step and a scale estimation step. Both steps can include extracting features from image patches, calculating the maximum feature response, and updating the feature model.

[0114] For example, the system can perform translation estimation of the target in the frequency domain based on a rectangular image patch x with M×N pixels, from which a d-dimensional feature map can be extracted. To calculate the maximum feature response, the goal is to find the optimal correlation filter h by minimizing the following cost function:

[0115]

[0116] The symbol represents a circular dependency, is the total number of feature dimensions, yes of Feature dimensions, is the desired correlation output, which can be a Gaussian function, and is the regularization parameter, where .

[0117] In addition, the feature model can be updated based on the solution of the above optimization problem. For example, using Fast Fourier Transform (FFT), the solution of the above optimization problem is:

[0118]

[0119] Among them , represents the complex conjugate, and , is the element-wise product.

[0120] In addition, the system can perform scale estimation by constructing a target feature pyramid in the estimated translation position to facilitate handling of object scale changes. Then, the system can find the optimal object scale that maximizes the scale response. For example, for each scale, , where N is the number of scales, the system can extract Image blocks , the image block is located at the center of the estimated translation position. Moreover, for each block , the system can obtain the image block Extract Dimensional feature map and will Conjunct into a joint structure as a single dimension. Then, the system can calculate the scale response and update the scale model based on the joint structure.

[0121] Additionally, to perform a failure detection 1104 operation, the system can quantify the correlation in the feature response map by a metric that indicates when the vision-based tracker has failed to track the target. For example, the metric can be based on the peak-to-sidelobe ratio (PSR), which can quantify the correlation response map. The tracker can check the translation response in each frame. When the PSR value is below a threshold for multiple images (e.g., 10 frames in a row), the tracker can detect the translation response. , the tracker can stop updating the feature model and can trigger the detector to detect the object. For example, the tracker can sent to the detector.

[0122] Next, to perform the target re-detection 1105 operation, the detector may perform a task of detecting (or re-detecting) the target, which may include steps such as generating object proposals for the current image frame, filtering object proposals, obtaining the best object candidate, and calculating the confidence of the best object candidate.

[0123] For example, the system can obtain a set of object proposal candidates for a target in an image frame by using different object proposal algorithms, which allows avoiding searching the entire image for object candidates. Here, the object proposal algorithm can return a large number of object candidates from a single image. Each object candidate can be defined as a rectangular bounding box containing an "object" and can be associated with a corresponding object confidence (e.g., a score). For example, an object proposal can be defined as , where x, y represent the top left point position of the candidate in the image, and w, h represent the bounding box width and height, respectively. In addition, the object candidate proposals can be ranked in descending (or ascending) order based on the scores.

[0124] Next, the system can remove unlikely candidates from the set of object proposal candidates. For example, the system can utilize a pre-filtering method to speed up the removal process and avoid mismatches, because the object proposal algorithm can return a large number of candidates, including candidates that are unlikely to be the desired object. For example, the system can select only the top few (e.g., 10) candidates after applying the pre-filtering method.

[0125] According to different embodiments of the present invention, different types of filters can be used in the pre-filtering method. For example, the system can first compare the aspect ratio of the candidate with the aspect ratio of the initial object bounding box. If the aspect ratio is outside a certain range, the candidate can be filtered out. Moreover, the system can use color attributes to rank the remaining candidates. For example, the system can calculate the color attributes in the current image frame. For each candidate, the system can cut off the corresponding block and calculate the color histogram of the block. Then, the system can rank the candidates in descending order by histogram distance and select the front candidate. Here, the histogram distance can be defined as the L2 norm distance between the original block and each candidate.

[0126] In addition, the system can obtain the best object proposal, for example, by computing correlations in the spatial domain. For each selected proposal candidate, given the translation model in the frequency domain When the system can Apply inverse FFT to obtain the spatial domain translation model and stack them as a dimension in the model. The resulting model in the spatial domain (which can be called the kernel h) can be defined as

[0127]

[0128] Similarly, for the i-th proposal among the top candidates, the system can cut out the corresponding block from the current image frame , from the image block The feature map is extracted from , and then stacked as a dimension in the feature map. The resulting feature map can be expressed as .

[0129] Next, the spatial correlation response is h and The element-by-element dot product between

[0130]

[0131] in is an element-by-element dot product operation. Here, the system can select the proposal at the center of the proposal as the best proposal. Therefore, the system can speed up the process of obtaining the best proposal because it does not need to search all proposals.

[0132] In addition, the system can calculate the confidence of the selected best proposal to determine whether the best candidate proposal is the "real" target. For example, the system can cut an image block from the current image frame for the best proposal, extract features from the image block, generate a feature response map, and calculate the PSR of the feature response map. If the resulting PSR exceeds a threshold , then the system can determine that a “real” target is detected and the detector can stop further searching. In this case, the best proposal can be passed to the tracker to continuously update the feature model.

[0133] Fig.12 The present invention demonstrates the use of a positioning device to assist in target tracking in a movable object environment according to various embodiments of the present invention. Fig.12 As shown, a movable object 1210 in a movable object environment 1200 may include a carrier 1201 carrying an imaging device such as a camera 1202 .

[0134] Additionally, movable object 1210 may include a positioning device, such as GPS device 1205. Also, target 906 may carry a positioning device, such as GPS device 1215. For example, target 1206 may be a person carrying a mobile device with GPS capabilities, such as a watch, a band, a hat, and / or a pair of shoes.

[0135] In real time, the movable object 1210 (e.g., the controller 1203) can obtain the relative distance and orientation of the target 1206 in order to maintain the target within the proximity 1208 (e.g., a predetermined circular range) of the target. In addition, when the system determines that the target is lost, the system can rely on the positioning device to maintain the target 1206 within the proximity 1208 of the target.

[0136] In addition, the system (eg, controller 1203) can use a vision-based tracking system 1204, which can be used in indoor scenarios when GPS signals are not available. Thus, the system can utilize both vision-based tracking technology and a positioning device for performing long-term tracking tasks.

[0137] Fig.13 FIG. 2 shows a flow chart of supporting target tracking and re-detection in a movable object environment according to different embodiments of the present invention. Fig.13 As shown, at step 1301, the system may apply a feature model of the target to an image captured for tracking the target to generate a tracking output. Furthermore, at step 1302, the system may determine that a tracking failure has occurred based on the tracking output. Furthermore, at step 1303, the system may examine one or more subsequent images to detect the target.

[0138] Fig.14 Tracking a target based on distance measurements in a movable object environment according to various embodiments of the present invention is demonstrated. Fig.14 As shown, movable object 1410 in movable object environment 1400 may include a carrier 1401 carrying an imaging device such as camera 1402 .

[0139] The movable object 1401 may acquire a target 1406 from an image 1403 captured by the camera 1402. For example, the image 1403 may include a plurality of features, such as objects AC 1411-1413, and a user may select the object 1411 as a target to follow (or track).

[0140] According to various embodiments of the invention, the system can obtain the (3D) position of the target from the (2D) tracking information. For example, the position of the target 1406 (which is important for tracking the target) can be determined based on the direction toward the target 1406 and the relative distance 1415 between the movable object 1410 (e.g., UAV) and the target 1406. The direction of the object can be obtained by calculating the direction vector from the calibrated camera. In addition, the system can estimate the relative distance between the tracking device and the target based on the state information associated with the imaging device and the movable object.

[0141] The state information includes altitude information of the imaging device carried by the movable object. The altitude information of the imaging device may be received from a control module for the movable object. The state information may include attitude information of the imaging device carried by the movable object. The attitude information of the imaging device may be received from a payload augmentation control module, wherein the payload augmentation control module controls an augmentation system that stabilizes the imaging device on the movable object.

[0142] According to various embodiments of the present invention, the controller 1405 may obtain a relative distance 1415 between the movable object 1410 and the target 1406 from the image 1403. Furthermore, the controller 1405 may generate one or more flight control signals 1404 to guide the movable object 1410 to track the target 1406. The control signal 1404 may include an acceleration / deceleration signal and a gimbal attitude adjustment signal. For example, when the movable object 1410 is tracking the target 1406, the controller 1405 may adjust the movable object or the gimbal to rotate it around the yaw direction based on the distance between the target and the center point of the image.

[0143] Thus, the controller 1405 can maintain a desired tracking distance (which can be a constant distance or a dynamically configured distance) from the target 1406. For example, the system can calculate the velocity v_target of the target relative to the movable object 1410 based on the relative distance of the target from the movable object 1410 at different points in time. Then, the system can determine the required movement change of the movable object 1410 based on the velocity v_uav of the movable object 1410 and the current relative distance 1415.

[0144] Fig.15 The present invention shows how to measure the relative target distance in a movable object environment according to various embodiments of the present invention. Fig.15 As shown, movable object 1501 in movable object environment 1500 may include carrier 1502 carrying an imaging device such as camera 1503. Camera 1503 may capture an image of target 1504 on image plane 1513 having image coordinates 1515.

[0145] According to different embodiments of the present invention, the system can utilize different distance measurement methods. For example, the distance measurement method can utilize bounding box 1520, which can handle complex geometric shapes and provide continuous and stable results.

[0146] like Fig.15 As shown, the imaging of the target can be represented based on a pinhole imaging model, which assumes that light from an object point in three-dimensional space can be projected onto an image plane to form an image point. The optical axis 1508 can pass through the reflector center 1507 and the image center. The distance between the reflector center and the image center can be equal to or substantially similar to the focal length 1509. For illustration purposes only, the image plane 1513 can be moved to a reflector position on the optical axis 1008 between the reflector center 1507 and the target 1504.

[0147] According to various embodiments of the invention, the system may perform an initialization step that includes (automatically) estimating both the target distance and the target height. Here, the system may determine the relative distance 1510 between the reflector center 1507 and the target 1504 projected on the ground based on a geometric relationship. Next, the system may determine the target height.

[0148] During the initialization phase (i.e., when t=0), the system may assume that the altitude of the movable object 1501 (i.e., camera 1502) is measured at the same level (or horizon) as the target 1504. Without limitation, when the level is not horizontal, the system may use an effective altitude that takes into account the altitude difference instead of the actually measured altitude of the movable object 1501 for measuring target distance and target height.

[0149] In such Fig.15 In the example shown, target 1504 may have the highest target point in world coordinates 1505. and the lowest target point , the points are projected on the image plane 1513 as the highest image points in the target image 1514 and the lowest image point The top line passes through the reflector center 1507, the highest image point and the highest target point and may have an inclination angle 1512 relative to the axis Z of the world coordinate 1505. Also, the bottom line passes through the reflector center 1507, the lowest image point and the lowest target point and may have an inclination angle 1512 relative to the axis Z.

[0150] Therefore, the target top and bottom direction vectors and It can be expressed as follows.

[0151]

[0152] in represents the intrinsic matrix of the camera, and Indicates camera rotation.

[0153] Next, the system can estimate the target distance based on the measured or effective camera altitude 1506 and the position of the bounding box 1520 in image coordinates 1515. For example, the distance 1510 can be calculated as , and the target height 1516 can be calculated as ,in yes The projected length on the ground, and yes The projected lengths on the ground are defined as follows.

[0154]

[0155] After the initialization step, the system can estimate the distance 1510 to the target 1504 even in the case where the target's altitude changes (e.g., when the target 1504 leaves the ground) and the altitude of the movable object 1501 (e.g., UAV) is unknown. This is beneficial because during tracking, the object may climb or descend and the altitude of the UAV may be unreliable when the UAV is flying over grass or climbing (e.g., 5 meters above the ground).

[0156] like Fig.15 As shown, after initialization, the relative distance 1510 between the target 1501 and the movable object 1501 projected on the ground can be calculated as , where dh represents the estimated height of the target at a unit distance away from the camera, and the estimated height can be calculated using the following formula.

[0157]

[0158] Thus, the method may be efficient and may have very few limitations once initialized because the system may estimate the distance 1510 to an object based on the height 1516 of the target after initialization.

[0159] In addition, the system can perform movement control, such as flight control, based on the estimated distance 1510. According to various embodiments of the present invention, for a movable object 1501 (such as a UAV), the goal of movement control is to maintain the movable object 1501 within a desired distance from a target 1504 in both the X and Y directions of the world coordinate 1505.

[0160] First, the system can smooth the estimated distance 1510 over a period of time, for example by applying a Kalman filter. Next, the system can calculate the velocity of the target 1504 relative to the movable object 1501 (eg, UAV) in order to control the tracking of the target 1504 by the movable object 1501.

[0161] According to various embodiments of the present invention, the system may utilize a proportional integral derivative controller (PID controller) to maintain the distance between the movable object 1501 and the target 1504 within a desired distance by utilizing the relative distance between the movable object 1501 and the target 1504 and their corresponding speeds. For example, the system may obtain the speed of the movable object 1501 relative to the ground based on the state information. Furthermore, the system may obtain the speed of the target 1504 relative to the ground based on the method described above.

[0162] Alternatively, the system can control the yaw angle of the movable object 1501 and the X direction movement in the world coordinate 1505. For example, the yaw angle θ of the movable object 1501 can be based on the following formula: and to calculate.

[0163]

[0164] in and is the displacement of the movable object 1501 in the X and Y directions in the world coordinates 1505. Then, the system can control the movement of the movable object 1501 based on the measured distance between the movable object 1501 and the target 1504. In addition, the system can implement side-by-side tracking, which allows the movable object 1501 to track the target 1504 in a side-by-side manner, thereby achieving ideal picture capture.

[0165] According to different embodiments of the present invention, by further considering the status information of the movable object 1501 and the camera 1507, the above method can be modified to cope with the tracking process when the movable object 1501 is highly unstable and / or the roll angle of the camera is different (for example, zero degrees).

[0166] Fig.16 FIG. 2 shows a flow chart of tracking a target based on distance measurement in a movable object environment according to different embodiments of the present invention. Fig.16 As shown, at step 1601, the system can acquire a target from one or more images captured by an imaging device carried by a movable object. Then, at step 1602, the system can obtain a relative distance between the movable object and the target. In addition, at step 1603, the system can generate one or more control signals to track the target.

[0167] Many features of the present invention may be implemented in, using or by means of hardware, software, firmware, or a combination thereof. Thus, features of the present invention may be implemented using a processing system (e.g., including one or more processors). Exemplary processors may include, but are not limited to, one or more general purpose microprocessors (e.g., single-core or multi-core processors), application specific integrated circuits, application specific instruction set processors, graphics processing units, physical processing units, digital signal processing units, coprocessors, network processing units, audio processing units, encryption processing units, and the like.

[0168] Features of the invention may be implemented in, using or by means of a computer program product, which is one or more storage media or computer-readable media having instructions stored thereon or therein, which instructions may be used to program a processing system to perform any of the features presented herein. Storage media may include, but are not limited to, any type of disk (including floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks), ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0169] The features of the present invention stored on any one or more machine-readable media may be incorporated into software and / or firmware for controlling the hardware of a processing system and enabling the processing system to interact with other mechanisms that utilize the results of the present invention. Such software or firmware may include, but is not limited to: application code, device drivers, operating systems, and execution environments / containers.

[0170] Features of the present invention may also be implemented in hardware using, for example, hardware components such as Application Specific Integrated Circuits (ASICs) and Field Programmable Gate Arrays (FPGAs). Implementation of a hardware state machine for performing the functions described herein will be apparent to those skilled in the relevant art.

[0171] In addition, the present invention can be conveniently implemented using one or more conventional general or special digital computers, computing devices, machines or microprocessors (including one or more processors, memories and / or computer-readable storage media programmed according to the teachings of the present disclosure). An experienced programmer can easily prepare appropriate software coding based on the teachings of the present disclosure, which will be obvious to those skilled in the software field.

[0172] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example rather than limitation. It will be apparent to those skilled in the relevant art that various changes in form and detail may be made herein without departing from the spirit and scope of the present invention.

[0173] The present invention has been described above by means of functional building blocks that demonstrate the performance of specified functions and their relationships. For ease of description, the boundaries of these functional building blocks are generally arbitrarily defined herein. Alternative boundaries may be defined as long as the specified functions and their relationships can be properly performed. Therefore, any such alternative boundaries are within the scope and spirit of the present invention.

[0174] The foregoing description of the present invention is provided for the purpose of illustration and description. The description is not intended to be exhaustive or to limit the present invention to the precise form disclosed. The breadth and scope of the present invention should not be limited by any of the exemplary embodiments described above. Many modifications and variations will be apparent to those skilled in the art. The modifications and variations include any relevant combination of the disclosed features. The embodiments are selected and described in order to best explain the principles of the present invention and its practical application, thereby enabling other technical personnel in the art to understand the present invention in different embodiments and in different modifications suitable for the intended specific use. The scope of the present invention is intended to be limited by the following claims and their equivalents.

Claims

1. A method for controlling a movable object to track a target, the method comprising: determining, in one or more images captured by the movable object, a target to be tracked by the movable object; controlling the movable object to track a target object using the target; as well as During tracking, if the target is lost in one or more of the images, using one or more positioning devices to maintain the movable object within proximity of the target; Wherein, during the tracking process, if the target is lost in one or more of the images, one or more positioning devices are used to maintain the movable object within the proximity of the target, including: Based on the position information obtained by the positioning device on the movable object and the orientation deviation of the position information provided by the positioning device carried by the target, the movable object and the imaging device carried by the movable object are controlled to rotate around the same rotation axis to adjust the shooting angle of the imaging device so that the movable object is maintained within the proximity of the target.

2. The method according to claim 1, wherein controlling the movable object and the imaging device carried by the movable object to rotate around the same rotation axis comprises: When the movable object moves along a predetermined navigation path, the movable object and an imaging device carried by the movable object are controlled to rotate around the same rotation axis without changing the navigation path of the movable object.

3. The method of claim 1, further comprising: Obtaining a feature model of an object, wherein the feature model represents image characteristics of the object; extracting one or more features from one or more images captured by an imaging device carried by the movable object; and The feature model is applied to the one or more features to determine a similarity.

4. The method of claim 3, further comprising: The target is acquired based on an initialization image captured by an imaging device carried by the movable object.

5. The method of claim 4, further comprising: receiving a selected point in the initialization image from a user; providing a plurality of object proposal candidates to the user, wherein each of the object proposals is represented using a bounding box; as well as An object proposal of a selection of the target is received from the user.

6. The method of claim 3, wherein: The one or more features are extracted from a search window in the one or more images.

7. The method of claim 3, further comprising: calculating a correlation between the feature model and each feature extracted from the one or more images, and / or The one or more features are matched to the feature model.

8. The method of claim 3, further comprising: A projective transformation of the target in the one or more images is obtained based on the state information of the imaging device.

9. The method of claim 3, wherein: The state information of the imaging device is determined based on the position and attitude information of the movable object and the attitude information of the imaging device.

10. The method of claim 9, wherein: The position and attitude information of the movable object is received from a control module for the movable object, and the attitude information of the imaging device is received from a payload stabilization control module, wherein the payload stabilization control module controls a stabilization system that stabilizes the imaging device on the movable object.

11. The method of claim 8, further comprising: A perspective transformation is applied to the feature model to compensate for the projective transformation of the object in the one or more images.

12. The method of claim 3, further comprising: The feature model is updated based on the matching of the one or more features with the feature model, wherein the updated feature model is suitable for application to subsequent images.

13. The method of claim 1, further comprising: applying a feature model of the target to an image captured for tracking the target to generate a tracking output; determining, based on the tracing output, that a tracing failure has occurred; as well as One or more subsequent images are examined to detect the object.

14. A system for controlling a movable object to track a target, the system comprising: A processor and a memory, the memory being used to store a computer program; The processor is used to: determining, in one or more images captured by the movable object, a target to be tracked by the movable object; controlling the movable object to track a target object using the target; as well as During tracking, if the target is lost in one or more of the images, using one or more positioning devices to maintain the movable object within proximity of the target; Wherein, during the tracking process, if the target is lost in one or more of the images, one or more positioning devices are used to maintain the movable object within the proximity of the target, including: Based on the position information obtained by the positioning device on the movable object and the orientation deviation of the position information provided by the positioning device carried by the target, the movable object and the imaging device carried by the movable object are controlled to rotate around the same rotation axis to adjust the shooting angle of the imaging device so that the movable object is maintained within the proximity of the target.

15. A movable object, comprising: processor, memory and imaging device; A processor and a memory, the memory being used to store a computer program; The processor is used to: determining a target to be tracked by the movable object in one or more images captured by the movable object; controlling the movable object to track a target object using the target; as well as During tracking, if the target is lost in one or more of the images, using one or more positioning devices to maintain the movable object within proximity of the target; Wherein, during the tracking process, if the target is lost in one or more of the images, one or more positioning devices are used to maintain the movable object within the proximity of the target, including: Based on the position information obtained by the positioning device on the movable object and the orientation deviation of the position information provided by the positioning device carried by the target, the movable object and the imaging device carried by the movable object are controlled to rotate around the same rotation axis to adjust the shooting angle of the imaging device so that the movable object is maintained within the proximity of the target.

Images