Aircraft dynamic target tracking navigation method, device and readable medium

Abstract

The present invention provides a method, device and computer-readable medium for tracking and navigating a dynamic target of an aircraft. The method includes: obtaining visual odometer information and an image frame sequence of the aircraft, the image frame sequence including an RGB image and a corresponding depth image; detecting whether there is a dynamic target in the RGB image, and if so, calculating the current three-dimensional position of the dynamic target in the local coordinate system, and predicting the future motion trajectory of the dynamic target; constructing a voxel grid map of the current local environment based on the visual odometer information and the depth image; expanding and generating a neighbor node set of the aircraft through discretized control input, and filtering out candidate tracking paths from the neighbor node set according to the voxel grid map; selecting a collision-free and optimal tracking performance tracking trajectory from the candidate tracking paths based on a heuristic loss function, and the aircraft flies according to the tracking trajectory. The present invention can track dynamic targets in real time and stably in a dense environment.

Description

Technical Field

[0001] The present invention mainly relates to the field of autonomous tracking and navigation, and in particular to a method, device and computer-readable medium for tracking and navigating a dynamic target of an aircraft. Background Art

[0002] With the rapid and high-quality development of aircraft technology, it has been widely used in various industries. However, considering that in some dense environments, such as intrusion interception, tracking photography and target search and rescue, the environment layout is unknown, obstacles are densely distributed, the target movement is unknown and severely blocked, it is not suitable for large reconnaissance equipment to enter, and manual exploration is inconvenient and even dangerous. Using aircraft to replace manual operations provides a new solution for the above environment, but at the same time it will inevitably put forward higher requirements on the autonomous dynamic target tracking performance of aircraft in dense environments, that is, it is hoped that the aircraft can achieve smooth and robust autonomous tracking flight for dynamic targets in dense environments.

[0003] Since the target motion in dense environments is unknown and severely obstructed by obstacles, and the visual sensor's field of view (FOV) is fixed and limited, it is extremely challenging for aircraft to locate and predict dynamic targets in dense environments. On the other hand, due to the unknown layout of dense environments and the large number of obstacles, the existing tracking trajectory planning algorithms are prone to generate collision paths, and dynamic targets are also easily lost during the tracking process. In addition, in order to improve the robustness of tracking flight, it is also necessary to design robust control strategies to deal with possible extreme situations, which also brings new challenges to aircraft. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, device and computer-readable medium for dynamic target tracking and navigation of an aircraft, so as to solve the problem that the existing tracking and navigation methods are not suitable for scenarios where the target motion conditions are unknown in dense environments.

[0005] To solve the above technical problems, the present invention provides a method for tracking and navigating a dynamic target of an aircraft, comprising the following steps: obtaining visual odometer information and an image frame sequence of the aircraft, wherein the image frame sequence includes an RGB image and a corresponding depth image; detecting whether there is a dynamic target in the RGB image, and if so, calculating the current three-dimensional position of the dynamic target in the local coordinate system, and predicting the future motion trajectory of the dynamic target; constructing a voxel grid map of the current local environment based on the visual odometer information and the depth image; expanding and generating a neighbor node set of the aircraft through discretized control input, and filtering out candidate tracking paths from the neighbor node set according to the voxel grid map; selecting a collision-free and optimal tracking performance tracking trajectory from the candidate tracking paths based on a heuristic loss function, and the aircraft flies according to the tracking trajectory, wherein the heuristic loss function includes an actual cost and a heuristic cost.

[0006] Optionally, the heuristic cost is a distance loss value between the target state of the aircraft and the predicted trajectory point of the dynamic target.

[0007] Optionally, the heuristic cost can be expressed as follows:

[0008] h A =D(X Abest ,X goal )

[0009]

[0010] Among them, h A is the heuristic cost, X Abest is the target state of the aircraft, X goal Predict the trajectory points for the dynamic target, X target is the current three-dimensional position of the dynamic target, X predicted is the future motion trajectory of the dynamic target, is the weight, D(X Abest ,X goal ) is the target state of the aircraft and the distance loss value of the predicted trajectory point of the dynamic target.

[0011] Optionally, detecting whether there is a dynamic target in the RGB image includes: detecting whether there are joint points in the RGB image through a 2D posture detector, and the joint points include a left waist joint point, a right waist joint point and a neck joint point.

[0012] Optionally, calculating the current three-dimensional position of the dynamic target in the local coordinate system includes: calculating the pixel position of the center of the dynamic target in the RGB image based on the joint point; calculating the three-dimensional coordinates of the dynamic target in the camera coordinate system based on the pixel position; and calculating the current three-dimensional position of the dynamic target in the local coordinate system based on the three-dimensional coordinates.

[0013] Optionally, the pixel position of the center of the dynamic target in the RGB image is calculated by the following formula:

[0014] [u cen ,v cen ] T =[((u rHip +u lHip ) / 2+u Neck ) / 2,((v rHip +v lHip ) / 2+v Neck ) / 2] T

[0015] where u cen ,v cen is the pixel position of the center of the dynamic target in the RGB image, [u rHip ,v rHip ] T ,[u lHip ,v lHip ] T ,[u Neck ,v Neck ] T They are the pixel positions of the right waist joint point, the pixel positions of the left waist joint point, and the pixel positions of the neck joint point.

[0016] Optionally, predicting the future motion trajectory of the dynamic target includes: adding the three-dimensional position to a fixed-length sequence composed of three-dimensional positions, fitting a Bezier curve that takes sequence weights into account, and predicting the future motion trajectory of the dynamic target based on the Bezier curve.

[0017] Optionally, the method further includes: detecting the dynamic target during flight, and if the dynamic target is lost, the aircraft actively performs exploration flight and attempts to relocate the dynamic target according to the future motion trajectory of the dynamic target.

[0018] Optionally, actively exploring flight and attempting to relocate the dynamic target includes: using a path search algorithm to generate a first trajectory from the current position of the aircraft to the position when the dynamic target is lost; discretizing the latest predicted future motion trajectory, and using a path search algorithm to generate a second trajectory from the position when the dynamic target is lost to the vertex of the future trajectory of the dynamic target; connecting the first trajectory and the second trajectory into a complete relocation path; generating a safe trajectory based on the relocation path, and the aircraft flies according to the safe trajectory and attempts to relocate the dynamic target.

[0019] Optionally, the method further includes: if there is no dynamic target in the RGB image, cyclically monitoring an external voice signal; if a voice signal is monitored, understanding semantic information in the voice signal, and performing corresponding actions according to the semantic information.

[0020] Optionally, performing corresponding actions according to the semantic information includes: calculating the relative position and posture of the dynamic target relative to the current aircraft according to the semantic information; and controlling the aircraft to turn or move according to the relative position and posture, gradually approaching the area where the dynamic target is located.

[0021] In order to solve the above technical problems, the present invention provides an aircraft dynamic target tracking and navigation device, comprising: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the above method.

[0022] In order to solve the above technical problem, the present invention provides a computer readable medium storing a computer program code, and the computer program code implements the above method when executed by a processor.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] The aircraft dynamic target tracking and navigation method of the present invention is based on visual perception of the surrounding environment and dynamic targets, predicts the future motion trajectory of the target according to the three-dimensional position of the dynamic target, and selects a collision-free tracking trajectory with optimal tracking performance from candidate tracking paths based on a heuristic loss function. The tracking trajectory is collision-free for both static obstacles and dynamic targets, and the aircraft can track the dynamic target in real time and stably in a dense environment on the tracking trajectory. When the dynamic target is lost, the aircraft dynamic target tracking and navigation method of the present invention can significantly improve the robustness of dynamic target tracking flight in a dense environment by actively exploring flight and trying to reposition the dynamic target. In extreme cases, by monitoring external voice signals and controlling the movement of the aircraft based on the external voice signals, the failure of the tracking and navigation method in extreme cases can be overcome, and the robustness of the tracking and navigation method can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are included to provide a further understanding of the present application, and they are included and constitute a part of the present application. The accompanying drawings illustrate embodiments of the present application and together with the present specification serve to explain the principles of the present invention. In the accompanying drawings:

[0026] Figure 1 The present invention is a flowchart of a method for tracking and navigating a dynamic target of an aircraft according to an embodiment of the present invention.

[0027] Figure 2 is a pinhole imaging model diagram according to an embodiment of the present invention.

[0028] Figure 3 is a schematic diagram of coordinate system conversion according to an embodiment of the present invention.

[0029] Figure 4A is a schematic diagram of an aircraft flying in a dense environment according to an embodiment of the present invention.

[0030] Figure 4B It's the aircraft in Figure 4A A voxel grid of the local environment at the time of the position.

[0031] Figure 5 Schematic diagram of a neighbor node set of an aircraft generated according to an embodiment of the present invention.

[0032] Figure 6 It is a system block diagram of an aircraft dynamic target tracking and navigation device according to one embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0034] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0035] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be considered as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so that once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0036] In the description of the present application, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present application; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0037] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0038] In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above words have no special meaning and cannot be understood as limiting the scope of protection of this application. In addition, although the terms used in this application are selected from well-known and commonly used terms, some terms mentioned in the specification of this application may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand this application not only by the actual terms used, but also by the meaning implied by each term.

[0039] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0040] As mentioned in the background technology, due to the unknown layout of dense environments and the large number of obstacles, the existing tracking trajectory planning algorithms are prone to generate collision paths, and dynamic targets are also easily lost during the tracking process. In addition, in order to improve the robustness of tracking flight, it is also necessary to design robust control strategies to deal with possible extreme situations, which also brings new challenges to the aircraft. In view of this, the present invention provides an aircraft dynamic target tracking and navigation method, which enables it to stably locate and predict dynamic targets in real time in dense environments, ensure that the trajectory of the aircraft is collision-free with static obstacles and dynamic targets during trajectory planning, and actively explore relocation targets after losing target observation values, and adopt multi-modal control to deal with extreme situations.

[0041] Figure 1 FIG. 1 is a flow chart of a method for tracking and navigating a dynamic target of an aircraft according to an embodiment of the present invention. Figure 1 As shown, the aircraft dynamic target tracking and navigation method 100 (hereinafter referred to as the tracking and navigation method 100) includes the following steps:

[0042] Step S11: Obtain visual odometer information and an image frame sequence of the aircraft, where the image frame sequence includes an RGB image and a corresponding depth image.

[0043] The visual odometer information includes the horizontal coordinates, altitude information and quaternion attitude information of the aircraft in the local coordinate system. The acquisition of the visual odometer information and image frame sequence of the aircraft includes the following steps: using the visual positioning camera to estimate the horizontal coordinates and altitude information of the aircraft in the local coordinate system with the take-off point as the origin [p bx ,p by ,p bz ] Tand quaternion attitude information [q bx ,q by ,q bz ,q bw ] T ; Use the depth camera to collect RGB images and depth information, calculate the depth image, and align the depth image with the RGB image. The visual positioning camera can be an InterRealsense T265 camera, and the depth camera can be an InterRealsense D415 camera. The InterRealsense T265 camera is placed directly above the InterRealsense D415 camera, and they are fixed on the aircraft body so that they both face the front of the aircraft.

[0044] Step S12: Detect whether there is a dynamic target in the RGB image, if yes, proceed to step S13.

[0045] The dynamic target may be a moving person or a moving animal. Detecting whether there is a dynamic target in the RGB image includes detecting whether there are joints in the RGB image through the OpenPose real-time 2D pose detector, and the joints include the left waist joint, the right waist joint, and the neck joint. The OpenPose real-time 2D pose detector is a detector that can track human facial expressions, torsos, limbs, and even fingers.

[0046] Step S13: Calculate the current three-dimensional position of the dynamic target in the local coordinate system and predict the future motion trajectory of the dynamic target.

[0047] In some embodiments, the current three-dimensional position of the dynamic target in the local coordinate system is calculated using the pinhole imaging model and the coordinate system transformation, including the following steps:

[0048] Step a: Calculate the pixel position of the center of the dynamic target in the RGB image based on the joint points.

[0049] Taking a moving person as an example, the pixel position of the center of the dynamic target in the RGB image is calculated by the following formula:

[0050] [u cen ,v cen ] T =[((u rHip +u lHip ) / 2+u Neck ) / 2,((v rHip +v lHip ) / 2+v Neck ) / 2] T

[0051] where u cen ,vcen is the pixel position of the center of the dynamic target in the RGB image, [u rHip ,v rHip ] T ,[u lHip ,v lHip ] T ,[u Neck ,v Neck ] T They are the pixel positions of the right waist joint point, the pixel positions of the left waist joint point, and the pixel positions of the neck joint point. As can be seen from the above, in this embodiment, the center of the moving person's body is used as the dynamic target point to be tracked.

[0052] Step b: Calculate the three-dimensional coordinates of the dynamic target in the camera coordinate system based on the pixel position.

[0053] Figure 2 is a pinhole imaging model diagram according to an embodiment of the present invention. Figure 2 As shown, establish the camera coordinate system cx c -y c -z c , taking the optical center c of the D415 camera as the origin of the coordinate system. Establish the pixel coordinate system ouv on the pixel plane of the camera. The coordinate of the target P in the camera coordinate system is [X c ,Y c ,Z c ] T , the distance from the camera optical center to the physical imaging plane is the focal length f. The target P passes through the optical center and is projected onto the point P' on the pixel plane. Its pixel coordinates are [u cen ,v cen ] T According to the corresponding correspondence, the mapping relationship corresponds to the amount of scaling and translation of a scale, so the three-dimensional coordinates [X c ,Y c ,Z c ] T This can be achieved through the following formula:

[0054]

[0055] in is the depth value corresponding to the dynamic target, u cen ,v cen is the pixel position of the center of the dynamic target in the RGB image, f x ,f y ,c x ,c y It is the internal parameter of the camera and has been calibrated at the factory.

[0056] Step c: Calculate the current three-dimensional position of the dynamic target in the local coordinate system based on the three-dimensional coordinates.

[0057] Figure 3 is a schematic diagram of coordinate system conversion according to an embodiment of the present invention. Figure 3 As shown, the coordinate system conversion of the present application involves a local coordinate system 31, a body coordinate system 32 and a camera coordinate system 33. The local coordinate system 31 can be represented as ox local -y local -z local , with the take-off point position o as the origin of the coordinate system. The body coordinate system 32 refers to the coordinate system box established on the aircraft body b-o -y b-o -z b-o , with the optical center c of the D415 camera as the origin of the body coordinate system. The camera coordinate system 33 can be expressed as cx c -y c -z c , the optical center c of the D415 camera is also used as the origin of the coordinate system. Since the aircraft is moving, the three-dimensional position of the dynamic target in the local coordinate system 31 at the current moment should be obtained by the relationship between the body coordinate system 32 and the camera coordinate system 33, the horizontal coordinates of the aircraft in the local coordinate system 31, the height information and the quaternion attitude information. For example, to calculate the current three-dimensional position of the dynamic target in the local coordinate system [X b ,Y b ,Z b ] T This can be achieved through the following formula:

[0058]

[0059]

[0060] Where [X b_o ,Y b_o ,Z b_o ] T is the three-dimensional coordinate of the dynamic target in the body coordinate system, [X c ,Y c ,Z c ] T is the three-dimensional coordinate of the dynamic target in the camera coordinate system, [p bx ,p by ,p bz ] is the height information of the aircraft in the local coordinate system, [q bx ,q by ,q bz ,q bw ] is the quaternion attitude information of the aircraft in the local coordinate system.

[0061] The calculated current three-dimensional position of the dynamic target in the local coordinate system [X b ,Y b ,Z b ] T join in In a fixed-length sequence, the earliest data is deleted, that is, the sequence content is updated in chronological order. A Bezier curve that takes into account the sequence weight is fitted with the updated sequence, and the target's future motion trajectory is predicted based on this. Specifically, the loss function J of the Bezier curve designed considering the decrease in credibility of the sequence over time is as follows:

[0062]

[0063]

[0064] where w i is the weight size, which is inversely proportional to the time length. In the sequence, as time changes, the latest 3D position is very important in trajectory prediction, so the weight is the largest, and the older 3D position is meaningless for trajectory prediction, so the weight is smaller. B(t i ) represents the value predicted by the Bezier curve at the i-th moment, represents the three-dimensional position at the i-th moment in the sequence. p represents a weighted value, Represents the second-order orthogonal term of the predicted trajectory.

[0065] By continuously optimizing the above loss function, a Bezier curve that takes the sequence weight into account can be fitted, and the future motion trajectory of the target can be predicted using the fitted Bezier curve.

[0066] Step S14: construct a voxel grid map of the current local environment based on the visual odometer information and the depth image. The voxel grid map of the current local environment expresses obstacle information.

[0067] Figure 4A is a schematic diagram of an aircraft flying in a dense environment according to an embodiment of the present invention. Figure 4B It's the aircraft in Figure 4A A voxel grid of the local environment at the location. Figure 4A As shown, the aircraft 40A is flying in a dense environment, and there are a series of obstacles 41A in the dense environment. The aircraft 40 can use the odometer information and the depth image to construct a local voxel grid map of the aircraft's location in real time. The constructed local voxel grid map is as follows: Figure 4B As shown, each voxel in the voxel grid map stores the occupancy information of the environment. For example, the voxel grid map includes aircraft occupancy information 40B and obstacle occupancy information 41B.

[0068] Step S15: Expand and generate a neighbor node set of the aircraft by discretizing the control input, and select candidate tracking paths from the neighbor node set according to the voxel grid graph.

[0069] The neighbor node set of the aircraft can be expanded and generated by the following formula:

[0070]

[0071] U D : ΔT:

[0072] Where X A is the set of neighboring nodes, E is the 3*3 identity matrix, ΔT is the time interval sequence item, U D is the acceleration interval sequence term, X c is the current state of the aircraft. c Including the position and speed of the aircraft. m is the maximum acceleration that can be achieved considering the UAV dynamic constraints, t m is a fixed time interval, and N is a fixed value.

[0073] Figure 5 FIG. 1 is a schematic diagram of a neighbor node set of an aircraft generated according to an embodiment of the present invention. Figure 5 As shown, the neighbor node set of the aircraft includes path 51, path 52, path 53, path 54, path 55, path 56, and path 57. According to the voxel grid diagram, there is an obstacle B1 on path 51, there are obstacles B2 on paths 52 and 55, and there are obstacles B3 on paths 53 and 54. Therefore, the candidate tracking paths are selected as path 56 and path 57.

[0074] Step S16: A tracking trajectory with no collision and optimal tracking performance is selected from the candidate tracking paths based on a heuristic loss function, and the aircraft flies according to the tracking trajectory, wherein the heuristic loss function includes an actual cost and a heuristic cost.

[0075] The heuristic loss function can be expressed as follows:

[0076] min f A =g A +h A

[0077] where f A is the heuristic loss function, g A is the actual cost, which represents the cost of the current state of the aircraft to reach the target state of the aircraft, h A is the heuristic cost.

[0078] In some embodiments, the heuristic cost is a distance loss value between the target state of the aircraft and the predicted trajectory point of the dynamic target.

[0079] Continue to refer Figure 5 , taking path 56 as an example, the starting point of path 56 is the current state of the aircraft X c , the end point is the target state X of the aircraft Abest , then the actual cost of the heuristic loss function is the vehicle's change from the current state X c Arriving at the target state X Abest In the present invention, the dynamic target is in constant motion, such as Figure 5 As shown, when the aircraft is in the current state X c When the dynamic target is at the current three-dimensional position X target , when the dynamic target moves to the predicted trajectory point X goal When we assume that the aircraft reaches the target state X Abest The heuristic cost of the heuristic loss function is the target state X of the aircraft. Abest With the predicted trajectory point X goal The distance loss value.

[0080] The heuristic cost can be expressed as follows:

[0081] h A =D(X Abest ,X goal )

[0082]

[0083] Among them, h A is the heuristic cost, X Abest is the target state of the aircraft, X goal Predict trajectory points for dynamic targets, D(X Abest ,X goal ) is the distance loss value between the target state of the aircraft and the predicted trajectory point of the dynamic target, X target is the current three-dimensional position of the dynamic target, X predicted is the future motion trajectory of the dynamic target, is the weight, It is proportional to the time interval of the aircraft's flight.

[0084] Solve the heuristic loss function f that minimizes A The optimal target state X Abest , a collision-free tracking trajectory with optimal tracking performance can be determined. Figure 5 In the example, based on the heuristic loss function, a tracking trajectory with no collision and the best tracking performance is selected as path 56.

[0085] The aircraft dynamic target tracking and navigation method of the present invention is based on visual perception of the surrounding environment and dynamic targets, predicts the future motion trajectory of the target according to the three-dimensional position of the dynamic target, and selects a collision-free tracking trajectory with optimal tracking performance from candidate tracking paths based on a heuristic loss function. The tracking trajectory is collision-free for both static obstacles and dynamic targets, and on the tracking trajectory, the aircraft can track the dynamic target in real time and stably in a dense environment.

[0086] In some embodiments, the method further includes: if there is no dynamic target in the RGB image, then cyclically monitoring an external voice signal; if a voice signal is monitored, then understanding the semantic information in the voice signal, and performing corresponding actions according to the semantic information.

[0087] When the aircraft is in an extreme situation, for example, in the initial state, the dynamic target is far away from the aircraft or is located in the aircraft's observation blind spot, and the aircraft is trapped in a dead end surrounded by multiple obstacles during tracking. The aircraft can monitor the voice signal emitted by a third party who has global information through the voice recognition module. The voice signal is generally a voice simulation signal. The monitored voice simulation signal is converted into a string digital signal. Understand the semantic information in the string digital signal, and extract the posture relationship of the current dynamic target relative to the aircraft based on the semantic information. For example, the semantic information includes the current state of the dynamic target. The aircraft calculates the relative posture of the dynamic target relative to the current aircraft based on the current state of the aircraft and the current state of the dynamic target, and controls the aircraft to turn or move according to the relative posture, gradually approaching the area where the dynamic target is located.

[0088] The aircraft dynamic target tracking and navigation method of the present invention can overcome the failure of the tracking and navigation method in extreme cases by monitoring external voice signals and controlling the movement of the aircraft based on the external voice signals, thereby improving the robustness of the tracking and navigation method.

[0089] In some embodiments, the method further includes: detecting a dynamic target during flight, and if the dynamic target is lost, the aircraft actively explores the flight and attempts to relocate the dynamic target according to the future motion trajectory of the dynamic target. For example, the steps of actively exploring the flight and attempting to relocate the dynamic target according to the future motion trajectory of the dynamic target include:

[0090] The shortest safe path from the current position of the aircraft to the position when the dynamic target is lost is generated by using a path search algorithm, and the shortest safe path is recorded as the first trajectory;

[0091] Discretize the latest predicted future motion trajectory, use the path search algorithm to generate a collision-free path from the position when the dynamic target is lost to the vertex of the future trajectory of the dynamic target, and record the collision-free path as the second trajectory;

[0092] Connecting the first trajectory and the second trajectory into a complete relocation path, and generating a flight corridor based on the relocation path;

[0093] A spatiotemporally optimal and collision-free safe trajectory is generated within the flight corridor. The aircraft flies according to the safe trajectory and attempts to relocate the dynamic target.

[0094] In some embodiments, the step of actively exploring and attempting to relocate the dynamic target further includes generating a new safe trajectory if a new obstacle is observed while flying along the safe trajectory.

[0095] The aircraft dynamic target tracking and navigation method of the present invention can significantly improve the robustness of dynamic target tracking flight in dense environments by actively exploring flight and attempting to reposition the dynamic target.

[0096] The present application also includes an aircraft dynamic target tracking and navigation device, including a memory and a processor. The memory is used to store instructions executable by the processor; the processor is used to execute the instructions to implement the aircraft dynamic target tracking and navigation method mentioned above.

[0097] Figure 6 1 is a system block diagram of an aircraft dynamic target tracking and navigation device according to an embodiment of the present application. Figure 6 As shown, the aircraft dynamic target tracking navigation device 600 may include an internal communication bus 601, a processor 602, a read-only memory (ROM) 603, a random access memory (RAM) 604, and a communication port 605. When applied on a personal computer, the aircraft dynamic target tracking navigation device 600 may also include a hard disk 606. The internal communication bus 601 can realize data communication between the components of the aircraft dynamic target tracking navigation device 600. The processor 602 can make judgments and issue prompts. In some embodiments, the processor 602 can be composed of one or more processors. The communication port 605 can realize data communication between the aircraft dynamic target tracking navigation device 600 and the outside. In some embodiments, the aircraft dynamic target tracking navigation device 600 can send and receive information and data from the network through the communication port 605. The aircraft dynamic target tracking navigation device 600 may also include different forms of program storage units and data storage units, such as a hard disk 606, a read-only memory (ROM) 603 and a random access memory (RAM) 604, which can store various data files used for computer processing and / or communication, and possible program instructions executed by the processor 602. The processor executes these instructions to implement the main part of the method. The result of the processor processing is transmitted to the user equipment through the communication port and displayed on the user interface.

[0098] The above-mentioned operation method can be implemented as a computer program, stored in the hard disk 606, and loaded into the processor 602 for execution, so as to implement the aircraft dynamic target tracking and navigation method of the present application.

[0099] The present application also includes a computer-readable medium storing a computer program code, which, when executed by a processor, implements the above-mentioned aircraft dynamic target tracking and navigation method.

[0100] When the aircraft dynamic target tracking and navigation method is implemented as a computer program, it can also be stored in a computer-readable storage medium as a product. For example, a computer-readable storage medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD)), a smart card, and a flash memory device (e.g., an electrically erasable programmable read-only memory (EPROM), a card, a stick, a key drive). In addition, the various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) that can store, contain and / or carry code and / or instructions and / or data.

[0101] It should be understood that the embodiments described above are only illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions herein, or combinations thereof.

[0102] Some aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor may be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program codes. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes ...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs ...), smart cards, and flash memory devices (e.g., cards, sticks, key drives ...).

[0103] A computer-readable medium may include a propagated data signal containing computer program code, such as in baseband or as part of a carrier wave. The propagated signal may have a variety of manifestations, including electromagnetic, optical, etc., or a suitable combination. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium, which may be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit a program for use. The program code on the computer-readable medium may be propagated via any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or any combination of the above mediums.

[0104] The basic concepts have been described above. Obviously, for those skilled in the art, the above application disclosure is only an example and does not constitute a limitation of the present application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements and amendments to the present application. Such modifications, improvements and amendments are suggested in the present application, so such modifications, improvements and amendments still belong to the spirit and scope of the exemplary embodiments of the present application.

Claims

1. A method for tracking and navigating a dynamic target of an aircraft, characterized in that: The following steps are involved: Acquire visual odometer information and an image frame sequence of the aircraft, wherein the image frame sequence includes an RGB image and a corresponding depth image; Detect whether there is a dynamic target in the RGB image. If so, calculate the current three-dimensional position of the dynamic target in the local coordinate system and predict the future motion trajectory of the dynamic target; Constructing a voxel grid map of the current local environment based on the visual odometry information and the depth image; Expanding and generating a neighbor node set of the aircraft by discretizing control input, and screening candidate tracking paths from the neighbor node set according to the voxel grid graph; Selecting a tracking trajectory with no collision and optimal tracking performance from the candidate tracking paths based on a heuristic loss function, and the aircraft flying according to the tracking trajectory, wherein the heuristic loss function includes an actual cost and a heuristic cost; Wherein, detecting whether there is a dynamic target in the RGB image includes: detecting whether there are joint points in the RGB image by a 2D posture detector, wherein the joint points include a left waist joint point, a right waist joint point, and a neck joint point; Predicting the future motion trajectory of the dynamic target includes: adding the three-dimensional position to a fixed-length sequence composed of three-dimensional positions, fitting a Bezier curve that takes sequence weights into account, and predicting the future motion trajectory of the dynamic target based on the Bezier curve.

2. The method according to claim 1, characterized in that The heuristic cost is a distance loss value between the target state of the aircraft and the predicted trajectory point of the dynamic target.

3. The method according to claim 2, characterized in that The heuristic cost is expressed as follows: h A =D(X Abest ,X goal ) Among them, h A is the heuristic cost, X Abest is the target state of the aircraft, X goal Predict the trajectory points for the dynamic target, X target is the current three-dimensional position of the dynamic target, X predicted is the future motion trajectory of the dynamic target, is the weight, D(X Abest ,X goal ) is the target state of the aircraft and the distance loss value of the predicted trajectory point of the dynamic target.

4. The method according to claim 1, characterized in that Calculating the current three-dimensional position of the dynamic target in the local coordinate system includes: Calculate the pixel position of the center of the dynamic target in the RGB image according to the joint point; Calculate the three-dimensional coordinates of the dynamic target in the camera coordinate system according to the pixel position; The current three-dimensional position of the dynamic target in the local coordinate system is calculated according to the three-dimensional coordinates.

5. The method according to claim 4, characterized in that The pixel position of the center of the dynamic target in the RGB image is calculated by the following formula: [u cen ,v cen ] T =[((u rHip +u lHip ) / 2+u Neck ) / 2,((v rHip +v lHip ) / 2+v Neck ) / 2] T where u cen ,v cen is the pixel position of the center of the dynamic target in the RGB image, [u rHip ,v rHip ] T ,[u lHip ,v lHip ] T ,[u Neck ,v Neck ] T They are the pixel positions of the right waist joint point, the pixel positions of the left waist joint point, and the pixel positions of the neck joint point.

6. The method according to claim 1, characterized in that Also includes: The dynamic target is detected during the flight. If the dynamic target is lost, the aircraft actively conducts exploration flight and attempts to reposition the dynamic target according to the future motion trajectory of the dynamic target.

7. The method according to claim 6, characterized in that Active exploratory flight and attempting to relocate the dynamic target include: Generate a first trajectory from the current position of the aircraft to the position when the dynamic target is lost by using a path search algorithm; Discretize the latest predicted future motion trajectory, and use a path search algorithm to generate a second trajectory from the position when the dynamic target is lost to the vertex of the future trajectory of the dynamic target; Connecting the first trajectory and the second trajectory into a complete relocation path; A safety trajectory is generated according to the repositioning path, and the aircraft flies according to the safety trajectory and attempts to reposition the dynamic target.

8. The method according to claim 1, characterized in that Also includes: If there is no dynamic target in the RGB image, the external voice signal is monitored cyclically. If a voice signal is monitored, the semantic information in the voice signal is understood and corresponding actions are performed according to the semantic information.

9. The method according to claim 8, characterized in that Executing corresponding actions according to the semantic information includes: Calculate the relative position of the dynamic target with respect to the current aircraft according to the semantic information; The aircraft is controlled to turn or move according to the relative position and posture, and gradually approaches the area where the dynamic target is located.

10. An aircraft dynamic target tracking and navigation device, characterized in that: include: a memory for storing instructions executable by a processor; A processor, configured to execute the instructions to implement the method according to any one of claims 1 to 9.

11. A computer readable medium storing computer program code, wherein the computer program code, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images