A UAV visual target tracking system and method based on finite state machine

Abstract

The patent application of the present invention provides a UAV visual target tracking system and method based on a finite state machine, including: a UAV platform, a finite state machine and a long-term target tracking unit; the finite state machine is respectively connected to the UAV platform and the long-term target tracking unit for communication; the UAV platform is used to collect visual images and decode the visual images to obtain video stream information; the finite state machine is used to obtain the state information of the target in the video stream information according to the video stream information, and trigger an event corresponding to the state information; the long-term target tracking unit is used to receive the event corresponding to the state information and execute the action corresponding to the event; the patent application of the present invention is conducive to expanding the UAV visual target tracking system to achieve the effect of long-term strong and robust target tracking of the UAV under narrow field of view conditions within the regional range.

Description

Technical Field

[0001] The patent application of this invention belongs to the field of data communication and processing technology, and specifically relates to a UAV visual target tracking system and method based on a finite state machine. Background Art

[0002] At present, the UAV visual target tracking system refers to a system that can automatically identify, track and lock the target during the flight of the UAV. Such a system usually uses technologies such as image recognition and target tracking. With the rapid development of computer vision and artificial intelligence technology, the accuracy and efficiency of target recognition and tracking have been greatly improved. The current UAV target locking system can recognize and track various types of targets in real time, including vehicles, people and buildings.

[0003] The self-visual target tracking system of UAV usually uses various sensors and devices, such as cameras, radars, infrared sensors, etc., so that the UAV can perceive the surrounding environment and perform tasks. The optoelectronic payload is an optoelectronic sensor payload installed on the UAV, which can execute commands such as target locking, target tracking, and geographic tracking. It is mainly used for the target search, reconnaissance and monitoring tasks of the UAV. The optoelectronic payload can obtain high-definition images and data, and can quickly and efficiently complete the target search, reconnaissance, and monitoring tasks, thereby improving the efficiency and accuracy of the task.

[0004] Although the current drone target locking system has made significant technical progress, there are still some defects and challenges. For example, in complex environments and lighting conditions, the drone target tracking system may misidentify the target, especially when the target is similar to the surrounding environment or there is occlusion, which may cause the drone to track the wrong target, causing damage to the target or mission failure. The performance of the drone target tracking system may also be limited in complex scenes, such as urban environments or dense crowds, where there may be many occlusions, similar targets, and rapid dynamic changes, making target identification and tracking more difficult. Summary of the invention

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present invention patent application proposes a UAV visual target tracking system based on a finite state machine, comprising:

[0006] UAV platform, finite state machine and long-term target tracking unit;

[0007] The finite state machine is respectively connected to the UAV platform and the long-term target tracking unit for communication;

[0008] The UAV platform is used to collect visual images and decode the visual images to obtain video stream information;

[0009] The finite state machine is used to obtain the state information of the target in the video stream information according to the video stream information, and trigger an event corresponding to the state information;

[0010] The long-term target tracking unit is used to receive an event corresponding to the state information and execute an action corresponding to the event.

[0011] Preferably, the state information in the finite state machine includes: target discovery, target handover, target tracking and target loss.

[0012] Preferably, the finite state machine acquires the state information of the target in the video stream information and triggers an event corresponding to the state information, including:

[0013] When the state information of the target in the acquired video stream information is target discovery, triggering a target handover event;

[0014] When the state information of the target in the acquired video stream information is target handover, triggering a target tracking event;

[0015] When the state information of the target in the acquired video stream information is that the target is lost, a target discovery event is triggered.

[0016] Preferably, the finite state machine is further used to obtain the pixel coordinates of the target and generate a pixel offset according to the pixel coordinates of the target.

[0017] Preferably, the UAV platform is also used to receive the pixel offset and perform a corresponding servo action according to the pixel offset.

[0018] Preferably, the UAV platform and the finite state machine are communicatively connected via an asynchronous serial port.

[0019] Based on the same inventive concept, the patent application of the present invention also provides a UAV visual target tracking method based on a finite state machine, including:

[0020] Collecting visual images and decoding the visual images to obtain video stream information;

[0021] According to the video stream information, obtaining the state information of the target in the video stream information, and triggering an event corresponding to the state information;

[0022] According to the event corresponding to the status information, an action corresponding to the event is executed.

[0023] Preferably, the status information includes: target discovery, target handover, target tracking and target loss.

[0024] Preferably, the acquiring the state information of the target in the video stream information and triggering an event corresponding to the state information includes:

[0025] When the state information of the target in the acquired video stream information is target discovery, triggering a target handover event;

[0026] When the state information of the target in the acquired video stream information is target handover, triggering a target tracking event;

[0027] When the state information of the target in the acquired video stream information is that the target is lost, a target discovery event is triggered.

[0028] Preferably, after decoding the visual image to obtain the video stream information, the method further includes:

[0029] Obtaining pixel coordinates of a target in the video stream information;

[0030] Generate a pixel offset according to the pixel coordinates of the target;

[0031] According to the pixel offset, a corresponding servo action is performed.

[0032] Compared with the closest prior art, the patent application of this invention has the following beneficial effects:

[0033] The patent application of the present invention provides a UAV visual target tracking system and method based on a finite state machine, including: a UAV platform, a finite state machine and a long-term target tracking unit; the finite state machine is respectively connected to the UAV platform and the long-term target tracking unit for communication; the UAV platform is used to collect visual images and decode the visual images to obtain video stream information; the finite state machine is used to obtain the state information of the target in the video stream information according to the video stream information, and trigger the event corresponding to the state information; the long-term target tracking unit is used to receive the event corresponding to the state information and execute the action corresponding to the event; the patent application of the present invention uses a finite state machine to monitor the state information of the target in the video stream information, and triggers the state transition event according to the corresponding state information, and executes the corresponding action through the long-term target tracking unit; it is conducive to expanding the UAV visual target tracking system and achieving the effect of long-term strong and robust target tracking of the UAV under the condition of narrow field of view within the regional range. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of a UAV visual target tracking system based on a finite state machine provided for the patent application of the present invention;

[0035] Figure 2A schematic diagram of a framework of a UAV visual target tracking system based on a finite state machine provided in a specific embodiment of the present invention;

[0036] Figure 3 A schematic diagram of the operation logic of a computing board in a finite state machine-based unmanned aerial vehicle visual target tracking system provided by a specific embodiment of the present invention;

[0037] Figure 4 A schematic diagram of finite state machine state switching in a finite state machine-based UAV visual target tracking system provided by a specific embodiment of the present invention;

[0038] Figure 5 A schematic diagram of a flow chart of visual target tracking using a finite state machine-based UAV visual target tracking system provided in the patent application of the present invention;

[0039] Figure 6 A flowchart of a UAV visual target tracking method based on a finite state machine is provided for the patent application of the present invention. DETAILED DESCRIPTION

[0040] The specific implementation methods of the patent application of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0041] Embodiment 1:

[0042] The patent application of this invention provides a UAV visual target tracking system based on a finite state machine. The schematic diagram of the system structure is as follows: Figure 1 As shown, including:

[0043] UAV platform, finite state machine and long-term target tracking unit;

[0044] The finite state machine is respectively connected to the UAV platform and the long-term target tracking unit for communication;

[0045] The UAV platform is used to collect visual images and decode the visual images to obtain video stream information;

[0046] The finite state machine is used to obtain the state information of the target in the video stream information according to the video stream information, and trigger an event corresponding to the state information;

[0047] The long-term target tracking unit is used to receive an event corresponding to the state information and execute an action corresponding to the event.

[0048] The state information in the finite state machine includes: target discovery, target handover, target tracking and target loss.

[0049] The step of obtaining the state information of the target in the video stream information and triggering an event corresponding to the state information in the finite state machine includes:

[0050] When the state information of the target in the acquired video stream information is target discovery, triggering a target handover event;

[0051] When the state information of the target in the acquired video stream information is target handover, triggering a target tracking event;

[0052] When the state information of the target in the acquired video stream information is that the target is lost, a target discovery event is triggered.

[0053] The finite state machine is further used to obtain the pixel coordinates of the target and generate a pixel offset according to the pixel coordinates of the target.

[0054] The UAV platform is also used to receive the pixel offset and perform a corresponding servo action according to the pixel offset.

[0055] The UAV platform and the finite state machine are communicatively connected via an asynchronous serial port.

[0056] The patent application of the present invention provides a UAV visual target tracking system based on a finite state machine, which expands the UAV visual target tracking system based on a finite state machine, and is conducive to realizing long-term strong and robust single target tracking of the UAV under narrow field of view conditions within a regional range.

[0057] Embodiment 2:

[0058] A specific embodiment is used to illustrate a UAV visual target tracking system based on a finite state machine provided in the patent application of the present invention. Figure 2 As shown, it includes: a UAV platform and a computing board, wherein the UAV platform is equipped with an optoelectronic pod, which is used to collect visual images, receive pixel offsets and perform servo actions; the computing board is equipped with a finite state machine and a long-term target tracking unit, and the logical execution view of the computing board is as follows Figure 3 As shown, it is used to decode the video stream, monitor the state transition events in the finite state machine and execute the corresponding actions of the events; the finite state machine, the state includes [DETECTED, HANDOVER, LOCKED, LOST], the state is driven by the target, and represents target discovery, target handover, target tracking and target loss respectively, the event is the change of the corresponding state of the target, and the action is to execute the corresponding algorithm after the state changes; the long-term target tracking unit, corresponding to the state in the finite state machine, includes a target discovery module, a target handover module and a single target tracking module, each module has a preset algorithm, and receives an action instruction to load and run;

[0059] The target discovery module is pre-installed with deep learning target discovery algorithms such as Yolo and DETR, which can be used to search for specific targets;

[0060] The target handover module is pre-installed with a deep learning target handover algorithm such as Fast-Reid, which can be used for target handover;

[0061] The single target tracking module is pre-set with a target library, which includes target categories to be tracked, and / or target maps, and / or target feature maps for target matching; it is also pre-set with a deep learning single target tracking algorithm such as SiamRPN, which can be used for single target tracking;

[0062] The above algorithms are accelerated by NPU, GPU, XPU, etc. to improve the image processing speed;

[0063] In this embodiment, the computing board communicates with the optoelectronic pod to decode the video stream, monitor state transition events in the finite state machine, execute actions corresponding to the events, and load the algorithm to run;

[0064] The computing board is pre-installed with a real-time video stream decoding program, and the decoded image is used for each module of the long-term target tracking unit;

[0065] The computing board is a computer capable of executing one or more of the above functional combinations, and may be of X86 or ARM architecture;

[0066] The computing board supports the acceleration of intelligent algorithms such as CNN and can be a GPU, NPU, or XPU acceleration module;

[0067] Among them, the command sending and data / status return communication interface between the computing board and the optoelectronic pod adopts asynchronous 422 communication transmission.

[0068] The optoelectronic pod has a target servo tracking function and can receive pixel coordinate offset instructions to lock the target in the center of the field of view;

[0069] like Figure 4 The figure shows a state switching diagram of a finite state machine in an embodiment of the present invention, wherein a target state transition event is triggered by a state change, specifically including:

[0070] Event 1: After the target state changes to LOST, the target discovery event is triggered, driving the long-term target tracking unit to perform the target discovery action; the state can change to DETECTED;

[0071] Event 2: After the target state changes to DETECTD, the target handover event is triggered, driving the long-term target tracking unit to perform the target handover action; the state changes to HANDOVER;

[0072] Event 3: After the target state changes to HANDOVER, the target tracking event is triggered, driving the long-term target tracking unit to perform the target tracking action; if the tracking is successful, the state changes to LOCKED, and the target lost in tracking becomes LOST.

[0073] In this embodiment, after the finite state machine action is executed, the target queue state changes as follows:

[0074] When the target is empty or the target state is LOST, the target discovery action is executed, and the target queue state Q = {T1: DETECTED, T2: DETECTED, ..., T3: DETECTED}

[0075] When the target queue target state is DETECTED, the target handover action is executed, and the target queue state Q = {T1: HANDOVER, T2: DETECTED, ..., T2: DETECTED},

[0076] Furthermore, if the target handover fails, Q det The queue remains unchanged.

[0077] When the target queue target state is HANDOVER, the target tracking action is executed. After successful tracking, it is marked as LOCKED state, Q = {T1: LOCKED, T2: DETECTED, ..., T3: DETECTED}, and the target tracking algorithm continues to be executed;

[0078] Furthermore, if the tracking fails, Q = {T1:LOST, T2:DETECTED, ..., T3:DETECTED}, then return to the first step.

[0079] Embodiment 3:

[0080] Using the finite state machine-based UAV visual target tracking system provided by the patent application of this invention to perform visual target tracking, first, the test environment is constructed as follows:

[0081] The fixed-wing drone is equipped with an embedded AI computing device, the NVIDIA Jetson TX2 computing board, which has a 128-bit memory controller, 8GB RAM and 32GB ROM, including a 256-core NVIDIA Pascal GPU, a Denver 2 (Dual-Core) CPU cluster and 2 ARM Cortex-A57 QuadCore CPUs; it is equipped with a 3-light pod, visible light, infrared and laser ranging, and the communication method is video output interface: NET, supporting UDP / SDI, and the network port push streaming mode is H.264 stream.

[0082] The specific steps are as follows:

[0083] Step S1: The UAV is loaded with a computing board and an optoelectronic pod, and the computing board and the optoelectronic pod communicate via an asynchronous 422 serial port;

[0084] Step S2: Set the initial state:

[0085] Initially set the finite state machine target state to {NULL};

[0086] Preset video decoding program to obtain and decode H.264 visible light video stream in real time;

[0087] Initialize the target handover module target library;

[0088] Preset target discovery algorithm YoloV3;

[0089] Preset target handover algorithm Fast-Reid;

[0090] Preset single target tracking algorithm SiamRPN;

[0091] The above algorithm is accelerated by GPU;

[0092] Step S3: The drone executes the target discovery module:

[0093] (1) Finite state machine target state {NULL}, real-time image input YoloV3 target discovery algorithm, search for the target;

[0094] (2) If the target is found, the state machine target state Q = {T1: DETECTED, T2: DETECTED}, driving the long-term target tracking unit to enter the target handover module;

[0095] (2) If a target is found, the pixel coordinates of the target are offset and the image of the area near the target is retained for target handover;

[0096] (3) If the target is not found, the target state of the finite state machine remains {NULL}, and the target discovery module continues to execute.

[0097] Step S4: Figure 5 As shown, the drone executes the target handover module:

[0098] (1) The DETECTED target in Q is matched with the target in the target library, and the matching algorithm is a preset Fast-Reid target handover algorithm;

[0099] (2) If the target handover is successful, the state machine target state Q = {T1: HANDOVER, T2: EDTECTED}, and the drive enters the target tracking module;

[0100] (3) The above target handover process uses a matching algorithm based on target image cutout;

[0101] (4) If the target handover fails, return to step S3 to continue executing the target discovery module;

[0102] Step S5: the drone executes the target tracking module;

[0103] (1) The HANDOVER state target in Q is used for target tracking, and the algorithm is the SiamRPN target tracking algorithm;

[0104] (2) If the tracking is successful, the state machine target state is marked as Q = {T1: LOCKED, T2: DETECTED}, and the target tracking phase is entered to continue tracking the target;

[0105] (3) The target pixel coordinates are sent to the pod servo command to lock the target in the center of the field of view;

[0106] (4) If target tracking fails, the finite state machine target state Q = {T1:LOST, T2:DETECTED}, then return to step S3 to continue executing the target discovery module;

[0107] S6: Repeat steps S1-S5 until the task is completed.

[0108] Embodiment 4:

[0109] Based on the same invention concept, the present invention patent application also provides a UAV visual target tracking method based on a finite state machine, the flow chart is as follows Figure 6 As shown:

[0110] Step 1: Collect visual images and decode the visual images to obtain video stream information;

[0111] Step 2: According to the video stream information, obtain the state information of the target in the video stream information, and trigger an event corresponding to the state information;

[0112] Step 3: According to the event corresponding to the status information, execute the action corresponding to the event.

[0113] Specifically, the status information in step 2 includes: target discovery, target handover, target tracking, and target loss.

[0114] The obtaining of the state information of the target in the video stream information and triggering an event corresponding to the state information includes:

[0115] When the state information of the target in the acquired video stream information is target discovery, triggering a target handover event;

[0116] When the state information of the target in the acquired video stream information is target handover, triggering a target tracking event;

[0117] When the state information of the target in the acquired video stream information is that the target is lost, a target discovery event is triggered.

[0118] After decoding the visual image to obtain the video stream information, the method further includes:

[0119] Obtaining pixel coordinates of a target in the video stream information;

[0120] Generate a pixel offset according to the pixel coordinates of the target;

[0121] According to the pixel offset, a corresponding servo action is performed.

[0122] It should be understood by those skilled in the art that the embodiments of the present invention patent application can be provided as methods, systems, or computer program products. Therefore, the present invention patent application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention patent application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0123] The present invention patent application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention patent application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the patent application of the present invention rather than to limit its protection scope. Although the patent application of the present invention is described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the patent application of the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims

1. A UAV visual target tracking system based on a finite state machine, characterized in that: include: An unmanned aerial vehicle platform and a computing board, wherein the unmanned aerial vehicle platform is equipped with an optoelectronic pod; The computing board is equipped with a finite state machine and a long-term target tracking unit; The finite state machine is respectively connected to the UAV platform and the long-term target tracking unit for communication; The optoelectronic pod is used to collect visual images and decode the visual images to obtain video stream information; It is also used to receive pixel coordinate offset instructions of the target and lock the target in the center of the field of view; The finite state machine is used to obtain the state information of the target in the video stream information according to the video stream information, and trigger an event corresponding to the state information; The long-term target tracking unit is used to receive an event corresponding to the state information and execute an action corresponding to the event; The state information in the finite state machine includes: target discovery, target handover, target tracking and target loss; The step of obtaining the state information of the target in the video stream information and triggering an event corresponding to the state information in the finite state machine includes: When the state information of the target in the acquired video stream information is target discovery, a target handover event is triggered, and if the target handover is successful, the finite state machine switches to a target handover state; When the state information of the target in the acquired video stream information is target handover, a target tracking event is triggered, and if the target tracking is successful, the finite state machine switches to the target tracking state; When the state information of the target in the acquired video stream information is that the target is lost, a target discovery event is triggered, and if the target is found, the finite state machine switches to a target discovery state; When the state information of the target in the acquired video stream information is target tracking, a target tracking event is triggered, and if the target tracking is lost, the finite state machine switches to a target lost state; The long-term target tracking unit receives a target handover event and executes an action corresponding to the target handover event, including: When the state information of the target in the acquired video stream information is target discovery, the target is matched with the stored target in the target library through a preset Fast-Reid deep learning matching algorithm; if the target is successfully matched with the stored target in the target library, the target handover is determined to be successful, and target tracking is performed; if the target is not matched with the stored target in the target library, the target handover is determined to be failed, and target discovery is performed; The long-term target tracking unit receives a target tracking event and executes an action corresponding to the target tracking event by adopting a deep learning tracking algorithm; The long-term target tracking unit receives a target discovery event and executes an action corresponding to the target discovery event by adopting a deep learning detection algorithm; The Fast-Reid deep learning matching algorithm, the deep learning tracking algorithm and the deep learning detection algorithm are accelerated by an NPU, a GPU or an XPU accelerator card; The finite state machine is further used to obtain the pixel coordinates of the target and generate a pixel offset according to the pixel coordinates of the target; The UAV platform is also used to receive the pixel offset and perform a corresponding servo action according to the pixel offset.

2. The system according to claim 1, characterized in that The UAV platform and the finite state machine are communicatively connected via an asynchronous serial port.

3. A method for tracking a UAV visual target based on a finite state machine, characterized in that: include: Collecting visual images and decoding the visual images to obtain video stream information; According to the video stream information, obtaining the state information of the target in the video stream information, and triggering an event corresponding to the state information; According to the event corresponding to the status information, executing the action corresponding to the event; The status information includes: target discovery, target handover, target tracking and target loss; The obtaining of the state information of the target in the video stream information and triggering an event corresponding to the state information includes: When the state information of the target in the acquired video stream information is target discovery, a target handover event is triggered, and if the target handover is successful, the finite state machine switches to a target handover state; When the state information of the target in the acquired video stream information is target handover, a target tracking event is triggered, and if the target tracking is successful, the finite state machine switches to the target tracking state; When the state information of the target in the acquired video stream information is that the target is lost, a target discovery event is triggered, and if the target is found, the finite state machine switches to a target discovery state; When the state information of the target in the acquired video stream information is target tracking, a target tracking event is triggered, and if the target tracking is lost, the finite state machine switches to a target lost state; Wherein, according to the event corresponding to the target handover, executing the action corresponding to the event includes: When the state information of the target in the acquired video stream information is target discovery, the target is matched with the stored target in the target library by using a preset Fast-Reid deep learning matching algorithm; if the target is successfully matched with the stored target in the target library, the target handover is determined to be successful, and target tracking is performed; if the target is not matched with the stored target in the target library, the target handover is determined to be failed, and target discovery is performed; According to the event corresponding to the target discovery, executing an action corresponding to the event includes: according to the event corresponding to the target discovery, using a deep learning detection algorithm to execute an action corresponding to the target discovery event; According to the event corresponding to the target tracking, executing an action corresponding to the event, including: according to the event corresponding to the target tracking, using a deep learning tracking algorithm to execute the action corresponding to the target tracking event; The Fast-Reid deep learning matching algorithm, the deep learning detection algorithm and the deep learning tracking algorithm are accelerated by an NPU, a GPU or an XPU accelerator card; After decoding the visual image to obtain the video stream information, the method further includes: Obtaining pixel coordinates of a target in the video stream information; Generate a pixel offset according to the pixel coordinates of the target; According to the pixel offset, a corresponding servo action is performed.

Images