Multi-mode interactive indoor unmanned aerial vehicle target detection and tracking method and system

Abstract

The invention relates to a multi-mode interactive indoor unmanned aerial vehicle target detection and tracking method and system. The system comprises a target detection and tracking module, a laser radar inertial odometer module, a video display and interaction module and a flight control module. The method comprises the steps that an unmanned aerial vehicle end receives and processes RGB images, laser radar point cloud and IMU data; and optimizing a YOLOv8 target detection model and an OSTrack target tracking model through a TensorRT inference engine. By fusing LiDAR and IMU data, accurate state estimation and map construction are provided. And the ground station end displays a video stream and generates a control command through the video display and interaction module. And calculating a flight speed and a yaw angle rate by estimating an error between the target image position and the current position of the unmanned aerial vehicle, and generating a flight control instruction. The problems that an indoor unmanned aerial vehicle system often faces low target detection precision, poor tracking stability and difficult user interaction experience are solved, and the method and the system are more efficient, more accurate and higher in interactivity.

Description

Technical Field

[0001] This patent relates to the field of indoor drone technology, and in particular to a multi-mode interactive indoor drone target detection and tracking method and system. Background Art

[0002] Drones have strong flexibility and maneuverability, and can perform tasks in complex indoor environments to search and track specific targets.

[0003] In its patent application, “An Indoor UAV Target Tracking System Based on Depth Vision” (patent application number: CN202311858077.8, publication number CN117745765A), the Beijing University of Aeronautics and Astronautics proposed an indoor UAV target tracking system based on depth vision. This method uses a visual-inertial SLAM algorithm module, a target tracking algorithm module, and a UAV path planning algorithm module to realize the target tracking, flight following and other functions of the UAV in an indoor environment without GNSS (Global Navigation Satellite System). However, this method uses a visual-inertial SLAM algorithm, which is prone to poor positioning when faced with challenges such as lighting changes and texture loss. In addition, this method tracks directly based on the target detection results, lacks interactive target selection and tracking functions with the user, and limits the user's flexible control over the UAV tracking behavior.

[0004] Shijiazhuang Tiedao University proposed an embedded-based aerial photography multi-target tracking algorithm and system in its patent application document "An Embedded-Based Aerial Photography Multi-Target Tracking Algorithm and System" (patent application number: CN202411570502.8, publication number CN119360248A). The method includes a remote-end and a ground-end system, where the remote-end system consists of image acquisition and preprocessing, target detection, and target tracking modules. The image acquisition module acquires and preprocesses images, the target detection module outputs target coordinates, and the target tracking module tracks the target and assigns an ID. The remote end transmits the video stream to the ground end via the RTSP protocol. The ground end parses and displays the video stream and can control the module parameters of the remote end. However, this method is mainly aimed at aerial photography scenarios and cannot adapt to complex indoor environments. It also lacks closed-loop feedback for automated flight control.

[0005] In view of the defects in the above technologies, the technical problems to be solved by the present invention are embodied in the following points:

[0006] 1) The indoor environment in which drones fly is complex and changeable. Existing methods cannot extract sufficient visual features of targets when indoor lighting changes or there are obstacles blocking them, resulting in low robustness and accuracy of target detection and tracking.

[0007] 2) Existing methods can only track based on target detection results and lack interactive target selection and tracking functions with users. Therefore, it is impossible to achieve real-time interaction between the user and the drone tracking behavior.

[0008] 3) The indoor satellite positioning signal is weak and the space is small in the indoor environment where drones fly. The existing drone autonomous flight algorithms rely on satellite positioning signals, and the satellite signal positioning accuracy is low, resulting in the inability of drones to accurately locate themselves when flying indoors, which easily leads to the risk of crashes.

[0009] Since indoor environments usually contain complex obstacles, changing lighting conditions and confined spaces, existing indoor drone systems often face problems such as low target detection accuracy, poor tracking stability and difficult user interaction experience. Summary of the Invention

[0010] In view of the problems that existing indoor drone systems often face, such as low target detection accuracy, poor tracking stability and difficult user interaction experience, a multi-modal interactive indoor drone target detection and tracking method and system is proposed.

[0011] The technical solution of the present invention is:

[0012] A multi-modal interactive indoor drone target detection and tracking method includes the following steps:

[0013] Step 1: Data collection and preprocessing

[0014] The drone collects camera RGB images, LiDAR point clouds, and IMU data, and performs corresponding preprocessing on different data types.

[0015] Step 2: Acquisition of target space information

[0016] Perform YOLOv8 target detection and OSTrack target tracking on the pre-processed RGB image, output target position and category information for flight control, and synchronize the visualization results to the ground station for display;

[0017] Step 3: State Estimation and Map Construction

[0018] Extract features from pre-processed LiDAR and IMU data and match them to the local map. Update state estimates based on the error-state extended Kalman filter (ESKF). Build and update the global map to provide real-time state information for flight control.

[0019] Step 4: Human-computer interaction and control feedback

[0020] The ground station receives and displays the video stream, processes user operations to generate control commands, and sends them to the drone to dynamically adjust the control strategy;

[0021] Step 5: Flight control and target tracking

[0022] The relative position error is calculated based on the state estimation and target position information, and the flight speed and yaw rate are generated and limited by the control algorithm. The control instructions are output to adjust the UAV attitude and achieve target tracking.

[0023] Furthermore, the specific implementation process of step 1 is as follows:

[0024] Step 1.1: Receive and preprocess the input image data: For the YOLOv8 model, first perform an affine transformation on the input image and scale it to 640x640. Then perform pixel normalization to scale the image pixel values ​​to the range [0,1]. For the OSTrack model, first perform cropping, padding, and scaling on the original image based on the current target box position to generate the corresponding template image and search area image. Then perform normalization on the image data to scale the image pixel values ​​to the range [0,1].

[0025] Step 1.2: Receive and preprocess input LiDAR and IMU data: Read raw data from the LiDAR and IMU sensors. LiDAR data typically contains point cloud coordinate information, while IMU data includes measurements from the accelerometer and gyroscope. Use statistical filters and voxel filters to denoise the initial point cloud. In the filter prediction phase, integrate the IMU data to obtain the IMU integral result and update the covariance of the system state. Integrate the angular velocity to obtain a preliminary estimate of the attitude, and integrate the acceleration to obtain preliminary estimates of the position and velocity.

[0026] Furthermore, the specific implementation process of step 2 is as follows:

[0027] Step 2.1: Load the target detection model and target tracking model: Build and train the YOLOv8 and OSTrack models, read the trained YOLOv8 and OSTrack model weights, and convert the pt format weight files to onnx format weight files based on the input size; create a builder through the TensorRT interface and add each layer to the network definition; configure the builder, including the input and output parameters of the network; call the ONNX model parser to parse the model from the ONNX file and build the YOLOv8 and OSTrack serialized engine files; load the TensorRT engine file, and the contents of the YOLOv8 and OSTrack serialized engine files are read into the buffer and deserialized in memory; create an inference engine and create an execution context based on the engine file; allocate GPU buffers for storing the input image data and output inference results of the YOLOv8 and OSTrack models;

[0028] Step 2.2: Perform object detection inference and parse the output: Copy the preprocessed image data to the GPU buffer and call the YOLOv8 model inference interface to perform forward propagation; copy the inference results back to the host buffer and release the device memory allocation; decode the output data and extract the coordinates, category, and confidence of the bounding box; filter the model output results and apply the non-maximum suppression algorithm to remove the target boxes with high overlap and low confidence, retaining the best detection results; map the detection box coordinates back to the original image resolution based on the scaling and padding parameters used during image preprocessing;

[0029] Step 2.3: Perform target tracking inference and parse the output: Generate a template image based on the YOLOv8 detection results or a custom initial target bounding box position and initialize the OSTrack tracker. Then copy the preprocessed search area image to the GPU buffer, call the OSTrack model inference interface, and perform forward propagation. Post-process the model output results, use the Hanning window to weight the score map, find the maximum score position, and calculate the target position. Based on the cropping, scaling, and padding parameters used in the image preprocessing, map the target bounding box coordinates back to the original image resolution. Dynamically update the target bounding box by combining the model prediction results and the target position in the previous frame.

[0030] Step 2.4: Transmit detection and tracking results: Encapsulate the detection and tracking results into a message format and send the results to the flight control module through ROS; draw detection boxes and tracking boxes on the original image to visualize the inference effect; configure FFmpeg with hardware codec acceleration and call FFmpeg's H.265 encoder to compress the visualization image; use the UDP protocol to send the compressed video stream data to the ground station; the ground station receives and decodes the video stream, displaying the video image with the detection and tracking results; receive instructions and adjust the tracking range.

[0031] Furthermore, the specific implementation process of step 3 is as follows:

[0032] Step 3.1: Feature extraction and matching: Treat all point clouds as planar feature points and dedistort the current point cloud based on the predicted state. Match the extracted feature points with the IMU integration results. First, synchronize the LiDAR point cloud data with the IMU data in time to ensure that their timestamps are consistent. Then, match the feature points with the local map represented by the incremental KD tree structure.

[0033] Step 3.2: State estimation and update: Initialize the state vector, including position, velocity, attitude, and IMU bias. In the filter update phase, combine the LiDAR feature point matching and IMU data from step 2 to construct a residual function. Update the state estimate by iteratively extending the filter through the error state until convergence is achieved.

[0034] Step 3.3: Map construction and update: Incrementally update the point cloud data of the current frame with the existing local map; in addition, fuse the registered point cloud data into the global map to ensure the consistency and integrity of the map; regularly perform closed-loop detection, that is, check whether the current frame is repeated with a previous frame to correct accumulated errors; update the map in a timely manner, including adding new point cloud data, deleting redundant data, and optimizing the map structure.

[0035] Furthermore, the specific implementation process of step 4 is as follows:

[0036] Step 4.1: Initialize the system: load the user configuration file in JSON format and obtain the operating parameters; initialize the Qt main window, load the UI layout file, and set the user interface layout; create a UDP server and TCP client based on the information in the configuration file; start the timer to periodically refresh the UI interface and network status;

[0037] Step 4.2: Receive and decode the video stream: The UDP server receives the encoded video data packets from the drone. It uses the FFmpeg library to decode the received data packets into H.265 format and stores the decoded data in OpenCV format. The image is rendered on the custom control for real-time display.

[0038] Step 4.3: User Interaction and Command Sending: Implement mouse click, move, and release events in the custom control to draw the area of ​​interest; generate corresponding control commands based on user selections; use the TCP protocol to send control commands to the drone end, notifying it to perform the corresponding operations.

[0039] Furthermore, the specific implementation process of step 5 is as follows:

[0040] Step 5.1: Get the target location: parse the target detection information and extract the target's feature parameters in the image;

[0041] Step 5.2: Calculate flight speed: Combine the rotation matrix between the camera and the aircraft to convert the image plane error into an error in the drone's body coordinate system. Based on the target's error in the body coordinate system, use the PID control algorithm to calculate the drone's forward, lateral, and vertical speeds. Limit the speed to ensure that the drone's speed does not exceed the maximum safety threshold. Based on the current drone's attitude information, convert the speed in the body coordinate system to the global coordinate system.

[0042] Step 5.3: Calculate the yaw rate: Calculate the left and right offset angles of the target and generate the yaw rate; limit the yaw rate;

[0043] Step 5.4: Output control instructions: The returned vector is used as the final control instruction and input into the flight control system to adjust the flight trajectory of the drone.

[0044] Furthermore, the control command in step 4 includes:

[0045] Click mode: Press the "click" button in the video display and interactive interface, and the ground station sends a "click" signal. After receiving the signal, the drone starts the target detection and inference program, analyzes and visualizes the inference results, and finally encodes the image and sends it back to the ground station. In the video display and interactive interface, click the target rectangle in the image, and the system background sends the target coordinate position to the drone, starting the target tracking and inference program.

[0046] Frame selection mode: Press the "Frame selection" button in the video display and interactive interface, and the ground station sends a "frame selection" signal to switch from click mode to frame selection mode; in the video display and interactive interface, frame the target in the image, and the system background sends the target coordinate position to the drone, starting the target tracking inference program;

[0047] Tracking mode: Click the "Tracking" button in the video display and interactive interface. The ground station sends a "tracking" signal to start tracking mode. After receiving the signal, the drone calculates the flight speed and yaw rate based on the target image position and the current drone position, outputs control commands, and begins to perform continuous and stable target tracking.

[0048] A multi-modal interactive indoor drone target detection and tracking system, including the following modules:

[0049] The target detection and tracking module receives raw image data from the camera and uses deep learning models to detect and track targets. First, it loads the target detection and tracking models and optimizes them using the TensorRT inference engine. Second, it uses the YOLOv8 model for target detection, extracts and optimizes the detection bounding box. Third, it uses the OSTrack model to track targets and dynamically update the target position. Finally, it sends the detection and tracking results to the drone's flight control module and the ground station's video display and interaction module.

[0050] LiDAR Inertial Odometry Module: This module provides accurate attitude estimation and map construction by fusing LiDAR and IMU data. It first receives data from the LiDAR and IMU sensors, performs noise reduction on the LiDAR point cloud data, and integrates the IMU data. It then matches feature points to the local map after time synchronization between the LiDAR and IMU data. It then updates the state estimate using an error state iterative extension filter by constructing a residual function. Finally, it uses the optimized state estimate to build and update the global map.

[0051] Video Display and Interaction Module: This module is responsible for receiving, decoding, and displaying video streams, and processing user input to generate control commands. First, it receives encoded video stream data from the drone, decodes and converts it into different formats, and renders it to the user interface for real-time video display. Second, it receives interactive signals input by the user through the mouse and generates corresponding control commands based on the user's operation type. Finally, it sends the generated control commands to the target detection and tracking module and flight control module on the drone via the network protocol, instructing them to perform the corresponding operations.

[0052] Flight control module: Generates flight control instructions based on target position information. First, it receives the target position information sent by the target detection module. If a target is detected, its center position and area ratio are calculated. Second, based on the error between the target and the current position of the UAV, it uses a control algorithm to calculate the required flight speed and yaw rate, and limits the speed to ensure smooth flight. Finally, according to the flight control instructions, the UAV's attitude is adjusted to track the target. If the target is not detected within the set time, the UAV enters standby mode.

[0053] The beneficial effects of the present invention are:

[0054] This system integrates a target detection and tracking module, a lidar and inertial odometry module, a video display and interaction module, and a flight control module. By combining deep learning, multi-sensor fusion, and optimized control strategies, it provides an efficient, accurate, and highly interactive indoor drone target detection and tracking system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of the indoor UAV target detection and tracking method of the present invention;

[0056] Figure 2 This is a flow chart of the target detection and tracking algorithm in the present invention;

[0057] Figure 3 This is a flow chart of the laser radar inertial odometer calculation method in the present invention;

[0058] Figure 4This is a schematic diagram of the interface of the video display and interaction module in the present invention;

[0059] Figure 5 This is a flow chart of the flight control algorithm in the present invention;

[0060] Figure 6 This is the architecture diagram of the indoor drone target detection and tracking system of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0062] A multi-modal interactive indoor drone target detection and tracking system, including the following modules:

[0063] Target Detection and Tracking Module: Receives camera images, uses the YOLOv8 model optimized by TensorRT (NVIDIA's deep learning model inference acceleration tool) to detect targets, and implements target tracking through the OSTrack model. It dynamically updates the target position and sends the results to the flight control module and ground station.

[0064] LiDAR Inertial Odometry Module: This module integrates LiDAR and IMU data to provide state estimation and map construction, optimizes state estimation through error state iterative extension filters, and updates the global map.

[0065] Video display and interaction module: decodes and displays the drone video stream, processes user input (click, drag, etc.) to generate control commands, and sends them to the drone end through the network protocol.

[0066] Flight control module: Receives target location information, generates flight instructions, adjusts the drone's attitude to track the target, and enters standby mode if the target is lost.

[0067] Please refer to Figure 1 As shown, Figure 1 This is a flow chart of a multi-modal interactive indoor drone target detection and tracking method of the present invention. It includes the following steps:

[0068] Step 1: First, the drone receives and processes RGB images, LiDAR point clouds, and IMU data. Next, the TensorRT inference engine optimizes the YOLOv8 object detection and OSTrack object tracking models. Third, by fusing LiDAR and IMU data, accurate state estimation and map construction are provided. Finally, by estimating the error between the target image position and the drone's current position, the flight speed and yaw rate are calculated, generating flight control commands.

[0069] Step 2: The ground station uses the video display and interaction module to display the video stream and generate control commands. Users trigger the corresponding background program by clicking, dragging, and releasing the mouse, realizing multi-mode interaction between the ground station and the drone.

[0070] Click Mode: Press the "Click" button in the video display and interactive interface. The ground station sends a "click" signal. Upon receiving the signal, the drone initiates the target detection and inference process, analyzes and visualizes the inference results, and finally encodes the image and transmits it back to the ground station. In the video display and interactive interface, click the target rectangle in the image. The system background sends the target coordinates to the drone, initiating the target tracking and inference process.

[0071] Frame Selection Mode: Press the "Frame Selection" button in the video display and interactive interface. The ground station sends a "Frame Selection" signal, switching from click mode to frame selection mode. In the video display and interactive interface, frame the target in the image. The system background sends the target coordinates to the drone, starting the target tracking inference program.

[0072] Tracking Mode: Click the "Track" button in the video display and interactive interface. The ground station sends a "tracking" signal, initiating tracking mode. Upon receiving the signal, the drone calculates the flight speed and yaw rate based on the target image position and the current drone position, outputs control commands, and begins continuous and stable target tracking.

[0073] Please refer to Figure 2 As shown, Figure 2 This is a flow chart of the target detection and tracking algorithm of the present invention. The target detection and tracking algorithm is responsible for receiving the raw image data sent by the camera and using the deep learning model to perform target detection and tracking. It includes the following steps:

[0074] Step 1: Receive and preprocess input image data;

[0075] Step 2: Load the YOLOv8 target detection model and OSTrack target tracking model;

[0076] Step 3: Perform object detection inference and parse the output results;

[0077] Step 4: Perform target tracking inference and parse the output results;

[0078] Step 5: Transmit the detection and tracking results through network protocols, etc.

[0079] The specific steps include:

[0080] Step 1: Receive and preprocess input image data: Capture raw image frames from the drone's camera and convert the image data into a format suitable for model input. For the YOLOv8 model, first perform an affine transformation on the input image, scaling it to 640x640. Then, perform pixel normalization to bring the image pixel values ​​into the [0, 1] range. For the OSTrack model, first crop, pad, and scale the raw image based on the current target bounding box position to generate the corresponding template image and search area image. Then, perform a normalization operation on the image data to bring the image pixel values ​​into the [0, 1] range.

[0081] Step 2: Load the Object Detection and Tracking Models: Train the YOLOv8 and OSTrack models, read the trained YOLOv8 and OSTrack model weights, and convert the pt-format weight files to onnx-format weight files based on the input size. Create a builder using the TensorRT interface and add layers to the network definition, including convolutional layers, activation layers, and pooling layers. Configure the builder, including parameters such as the network's input and output tensor sizes, maximum batch size, and computational precision. Call the ONNX model parser to parse the model from the ONNX file and construct the YOLOv8 and OSTrack serialized engine files. Load the TensorRT engine files, read the contents of the YOLOv8 and OSTrack serialized engine files into a buffer, and deserialize them in memory. Create an inference engine and create an execution context based on the engine files. Allocate GPU buffers to store the input image data and output inference results for the YOLOv8 and OSTrack models.

[0082] Step 3: Perform object detection inference and parse the output: Copy the preprocessed image data to a GPU buffer and call the YOLOv8 model inference interface to perform forward propagation. Copy the inference results back to the host buffer and release the device memory allocation. Decode the output data to extract the coordinates, category, and confidence of the bounding box. Filter the model output results and apply the non-maximum suppression algorithm to remove object boxes with high overlap and low confidence, retaining the best detection results. Based on the scaling and padding parameters used during image preprocessing, map the detection box coordinates back to the original image resolution.

[0083] Step 4: Perform object tracking inference and parse the output: Based on the YOLOv8 detection results or a custom initial object bounding box position, a template image is generated and the OSTrack tracker is initialized. The preprocessed search area image is then copied to the GPU buffer, and the OSTrack model inference interface is called to perform forward propagation. The model output is post-processed, and the score map is weighted using a Hanning window to find the location with the maximum score and calculate the object position. The object bounding box coordinates are mapped back to the original image resolution based on the cropping, scaling, and padding parameters used during image preprocessing. The object bounding box is dynamically updated by combining the model prediction results with the object position from the previous frame.

[0084] Step 5: Transmit detection and tracking results: Encapsulate the detection and tracking results into a message format and send them to the flight control module via ROS (Robot Operating System). Draw detection and tracking boxes on the original image to visualize the inference effect. Configure FFmpeg (an open source multimedia processing framework) that supports hardware codec acceleration and call FFmpeg's H.265 encoder to compress the visualization image, reducing transmission bandwidth requirements. Use the UDP protocol to send the compressed video stream data to the ground station. The ground station receives and decodes the video stream, displaying the video image with the detection and tracking results. Receive instructions such as the region of interest and adjust the tracking range to achieve multi-modal interactive tracking control.

[0085] Please refer to Figure 3 As shown, Figure 3 This is a flow chart of the LiDAR inertial odometry calculation method of the present invention. The LiDAR inertial odometry calculation method provides accurate state (including position, velocity, and attitude) estimation and map construction by fusing LiDAR and IMU data. It includes the following steps:

[0086] Step 1: Data input and preprocessing: Read and process LiDAR and IMU data;

[0087] Step 2: Feature extraction and matching: Match feature points with local maps;

[0088] Step 3: State estimation and update: Use the error state to iteratively expand the filter to update the state estimate;

[0089] Step 4: Map construction and update: The optimized state estimation results are used to build and update the global map.

[0090] The specific steps include:

[0091] Step 1: Data Input and Preprocessing: Raw data is read from the LiDAR and IMU sensors. LiDAR data typically contains point cloud coordinate information, while IMU data includes accelerometer and gyroscope measurements. Statistical and voxel filters are used to reduce noise on the initial point cloud. During the filter prediction phase, the IMU data is integrated to obtain the IMU integral result and update the system state covariance. An initial estimate of attitude is obtained by integrating the angular velocity, and an initial estimate of position and velocity is obtained by integrating the acceleration.

[0092] Step 2: Feature Extraction and Matching: All point clouds are treated as planar feature points, and the current point cloud is dedistorted based on the predicted state. The extracted feature points are matched with the IMU integration results. First, the LiDAR point cloud data and IMU data are synchronized in time to ensure that their timestamps are consistent. Then, the feature points are matched with the local map (represented by an incremental KD tree structure).

[0093] Step 3: State Estimation and Update: Initialize the state vector, which typically includes position, velocity, attitude, and IMU bias. In the filter update phase, the residual function is constructed by combining the LiDAR feature point matching and IMU data from step 2. The state estimate is updated through the Error State Iterative Extension Kernel Filter (ESIEKF) until convergence is achieved.

[0094] Step 4: Map Construction and Update: Incrementally update the existing local map with the point cloud data of the current frame. Furthermore, the registered point cloud data is integrated into the global map to ensure map consistency and integrity. Regular loop closure checks are performed to check whether the current frame overlaps with a previous frame to correct for accumulated errors. Map updates are performed promptly, including adding new point cloud data, removing redundant data, and optimizing the map structure.

[0095] Please refer to Figure 4 As shown, Figure 4 This is a schematic diagram of the video display and interaction module interface of the present invention. The video display and interaction algorithm is responsible for receiving, decoding, and displaying video streams, and processing user input to generate control commands. It includes the following steps:

[0096] Step 1: Initialize the system: load the configuration file, create the main window and initialize the network module;

[0097] Step 2: Display video stream: Receive and decode the video stream and display it on the user interface;

[0098] Step 3: User interaction and command sending: Use the mouse to select the area of ​​interest and send control commands.

[0099] The specific steps include:

[0100] Step 1: Initialize the system: Load the user configuration file in JSON format and obtain the IP address, port, and other operating parameters. Initialize the Qt main window, load the UI layout file, and set the user interface layout, including buttons, labels, etc. Based on the information in the configuration file, create a UDP server and TCP client. Start a timer to periodically refresh the UI and network status.

[0101] Step 2: Receive and decode the video stream: The UDP server receives the encoded video data packets from the drone. It decodes the received data packets into H.265 using the FFmpeg library and stores the decoded data in OpenCV format. The image is rendered on a custom control for real-time display.

[0102] Step 3: User Interaction and Command Transmission: Implement mouse click, move, and release events in the custom control to draw a region of interest. Generate corresponding control commands based on the user's selected region of interest or other operations (such as "click," "box selection," and "tracking"). Use TCP to send these control commands to the drone, instructing it to execute the corresponding operation.

[0103] Please refer to Figure 5 As shown, Figure 5 This is a flow chart of the flight control algorithm of the present invention. Flight control algorithm: The drone adjusts its flight trajectory in real time based on the position of the target in the camera to achieve stable tracking of the target. It includes the following steps:

[0104] Step 1: Obtain target position: Analyze target detection information and extract target coordinate position and area ratio feature parameters;

[0105] Step 2: Calculate flight speed: Based on the position error, the flight control algorithm is used to calculate the flight speed of the drone, and the limited speed is converted to the speed in the global coordinate system.

[0106] Step 3: Calculate the yaw rate: Calculate the left and right offset angles of the target, generate the yaw rate, and limit the yaw rate.

[0107] Step 4: Output control instructions: The returned vector is used as the final control instruction and input into the flight control system to adjust the flight trajectory of the drone.

[0108] The specific steps include:

[0109] Step 1: Obtain target location: Analyze target detection information and extract characteristic parameters such as the target's center point coordinates and area ratio in the image.

[0110] Step 2: Calculate flight velocity: The image plane error is converted to the drone's body coordinate system using the rotation matrix between the camera and the aircraft. Based on the target's body coordinate system error, a PID control algorithm is used to calculate the drone's forward, lateral, and vertical velocities. The velocity is capped to ensure it does not exceed the maximum safety threshold. Based on the current drone attitude information, the velocity in the body coordinate system is converted to the global coordinate system.

[0111] Step 3: Calculate the yaw rate: Calculate the left and right offset angles of the target and generate the yaw rate. Limit the yaw rate to avoid excessive rotation and instability.

[0112] Step 4: Output control instructions: The returned vector is used as the final control instruction and input into the flight control system to adjust the flight trajectory of the drone.

[0113] Please refer to Figure 6 As shown, Figure 6 This is a diagram of a multi-modal interactive indoor drone target detection and tracking system of the present invention. It specifically includes:

[0114] The Object Detection and Tracking module receives raw image data from the camera and uses deep learning models to detect and track objects. First, it loads the object detection and tracking models and optimizes them using the TensorRT inference engine. Next, it uses the YOLOv8 model to detect objects, extracting and optimizing the bounding boxes. Finally, it uses the OSTrack model to track objects and dynamically update their positions. Finally, the detection and tracking results are sent to the drone's flight control module and the ground station's video display and interaction module.

[0115] LiDAR / Inertial Odometry Module: This module provides accurate pose estimation and map construction by fusing LiDAR and IMU data. First, it receives data from the LiDAR and IMU sensors, performs noise reduction on the LiDAR point cloud data, and integrates the IMU data. Second, after time synchronization between the LiDAR and IMU data, it matches feature points to the local map. Third, it constructs a residual function and uses an error state iterative extension filter to update the state estimate. Finally, it uses the optimized state estimate to build and update the global map.

[0116] Video Display and Interaction Module: This module is responsible for receiving, decoding, and displaying video streams, as well as processing user input to generate control commands. First, it receives the encoded video stream data sent by the drone, decodes and converts it into a different format, and renders it to the user interface for real-time video display. Second, it receives interactive signals input by the user via the mouse, including clicks and drags, determines the region of interest, and generates corresponding control commands based on the user's operation. Finally, it sends the generated control commands via network protocols to the drone's target detection and tracking module and flight control module, instructing them to perform the corresponding operations.

[0117] The Flight Control Module generates flight control commands based on target location information. First, it receives target location information from the Target Detection Module. If a target is detected, its center position and area percentage are calculated. Next, based on the error between the target and the drone's current position, a control algorithm is used to calculate the required flight speed and yaw rate. The speed is then limited to ensure smooth flight. Finally, according to the flight control commands, the drone's attitude is adjusted to track the target. If no target is detected within a set time, the drone enters standby mode.

[0118] The above-described embodiment merely represents one embodiment of the present invention. While the description is relatively specific and detailed, it should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A multi-modal interactive indoor drone target detection and tracking method, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing The drone collects camera RGB images, LiDAR point clouds, and IMU data, and performs corresponding preprocessing on different data types. Step 2: Acquisition of target space information Perform YOLOv8 target detection and OSTrack target tracking on the pre-processed RGB image, output target position and category information for flight control, and synchronize the visualization results to the ground station for display; Step 3: State Estimation and Map Construction Extract features from pre-processed LiDAR and IMU data and match them to the local map. Update state estimates based on the error-state extended Kalman filter (ESKF). Build and update the global map to provide real-time state information for flight control. Step 4: Human-computer interaction and control feedback The ground station receives and displays the video stream, processes user operations to generate control commands, and sends them to the drone to dynamically adjust the control strategy; Step 5: Flight control and target tracking The relative position error is calculated based on the state estimation and target position information, and the flight speed and yaw rate are generated and limited by the control algorithm. The control instructions are output to adjust the UAV attitude and achieve target tracking.

2. The multi-modal interactive indoor drone target detection and tracking method according to claim 1 is characterized in that: The specific implementation process of step 1 is as follows: Step 1.1: Receive and preprocess the input image data: For the YOLOv8 model, first perform an affine transformation on the input image and scale it to 640x640. Then perform pixel normalization to scale the image pixel values ​​to the range [0,1]. For the OSTrack model, first perform cropping, padding, and scaling on the original image based on the current target box position to generate the corresponding template image and search area image. Then perform normalization on the image data to scale the image pixel values ​​to the range [0,1]. Step 1.2: Receive and preprocess input LiDAR and IMU data: Read raw data from the LiDAR and IMU sensors. LiDAR data typically contains point cloud coordinate information, while IMU data includes measurements from the accelerometer and gyroscope. Use statistical filters and voxel filters to denoise the initial point cloud. In the filter prediction phase, integrate the IMU data to obtain the IMU integral result and update the covariance of the system state. Integrate the angular velocity to obtain a preliminary estimate of the attitude, and integrate the acceleration to obtain preliminary estimates of the position and velocity.

3. The multi-modal interactive indoor drone target detection and tracking method according to claim 1, characterized in that: The specific implementation process of step 2 is as follows: Step 2.1: Load the object detection model and the object tracking model: Build and train the YOLOv8 and OSTrack models, read the trained YOLOv8 and OSTrack model weights, and convert the pt format weight files to onnx format weight files based on the input size; Create a builder through the TensorRT interface and add each layer to the network definition; configure the builder, including the input and output parameters of the network; call the ONNX model parser to parse the model from the ONNX file and build the YOLOv8 and OSTrack serialization engine files; The TensorRT engine file is loaded. The contents of the YOLOv8 and OSTrack serialized engine files are read into a buffer and deserialized in memory. An inference engine is created and an execution context is created based on the engine file. A GPU buffer is allocated to store the input image data and output inference results of the YOLOv8 and OSTrack models. Step 2.2: Perform object detection inference and parse the output: Copy the preprocessed image data to the GPU buffer and call the YOLOv8 model inference interface to perform forward propagation; copy the inference results back to the host buffer and release the device memory allocation; decode the output data and extract the coordinates, category, and confidence of the bounding box; filter the model output results and apply the non-maximum suppression algorithm to remove the target boxes with high overlap and low confidence, retaining the best detection results; map the detection box coordinates back to the original image resolution based on the scaling and padding parameters used during image preprocessing; Step 2.3: Perform target tracking inference and parse the output: Generate a template image based on the YOLOv8 detection results or a custom initial target bounding box position and initialize the OSTrack tracker. Then copy the preprocessed search area image to the GPU buffer, call the OSTrack model inference interface, and perform forward propagation. Post-process the model output results, use the Hanning window to weight the score map, find the maximum score position, and calculate the target position. Based on the cropping, scaling, and padding parameters used in the image preprocessing, map the target bounding box coordinates back to the original image resolution. Dynamically update the target bounding box by combining the model prediction results and the target position in the previous frame. Step 2.4: Transmit detection and tracking results: Encapsulate the detection and tracking results into a message format and send them to the flight control module via ROS. Draw detection and tracking boxes on the original image to visualize the inference results. Configure FFmpeg with hardware codec acceleration and use FFmpeg's H.265 encoder to compress the visualization image. Use UDP to send the compressed video stream data to the ground station. The ground station receives and decodes the video stream, displaying the video image with the detection and tracking results. Receive instructions and adjust tracking range.

4. The multi-modal interactive indoor drone target detection and tracking method according to claim 1, characterized in that: The specific implementation process of step 3 is as follows: Step 3.1: Feature extraction and matching: Treat all point clouds as planar feature points and dedistort the current point cloud based on the predicted state. Match the extracted feature points with the IMU integration results. First, synchronize the LiDAR point cloud data with the IMU data in time to ensure that their timestamps are consistent. Then, match the feature points with the local map represented by the incremental KD tree structure. Step 3.2: State estimation and update: Initialize the state vector, including position, velocity, attitude, and IMU bias. In the filter update phase, combine the LiDAR feature point matching and IMU data from step 2 to construct a residual function. Update the state estimate by iteratively extending the filter through the error state until convergence is achieved. Step 3.3: Map construction and update: Incrementally update the point cloud data of the current frame with the existing local map. In addition, fuse the registered point cloud data into the global map to ensure map consistency and integrity. Regularly perform closed-loop detection to check whether the current frame overlaps with a previous frame to correct accumulated errors. Update maps in a timely manner, including adding new point cloud data, deleting redundant data, and optimizing map structure.

5. The multi-modal interactive indoor drone target detection and tracking method according to claim 1, characterized in that: The specific implementation process of step 4 is as follows: Step 4.1: Initialize the system: load the user configuration file in JSON format and obtain the operating parameters; initialize the Qt main window, load the UI layout file, and set the user interface layout; create a UDP server and TCP client based on the information in the configuration file; start the timer to periodically refresh the UI interface and network status; Step 4.2: Receive and decode the video stream: The UDP server receives the encoded video data packets from the drone. It uses the FFmpeg library to decode the received data packets into H.265 format and stores the decoded data in OpenCV format. The image is rendered on the custom control for real-time display. Step 4.3: User interaction and command sending: Implement mouse click, move, and release events in the custom control, draw the area of ​​interest; generate corresponding control commands based on user selection; Use the TCP protocol to send control commands to the drone to notify it to perform corresponding operations.

6. The multi-modal interactive indoor drone target detection and tracking method according to claim 1, characterized in that: The specific implementation process of step 5 is as follows: Step 5.1: Get the target location: parse the target detection information and extract the target's feature parameters in the image; Step 5.2: Calculate flight speed: Combine the rotation matrix between the camera and the aircraft to convert the image plane error into an error in the drone's body coordinate system. Based on the target's error in the body coordinate system, use the PID control algorithm to calculate the drone's forward, lateral, and vertical speeds. Limit the speed to ensure that the drone's speed does not exceed the maximum safety threshold. Based on the current drone's attitude information, convert the speed in the body coordinate system to the global coordinate system. Step 5.3: Calculate the yaw rate: Calculate the left and right offset angles of the target and generate the yaw rate; limit the yaw rate; Step 5.4: Output control instructions: The returned vector is used as the final control instruction and input into the flight control system to adjust the flight trajectory of the drone.

7. The multi-modal interactive indoor drone target detection and tracking method according to claim 1, characterized in that: The control commands in step 4 include: Click mode: Press the "click" button in the video display and interactive interface, and the ground station sends a "click" signal. After receiving the signal, the drone starts the target detection and inference program, analyzes and visualizes the inference results, and finally encodes the image and sends it back to the ground station. In the video display and interactive interface, click the target rectangle in the image, and the system background sends the target coordinate position to the drone, starting the target tracking and inference program. Frame selection mode: Press the "Frame selection" button in the video display and interactive interface. The ground station sends a "frame selection" signal, switching from click mode to frame selection mode. In the video display and interactive interface, frame the target in the image. The system background sends the target coordinate position to the drone, starting the target tracking inference program. Tracking mode: Click the "Tracking" button in the video display and interactive interface. The ground station sends a "Tracking" signal to start tracking mode. After receiving the signal, the drone calculates the flight speed and yaw rate based on the target image position and the current drone position, outputs control commands, and begins continuous and stable target tracking.

8. A multi-modal interactive indoor drone target detection and tracking system, characterized by: Includes the following modules: The target detection and tracking module receives raw image data from the camera and uses deep learning models to detect and track targets. First, it loads the target detection and tracking models and optimizes them using the TensorRT inference engine. Second, it uses the YOLOv8 model for target detection, extracts and optimizes the detection bounding box. Third, it uses the OSTrack model to track targets and dynamically update the target position. Finally, it sends the detection and tracking results to the drone's flight control module and the ground station's video display and interaction module. LiDAR Inertial Odometry Module: This module provides accurate attitude estimation and map construction by fusing LiDAR and IMU data. It first receives data from the LiDAR and IMU sensors, performs noise reduction on the LiDAR point cloud data, and integrates the IMU data. It then matches feature points to the local map after time synchronization between the LiDAR and IMU data. It then updates the state estimate using an error state iterative extension filter by constructing a residual function. Finally, it uses the optimized state estimate to build and update the global map. Video display and interaction module: responsible for receiving, decoding and displaying video streams, and processing user input to generate control commands; First, it receives the encoded video stream data sent by the drone, decodes and converts its format, and renders it to the user interface to achieve real-time video display. Second, it receives interactive signals input by the user through the mouse and generates corresponding control commands based on the user operation type. Finally, it sends the generated control commands to the target detection and tracking module and flight control module on the drone through the network protocol, instructing them to perform the corresponding operations. Flight control module: Generates flight control instructions based on target location information. First, it receives target location information from the target detection module. If a target is detected, it calculates its center position and area ratio. Secondly, based on the error between the target and the current position of the UAV, a control algorithm is used to calculate the required flight speed and yaw rate, and the speed is limited to ensure smooth flight. Finally, according to the flight control instructions, the UAV's attitude is adjusted to track the target. If the target is not detected within the set time, the UAV enters standby mode.

Images