A method and system for indoor environmental positioning, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms

Abstract

This invention discloses a method and system for indoor environment localization, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms, comprising: multi-UAV collaborative deployment based on SLAM; running multi-UAV collaborative perception and mapping nodes on multiple UAV devices, with each UAV device independently completing mapping, localization, and recognition tasks; performing initial SLAM synchronization and Rviz visualization operations on multiple UAV devices; sharing the location of each UAV device across device scenes; calculating the coordinate transformation relationship between scenes for each UAV device and displaying the position of each UAV device in real time in the global scene; performing target detection while the UAV devices are moving, with multiple UAV devices performing multi-source target fusion perception and visualization; and updating the global scene and target data when multiple UAV devices meet again during their movement. This invention allows multiple UAVs to share mapping data, significantly improving the autonomous localization and target recognition capabilities of UAVs in complex indoor environments.

Description

Technical Field

[0001] This invention relates to the field of SLAM (Simultaneous Localization and Mapping) and target recognition technology, and particularly to a method and system for indoor environmental localization, mapping and target recognition of unmanned aerial vehicle (UAV) swarms. Background Technology

[0002] Currently, drones are increasingly used in indoor environments, such as disaster search and rescue, security patrols, and warehouse management. However, due to the lack of GPS signals indoors, traditional GNSS-based positioning methods are difficult to apply. Furthermore, complex indoor scenarios (such as narrow passages and obstructed views) pose challenges to drone navigation and target recognition. Therefore, how to achieve efficient and accurate drone swarm positioning, mapping, and target recognition in indoor environments is a key research focus.

[0003] In existing technologies, VIO (Visual Inertial Odometry) or SLAM (Simultaneous Localization and Mapping) techniques are commonly used for single-drone positioning. However, in multi-drone collaborative environments, data synchronization, information sharing, and swarm intelligent decision-making capabilities remain insufficient. Furthermore, target recognition accuracy is limited by the sensor's local field of view, leading to unstable recognition results. Therefore, there is an urgent need for a drone swarm system capable of accurate positioning, real-time mapping, and target recognition.

[0004] Currently, multi-sensor fusion SLAM technology has been widely studied in the field of indoor robot navigation, with LiDAR SLAM and Visual SLAM (V-SLAM) being the mainstream methods. LiDAR SLAM has high environmental perception capabilities and is suitable for scenarios with large lighting changes, but the equipment cost is high, and there is a certain error accumulation problem in dynamic environments. V-SLAM relies on cameras to obtain environmental feature information, with lower computational cost, but the positioning accuracy decreases in weak texture environments or under insufficient lighting. In recent years, multi-sensor SLAM schemes that integrate IMU, LiDAR, and RGB-D cameras have gradually become mainstream, but their synchronous processing, data sharing, and real-time optimization in multi-drone swarms still face challenges.

[0005] Furthermore, during UAV swarm missions, communication latency, limited information transmission bandwidth, and balanced distribution of computational load are pressing issues that need to be addressed. Most existing methods are based on centralized control architectures, which are prone to system crashes due to single points of failure. While decentralized architectures improve system robustness, they still face technical bottlenecks in information consistency and global optimization. How to efficiently allocate computing resources in UAV swarms to achieve real-time collaborative mapping and autonomous decision-making remains a research direction that urgently needs breakthroughs.

[0006] On the other hand, the application of target recognition in indoor environments also faces challenges. Due to the complex and ever-changing environment, targets may be occluded, and changes in lighting conditions can affect recognition accuracy. Furthermore, deep learning-based target detection methods struggle to achieve efficient real-time inference under the limited computing resources of drones. Therefore, how to combine multi-drone collaborative mechanisms to improve the accuracy and real-time performance of target recognition remains an important research direction in this field. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method and system for indoor environmental localization, mapping, and target recognition of UAV swarms, solving the problems of inaccurate UAV swarm localization, low mapping efficiency, and limited target recognition capabilities in existing technologies for indoor environments. This invention achieves high-precision localization, environmental mapping, and target recognition of UAV swarms based on multi-sensor fusion (IMU, RGB-D), distributed SLAM algorithms, and swarm collaboration mechanisms. It employs an advanced multimodal data fusion method to jointly optimize the inertial information from the IMU and the depth data from the RGB-D camera, improving the accuracy and stability of single-UAV localization. Through a decentralized distributed SLAM architecture, multiple UAVs can share mapping data, improving the reliability of mapping.

[0008] In a first aspect, embodiments of the present invention provide a method for indoor environment positioning, mapping, and target identification of unmanned aerial vehicle (UAV) swarms, comprising:

[0009] The S100 is based on SLAM for multi-drone collaborative deployment, configuring data communication between multiple drone devices and modifying the camera configuration file of the SLAM algorithm.

[0010] S200: Multi-machine collaborative perception and mapping nodes are run on multiple UAV devices respectively, enabling each UAV device to independently complete mapping, positioning and identification tasks.

[0011] S300 performs initial SLAM synchronization and Rviz visualization operations on multiple drone devices.

[0012] S400, each of the aforementioned drone devices uses the Rviz visualization tool to share locations and collaboratively perceive scenarios across devices.

[0013] S500, calculate the coordinate transformation relationship between the scenes of each of the drone devices, unify the local scenes and poses constructed by each of the drone devices into the coordinate system of the global scene, and display the position of each of the drone devices in real time in the global scene.

[0014] S600, target detection is performed when the UAV equipment is moving, and multiple UAV equipment perform multi-source target fusion perception and visualization.

[0015] S700: When multiple drone devices meet again during their advance, they re-share data, update the global scene and target data, until the mission is completed and they exit.

[0016] In conjunction with the first aspect, the present invention provides a first possible implementation of the first aspect, wherein, in S100, the data communication between the configured multiple UAV devices includes:

[0017] The S110 is configured with wireless communication, IP address allocation, and data synchronization strategies between multiple drone devices.

[0018] In conjunction with the first aspect, this embodiment of the invention provides a second possible implementation of the first aspect, wherein, in S100, the modification of the camera configuration file for the SLAM algorithm includes:

[0019] S120, modify the camera's intrinsic parameters, image resolution, image frame rate, camera type, image topic name, and image storage path or driver interface to match the camera parameters of multiple UAV devices with the hardware installation.

[0020] In conjunction with the first aspect, this embodiment of the invention provides a third possible implementation of the first aspect, wherein, in S200, the step of running multi-machine collaborative perception and mapping related nodes on multiple UAV devices respectively, so that each UAV device independently completes mapping, localization, and identification tasks, includes:

[0021] S210 starts the ROS core node and camera sensor data receiving node to achieve real-time transmission of image data.

[0022] S220: Start the single-machine SLAM mapping and localization node, relocalization node, multi-machine scene coordinate system management node and target detection node, subscribe to the received image data, perform single-machine mapping and localization and target detection, output the current position of the UAV device, and perform relocalization and multi-machine scene management based on the data shared between multiple UAV devices.

[0023] In conjunction with the first aspect, this embodiment of the invention provides a fourth possible implementation of the first aspect, wherein, in S300, the initial SLAM synchronization and Rviz visualization operation for multiple UAV devices includes:

[0024] S310, each of the aforementioned drone devices performs single-machine initialization at the same location to construct a local two-dimensional grid scene.

[0025] S320, start and initialize the Rviz visualization tool, display the constructed local two-dimensional raster scene in the Rviz visualization interface, and perform command interaction.

[0026] In conjunction with the first aspect, this embodiment of the invention provides a fifth possible implementation of the first aspect, wherein, in S400, the location sharing and collaborative perception between each of the drone devices across device scenarios via the Rviz visualization tool includes:

[0027] S410, using the command interaction buttons in the Rviz visualization tool, transmits the scene and location data of each of the drone devices to the other drone devices.

[0028] S420, after receiving the scene and location data, the other drone devices relocate the current image frame on the scene.

[0029] In conjunction with the first aspect, this embodiment of the invention provides a sixth possible implementation of the first aspect, wherein, in S500, calculating the coordinate transformation relationship between the scenes of each of the drone devices, unifying the local scenes and poses constructed by each of the drone devices into the coordinate system of the global scene, and displaying the position of each of the drone devices in real time in the global scene includes:

[0030] S510 calculates the coordinate system transformation relationship between the local scene and other device scenes based on the fact that the local device is in the same position in the world coordinate system. It continuously processes 60 frames of images, obtains 60 coordinate transformation matrices and calculates the average to obtain the relocation coordinate system transformation relationship, and records it in the multi-machine scene coordinate system management node.

[0031] S520, the other device scenes are transformed using the repositioning coordinate system transformation relationship to obtain a unified global scene.

[0032] S530, the location information of other devices is transformed by the repositioning coordinate system transformation relationship to obtain the current position and historical trajectory of each UAV device, and the data is displayed in different colors on the Rviz visualization tool interface.

[0033] In conjunction with the first aspect, this embodiment of the invention provides a seventh possible implementation of the first aspect, wherein, in S600, the step of performing target detection while the UAV equipment is moving, and performing multi-source target fusion perception and visualization on multiple UAV equipment, includes:

[0034] S610, when the drone device moves, visual target detection is performed on the target object within the camera's field of view, the target object is outlined with a rectangle to obtain an object label, and the position of the target object in the relative coordinate system corresponding to the local scene is calculated based on the size and position of the rectangle.

[0035] S620: Calculate the absolute position of the target object in the global scene based on the position of the drone equipment, and classify the object's label.

[0036] S630: When multiple devices observe the same target, position optimization and data fusion are performed based on the object label, and the target is uniformly displayed in the interface of the Rviz visualization tool.

[0037] In conjunction with the first aspect, this embodiment of the invention provides an eighth possible implementation of the first aspect, wherein, in S700, when multiple UAV devices meet again during their forward movement, data sharing is resumed, and the global scene and target data are updated until the task is completed and exited, including:

[0038] S710, when the drone device is close to other drone devices during its forward movement, determine whether it has entered the data sharing range.

[0039] S720, if the data sharing range is entered, the data of the global scene and the target object are synchronously updated. Based on the latest global scene, the data is updated according to the repositioning coordinate system transformation relationship until the task is completed and exited.

[0040] Secondly, embodiments of the present invention also provide an indoor environment positioning, mapping, and target recognition system for unmanned aerial vehicle (UAV) swarms, comprising:

[0041] The data collaboration configuration module is used for multi-drone collaborative deployment based on SLAM, configuring data communication between multiple drone devices, and modifying camera configuration files for the SLAM algorithm.

[0042] The mapping and localization module is used to run multi-machine collaborative perception and mapping related nodes on multiple UAV devices, enabling each UAV device to independently complete mapping, localization and recognition tasks.

[0043] The initial synchronization module is used to perform SLAM initial synchronization and Rviz visualization operations on multiple UAV devices.

[0044] The visualization control module is used by each of the aforementioned UAV devices to perform location sharing and collaborative perception across device scenarios using the Rviz visualization tool.

[0045] The coordinate transformation module is used to calculate the coordinate transformation relationship between the scenes of each of the UAV devices, unify the local scenes and poses constructed by each of the UAV devices into the coordinate system of the global scene, and display the position of each of the UAV devices in real time in the global scene.

[0046] The target detection module is used to detect targets when the UAV equipment is moving, and multiple UAV equipment perform multi-source target fusion perception and visualization.

[0047] The data update module is used to re-share data and update the global scene and target data when multiple drone devices meet again during their journey, until the mission is completed and the drone exits.

[0048] The beneficial effects of the embodiments of the present invention are:

[0049] This invention employs Visual Inertial Odometry (VIO) combined with distributed SLAM technology, enabling each UAV to achieve high-precision positioning in GPS-free environments. VIO utilizes IMU data for short-term, high-frequency position estimation, combined with camera data for error correction, thus improving positioning accuracy. Simultaneously, multiple UAVs share position information via wireless communication modules, achieving cluster collaborative optimization and further enhancing positioning accuracy.

[0050] This invention is based on a distributed SLAM architecture, where each UAV can share its own locally constructed scene in real time and synthesize a global scene through a data fusion algorithm. A coordinate transformation mechanism is employed to align scenes constructed by different UAVs to a unified coordinate system, improving global consistency. Simultaneously, loop closure detection technology is used to reduce scene drift and improve mapping accuracy.

[0051] This invention combines deep learning methods to achieve accurate target object detection in indoor environments. The drone uses an RGB-D camera to acquire color information of the target object, and combines this with the YOLO (You Only Look Once) target detection algorithm to achieve target recognition. By determining a formula for converting pixels to actual physical distances, the depth of the target in the environment is estimated, thereby obtaining the target's absolute position and annotating it in the global scene.

[0052] This invention supports dynamic joining and leaving of multiple drones, adapting to drone swarms of varying sizes. In complex environments (such as confined spaces and dynamic obstacles), an image keypoint matching algorithm is employed to improve positioning stability. It supports different types of sensor inputs (such as monocular cameras, binocular cameras, and RGB-D cameras), adapting to different hardware configurations and enhancing the system's generalization capabilities. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of the indoor environment positioning, mapping, and target recognition method for unmanned aerial vehicle (UAV) swarms according to the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0056] Please refer to Figure 1 The first embodiment of the present invention provides a method for indoor environment localization, mapping, and target recognition of a drone swarm, comprising: S100, multi-drone collaborative deployment based on SLAM, configuring data communication between multiple drone devices, and modifying the camera configuration file of the SLAM algorithm; S200, running relevant nodes for multi-drone collaborative perception and mapping on the multiple drone devices respectively, enabling each drone device to independently complete mapping, localization, and recognition tasks; S300, performing initial SLAM synchronization and Rviz visualization operations on the multiple drone devices; S400, each drone device using the Rviz visualization tool... Perform location sharing and collaborative perception across device scenarios; S500, calculate the coordinate transformation relationship between the scenarios of each UAV device, unify the local scenarios and poses constructed by each UAV device into the coordinate system of the global scenario, and display the position of each UAV device in real time in the global scenario; S600, perform target detection when the UAV devices move, and perform multi-source target fusion perception and visualization of multiple UAV devices; S700, when multiple UAV devices meet again during their forward movement, re-encode data sharing, update the global scenario and target data, until the task is completed and exits.

[0057] In step S100, configuring data communication between multiple drone devices includes: S110, configuring wireless communication, IP address allocation, and data synchronization strategies between the multiple drone devices. First, all devices are connected to the same local area network, and the IP address and working path of each device in the parameter list are set to ensure normal data transmission. Due to limited communication bandwidth, the maximum transmission bandwidth between two devices is set.

[0058] In step S100, modifying the camera configuration file for the SLAM algorithm includes: S120, modifying the camera's intrinsic parameters, image resolution, image frame rate, camera type, image topic name, and image storage path or driver interface to match the camera parameters of multiple UAV devices with the hardware installation. The camera's intrinsic parameters in the configuration file are modified according to the actual camera model used, and the camera's extrinsic parameters are modified based on the relative positions of the camera and IMU fixed on the device. If the camera's intrinsic parameters are unknown, they can be obtained through camera calibration.

[0059] In step S200, the step of running multi-drone collaborative perception and mapping nodes on multiple drone devices, enabling each drone device to independently complete mapping, localization, and recognition tasks, includes: S210, starting the ROS core node and camera sensor data receiving node to achieve real-time image data transmission; S220, starting the single-drone SLAM mapping and localization node, relocalization node, multi-drone scene coordinate system management node, and target detection node, subscribing to received image data, performing single-drone mapping, localization, and target detection, outputting the current position of the drone device, and performing relocalization and multi-drone scene management based on data shared among the multiple drone devices. The single-drone SLAM algorithm used is the ORB-SLAM3 algorithm, which supports various visual inertial sensor types, including monocular camera + IMU, binocular camera + IMU, and even binocular camera only. The configuration file selects to use binocular camera data + IMU inertial measurement unit data. Then, the relocalization node and multi-scene coordinate system management node are started to realize multi-drone relocalization and scene coordinate system management.

[0060] In step S300, the initial SLAM synchronization and Rviz visualization operation for multiple UAV devices includes: S310, each UAV device performs single-machine initialization at the same location, constructing a local two-dimensional grid scene; S320, the Rviz (ROS Visualization tool) visualization tool is started and initialized, displaying the constructed local two-dimensional grid scene on the Rviz visualization interface, and allowing command interaction. After the single-machine SLAM node is started, all UAVs initialize at the same location facing the same scene. Some movement is required during initialization to accelerate the process. After initialization, starting the Rviz visualization tool displays the constructed local scene.

[0061] In step S400, the location sharing and collaborative perception between each drone device across devices using the Rviz visualization tool includes: S410, using the command interaction buttons in the Rviz visualization tool, transmitting the scene and location data of each drone device to other drone devices; S420, after receiving the scene and location data, the other drone devices perform image relocalization on the scene using the current image frame. Data sharing occurs after initialization or when two drone devices are in a data sharing range. Upon receiving scene data from other devices, image relocalization is performed on that scene.

[0062] The relocation code is shown below:

[0063] C++

[0064] void ImageGrabber::GrabStereo(const sensor_msgs::ImageConstPtr &msgLeft, const sensor_msgs::ImageConstPtr &msgRight) {

[0065] / / Switch scenes

[0066] if (flag) {

[0067] Eigen::Quaterniond quat;

[0068] Publish_occpose(540, 540, quat, "Voccmap" + to_string(self_device_id), relo_pose_2D_pub);

[0069] for (int i = 0; i < device_num; i++) {

[0070] if (i + 1 == self_device_id) {

[0071] continue?

[0072] }

[0073] / / Check if scene update information has been received. If so, switch to the scene to be relocated.

[0074] ReadMapFile(i + 1);

[0075] / / If a scene update message is received, switch scenes.

[0076] if (mapflag[i] == 1) {

[0077] flag = false;

[0078] / / Switch scenes

[0079] ChangeMap();

[0080] mapflag[i] = 0;

[0081] break

[0082] }

[0083] }

[0084] }

[0085] / / Perform relocation

[0086] else {

[0087] / / Get left and right images

[0088] cv_bridge::CvImageConstPtr cv_ptrLeft;

[0089] try {

[0090] cv_ptrLeft = cv_bridge::toCvShare(msgLeft);

[0091] } catch (cv_bridge::Exception &e) {

[0092] ROS_ERROR("cv_bridge exception: %s", e.what());

[0093] return;

[0094] }

[0095] cv_bridge::CvImageConstPtr cv_ptrRight;

[0096] try {

[0097] cv_ptrRight = cv_bridge::toCvShare(msgRight);

[0098] } catch (cv_bridge::Exception &e) {

[0099] ROS_ERROR("cv_bridge exception: %s", e.what());

[0100] return;

[0101] }

[0102] / / Obtain the relocation result

[0103] relo_pose = mpSLAM->Trackonlyonce(cv_ptrLeft->image, cv_ptrRight->image, cv_ptrLeft->header.stamp.toSec());

[0104] / / cout << relo_pose.matrix() << endl;

[0105] / / Publish relocation results

[0106] Publish();

[0107] / / Record the number of relocations and the number of successful relocations

[0108] if (relo_pose.matrix().block<1, 1>(0, 3)(0, 0) != 0) {

[0109] succ_num++;

[0110] cout << "relocalization success!" << endl;

[0111] }

[0112] num++;

[0113] / / The relocation process will end if the number of relocation attempts exceeds 300 or the number of successful relocations exceeds 60.

[0114] if (succ_num > 60 || num > 300) {

[0115] flag = true)

[0116] cout << "end to relocalization, total: " + to_string(num) + "times" << endl;

[0117] cout << "end to relocalization, succ: " + to_string(succ_num) +"times" << endl;

[0118] num = 0;

[0119] succ_num = 0;

[0120] }

[0121] point_cloud_msg.header.stamp = ros::Time::now();

[0122] point_cloud_msg.header.frame_id = "map" + Frame_id;

[0123] point_cloud_pub.publish(point_cloud_msg);

[0124] }

[0125] }

[0126] ```

[0127] The specific steps are as follows: First, determine whether new scene data information has been received. If so, relocate the scene to the received scene, then call TrackOnlyOnce to enter relocation and record the result as relo_pose. Then call Publish() to publish the relocation result information. Repeat the relocation until 60 frames are successfully captured or the total number of frames reaches 300, then repeat the process of determining whether new scene data has been received.

[0128] In S500, the calculation of the coordinate transformation relationship between the scenes of each UAV device, unifying the local scene and pose constructed by each UAV device into the coordinate system of the global scene, and displaying the position of each UAV device in real time in the global scene includes: S510, based on the fact that the local device has the same position in the world coordinate system, calculating the coordinate system transformation relationship between the local scene and the scenes of other devices, continuously processing 60 frames of images, obtaining 60 coordinate transformation matrices and averaging them to obtain the repositioning coordinate system transformation relationship with smaller error, and recording it in the multi-machine scene coordinate system management node.

[0129] The coordinate system management node receives the localization results from single-machine SLAM and the relocalization results from step S4, and then calculates the coordinate system transformation relationship. The C++ code for the calculation is shown below:

[0130] ```C++

[0131] geometry_msgs::Pose multi_t_voc::GetTF(const geometry_msgs::PoseStamped::ConstPtr &parent_frame_pose,

[0132] const geometry_msgs::PoseStamped::ConstPtr &child_frame_pose) {

[0133] geometry_msgs::Pose d_P;

[0134] Eigen::Quaterniond q, q1, q2;

[0135] q1.x() = parent_frame_pose->pose.orientation.x;

[0136] q1.y() = parent_frame_pose->pose.orientation.y;

[0137] q1.z() = parent_frame_pose->pose.orientation.z;

[0138] q1.w() = parent_frame_pose->pose.orientation.w;

[0139] q2.x() = child_frame_pose->pose.orientation.x;

[0140] q2.y() = child_frame_pose->pose.orientation.y;

[0141] q2.z() = child_frame_pose->pose.orientation.z;

[0142] q2.w() = child_frame_pose->pose.orientation.w;

[0143] q = q1 * q2.inverse();

[0144] d_P.orientation.x = q.x();

[0145] d_P.orientation.y = q.y();

[0146] d_P.orientation.z = q.z();

[0147] d_P.orientation.w = q.w();

[0148] Eigen::Matrix3d rotation_matrix = q.toRotationMatrix();

[0149] Eigen::Vector3d initial_position(child_frame_pose->pose.position.x, child_frame_pose->pose.position.y, child_frame_pose->pose.position.z);

[0150] / / Calculate the rotation matrix of the transformation

[0151] Eigen::Vector3d transformed_position = rotation_matrix * initial_position;

[0152] / / Calculate the translation matrix of the transformation

[0153] d_P.position.x = parent_frame_pose->pose.position.x - transformed_position[0];

[0154] d_P.position.y = parent_frame_pose->pose.position.y - transformed_position[1];

[0155] d_P.position.z = parent_frame_pose->pose.position.z - transformed_position[2];

[0156] return d_P;

[0157] }

[0158] ```

[0159] After calculating the coordinate system transformation relationship, the result is updated in the multi-machine coordinate system relationship table, and the C++ code for updating the coordinate system relationship table is shown below:

[0160] C++

[0161] void multi_t_voc::UpdataTF() {

[0162] while (true) {

[0163] / / 3D pose update

[0164] if (!relo_Tque3D.empty() && !Tque3D.empty()) {

[0165] std::lock_guard <std::mutex>lock1(relo_Tque3D_mutex);

[0166] std::lock_guard <std::mutex>lock2 ( Tque3D_mutex ) ;

[0167] auto clock_pose_msg = clock_Tque3D.front();

[0168] auto pose_msg = Tque3D.front();

[0169] other3D_id = stoi(string(1, relo_pose_msg->header.frame_id.back()));

[0170] Update3D(GetTF(pose_msg, relo_pose_msg));

[0171] clock_Tque3D.pop();

[0172] Tque3D.pop();

[0173] }}

[0174] / / 2d row

[0175] if (!clock_Tque2D.empty() && !Tque2D.empty()) {

[0176] std : : lock_guard <std::mutex>lock1(relo_Tque2D_mutex);

[0177] std::lock_guard <std::mutex>lock2 ( Tque2D_mutex ) ;

[0178] auto pose_msg = Tque2D.front();

[0179] auto clock_pose_msg = clock_Tque2D.front();

[0180] other2D_id = stoi(string(1, relo_pose_msg->header.frame_id.back()));

[0181] Update2D(GetTF(pose_msg, relo_pose_msg));

[0182] clock_Tque2D.pop();

[0183] Tque2D.pop();

[0184] }}

[0185] }}

[0186] }}

[0187] void multi_t_voc::Update2D(geometry_msgs::Pose T) {

[0188] if ( Tque2D_vec [ other2D_id -- 1 ]. size ( ) >= 10 ) { ;

[0189] Tque2D_vec[other2D_id -- 1].pop_front();

[0190] Tque2D_vec[other2D_id -- 1].push_back(T);

[0191] } else {

[0192] Tque2D_vec[other2D_id -- 1].push_back(T);

[0193] }}

[0194] / / Install the snowflake snowflakes

[0195] std::cout << "The coordinate system relationship before the update is " << MTvoc2D[self_id - 1][other2D_id - 1].position.x << std::endl;

[0196] MTvoc2D[self_id - 1][other2D_id - 1] = calculateAveragePose(Tque2D_vec[other2D_id - 1]);

[0197] MTvoc2D[other2D_id - 1][self_id - 1] = invertPose(calculateAveragePose(Tque2D_vec[other2D_id - 1]));

[0198] / / Output the updated coordinate system relationship

[0199] std::cout << "The updated coordinate system relationship is " << MTvoc2D[self_id - 1][other2D_id - 1].position.x << std::endl;

[0200] / / check();

[0201] SaveMTvoc2D();

[0202] cout << "update 2d" << endl;

[0203] }

[0204] ```

[0205] Since we need to visualize 2D raster scenes, we also need to maintain a 2D scene coordinate system relationship table. The specific update steps are the same as those for updating a 3D coordinate system relationship table. We input the latest coordinate system relationship by calling the UpdateTF() function, then use a sliding window to determine whether to delete historical information outside the sliding window length, and then calculate the average of all coordinate system relationships within the sliding window to obtain the final coordinate system transformation relationship between scenes.

[0206] S520, the other device scenes are transformed using the repositioning coordinate system transformation relationship to obtain a unified global scene.

[0207] By publishing scene coordinate system transformation relationships, all scenes from other devices are converted to the local scene coordinate system, thereby merging scene information that has been explored by all devices. The specific C++ code for publishing is shown below:

[0208] C++

[0209] / / Publish coordinate system TF relationship thread

[0210] void boradTF(multi_t_voc &mpvoc) {

[0211] tf::Transform Transform;

[0212] tf::TransformBroadcaster broadcaster;

[0213] cout << "Publish coordinate system TF relationship thread" << endl;

[0214] while (1) {

[0215] for (int i = 0; i < mpvoc.device_num; i++) {

[0216] for (int j = i + 1; j < mpvoc.device_num; j++) {

[0217] if (mpvoc.MTvoc2D[i][j].position.x == 0) {

[0218] continue?

[0219] }

[0220] cout << "Publish coordinate system TF relationship" << endl;

[0221] tf::poseMsgToTF(mpvoc.MTvoc2D[i][j], Transform);

[0222] broadcaster.sendTransform(tf::StampedTransform(Transform,ros::Time::now(), "Voccmap" + to_string(i + 1), "Voccmap" + to_string(j +1)));

[0223] }

[0224] }

[0225] sleep(0.1);

[0226] }

[0227] }

[0228] ```

[0229] S530, the location information of other devices is transformed by the repositioning coordinate system transformation relationship to obtain the current position and historical trajectory of each UAV device, and the data is displayed in different colors on the Rviz visualization tool interface.

[0230] The scene coordinate system management node also shares its existing coordinate system relationships with other devices, ensuring consistency in the coordinate system relationship tables maintained by all devices and reducing redundant calculations. Upon the next encounter, the scene coordinate system relationships are updated based on the existing relationships to minimize errors caused by environmental factors.

[0231] In S600, the step of performing target detection while the UAV is moving, and performing multi-source target fusion perception and visualization on multiple UAVs, includes: S610, when the UAV is moving, performing visual target detection on target objects within the camera's field of view, outlining the target objects with a rectangle to obtain object labels, and calculating the position of the target object in the relative coordinate system corresponding to the local scene based on the size and position of the rectangle; S620, calculating the absolute position of the target object in the global scene based on the position of the UAV, and classifying the object labels; S630, when multiple devices observe the same target, performing position optimization and data fusion based on the object labels, and uniformly displaying the target in the interface of the Rviz visualization tool.

[0232] A lightweight single-camera object detection technology based on YOLOv5 is employed to achieve efficient and accurate 2D object detection. YOLOv5 is chosen as the core object detection model, maintaining high detection accuracy while offering fast processing speed and lightweight characteristics, making it suitable for real-time applications. The design of YOLOv5 optimizes runtime and resource consumption, enabling it to process large amounts of visual data with low latency. Furthermore, even in complex multi-object environments, such as those with occlusion and varying lighting conditions, YOLOv5 maintains high detection accuracy. The node output includes the 2D bounding box location of the object, the object category, and the confidence score.

[0233] In S700, when multiple UAV devices encounter each other again during their forward movement, data sharing is restarted, and the global scene and target data are updated until the task is completed and exited. This includes: S710, when a UAV device is close to other UAV devices during its forward movement, it is determined whether it has entered the data sharing range; S720, if it has entered the data sharing range, the global scene and target object data are synchronously updated, and the update is performed based on the latest global scene and the repositioning coordinate system transformation relationship, until the task is completed and exited.

[0234] A second embodiment of the present invention provides an indoor environment localization, mapping, and target recognition system for a drone swarm, comprising: a data collaboration configuration module for multi-drone collaborative deployment based on SLAM, configuring data communication between multiple drone devices, and modifying the camera configuration file of the SLAM algorithm; a mapping and localization module for running relevant nodes for multi-drone collaborative perception and mapping on the multiple drone devices, enabling each drone device to independently complete mapping, localization, and recognition tasks; an initial synchronization module for performing initial SLAM synchronization and Rviz visualization operations on the multiple drone devices; and a visualization control module for enabling each drone device to be visualized via Rviz. The system includes a tool for location sharing and collaborative perception across devices and scenarios; a coordinate transformation module for calculating the coordinate transformation relationship between the scenarios of each UAV device, unifying the local scenarios and poses constructed by each UAV device into the coordinate system of the global scenario, and displaying the position of each UAV device in real time in the global scenario; a target detection module for detecting targets when the UAV devices are moving, and performing multi-source target fusion perception and visualization across multiple UAV devices; and a data update module for re-sharing data and updating the global scenario and target data when multiple UAV devices meet again during their movement, until the task is completed and the system exits.

[0235] The embodiments of the present invention aim to protect a method and system for indoor environmental positioning, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms, which has the following effects:

[0236] This invention provides a method and system for indoor environment localization, mapping, and target recognition in a human-machine swarm, optimizing the autonomous localization, mapping, and target detection capabilities of unmanned aerial vehicles (UAVs) in GPS-free environments. This invention improves the positioning accuracy, mapping efficiency, and target recognition capabilities of UAVs through multi-sensor fusion, distributed SLAM algorithms, and swarm collaboration mechanisms. By employing visual inertial odometry (VIO) combined with distributed SLAM, each UAV can independently locate itself while sharing scene data, optimizing global consistency. Wireless communication technology enables pose information synchronization among multiple UAVs, ensuring collaborative operation within the UAV swarm and improving task execution efficiency. Furthermore, the target detection module combines deep learning and geometric feature matching to achieve high-precision target recognition in indoor environments and integrates the recognition results with the global scene, providing reliable data support for inspection, security, and rescue missions.

[0237] Experimental results show that this invention can significantly improve the autonomous localization and target recognition capabilities of UAVs in complex indoor environments. Compared with traditional single-unit SLAM solutions, it exhibits higher stability, real-time performance, and collaborative operation capabilities. This invention is applicable to multiple application scenarios such as warehouse inspection, security monitoring, and post-disaster search and rescue, providing strong technical support for the intelligent development of UAV swarms.

[0238] The computer program product of the UAV swarm indoor environment positioning, mapping and target recognition method and device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0239] Specifically, the storage medium can be a general-purpose storage medium, such as a portable disk or hard disk. When the computer program on the storage medium is run, it can execute the above-mentioned UAV swarm indoor environment positioning, mapping and target recognition method, thereby significantly improving the UAV's autonomous positioning and target recognition capabilities in complex indoor environments.

[0240] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / std::mutex> < / std::mutex> < / std::mutex> < / std::mutex>

Claims

1. A method for indoor environmental positioning, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms, characterized in that, include: S100, based on SLAM, enables multi-drone collaborative deployment, configures data communication between multiple drone devices, and modifies the camera configuration file of the SLAM algorithm; S200, running multi-machine collaborative perception and mapping nodes on multiple UAV devices respectively, so that each UAV device can independently complete mapping, positioning and recognition tasks; S300, Perform SLAM initial synchronization and Rviz visualization operations on multiple UAV devices, specifically including: S310, Each UAV device performs single-machine initialization at the same location to construct a local two-dimensional grid scene; S320, Start and initialize the Rviz visualization tool, display the constructed local two-dimensional grid scene in the Rviz visualization interface, and perform command interaction; S400, each of the aforementioned drone devices performs cross-device scene sharing and collaborative perception through the Rviz visualization tool, specifically including: S410, using the command interaction button in the Rviz visualization tool, transmitting the scene and location data of each of the aforementioned drone devices to other of the aforementioned drone devices; S420, after receiving the scene and location data, the other of the aforementioned drone devices perform image relocalization on the scene for the current image frame; S500, calculate the coordinate transformation relationship between the scenes of each UAV device, unify the local scenes and poses constructed by each UAV device into the coordinate system of the global scene, and display the position of each UAV device in real time in the global scene. Specifically, this includes: S510, based on the fact that the local device has the same position in the world coordinate system, calculate the coordinate system transformation relationship between the local scene and the scenes of other devices, process 60 frames of images continuously, obtain 60 coordinate transformation matrices and calculate the average to obtain the repositioning coordinate system transformation relationship, and record it in the multi-machine scene coordinate system management node; S520, transform the scenes of other devices through the repositioning coordinate system transformation relationship to obtain a unified global scene; S530, transform the position information of other devices through the repositioning coordinate system transformation relationship to obtain the current position and historical trajectory of each UAV device, and use different colors to represent them in the interface of the Rviz visualization tool. S600, target detection is performed when the UAV is moving, and multiple UAVs perform multi-source target fusion perception and visualization, specifically including: S610, when the UAV is moving, visual target detection is performed on target objects within the camera's field of view, the target objects are outlined with a rectangle to obtain object labels, and the position of the target object in the relative coordinate system corresponding to the local scene is calculated based on the size and position of the rectangle; S620, the absolute position of the target object in the global scene is calculated based on the position of the UAV, and the object labels are classified; S630, when multiple devices observe the same target, position optimization and data fusion are performed based on the object labels, and the target is uniformly displayed in the interface of the Rviz visualization tool; S700, when multiple drone devices meet again during their forward movement, data sharing is restarted, and the global scene and target data are updated until the task is completed and exited. Specifically, this includes: S710, when a drone device is close to other drone devices during its forward movement, it is determined whether it has entered the data sharing range; S720, if it has entered the data sharing range, the global scene and target object data are synchronously updated, based on the latest global scene and the repositioning coordinate system transformation relationship, until the task is completed and exited.

2. The method for indoor environmental positioning, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms according to claim 1, characterized in that, In S100, the data communication between the configured multiple UAV devices includes: The S110 is configured with wireless communication, IP address allocation, and data synchronization strategies between multiple drone devices.

3. The method for indoor environmental positioning, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms according to claim 2, characterized in that, In S100, the camera configuration file for modifying the SLAM algorithm includes: S120, modify the camera's intrinsic parameters, image resolution, image frame rate, camera type, image topic name, and image storage path or driver interface to match the camera parameters of multiple UAV devices with the hardware installation.

4. The method for indoor environmental positioning, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms according to claim 1, characterized in that, In S200, the step of running relevant nodes for multi-drone collaborative perception and mapping on multiple drone devices, enabling each drone device to independently complete mapping, localization, and identification tasks, includes: S210 starts the ROS core node and camera sensor data receiving node to achieve real-time transmission of image data; S220: Start the single-machine SLAM mapping and localization node, relocalization node, multi-machine scene coordinate system management node and target detection node, subscribe to the received image data, perform single-machine mapping and localization and target detection, output the current position of the UAV device, and perform relocalization and multi-machine scene management based on the data shared between multiple UAV devices.

5. A system for indoor environmental positioning, mapping, and target recognition of unmanned aerial vehicle (UAV) swarms, characterized in that, include: The data collaboration configuration module is used for multi-machine collaborative deployment based on SLAM, configuring data communication between multiple UAV devices, and modifying camera configuration files for the SLAM algorithm. The mapping and localization module is used to run the relevant nodes of multi-machine collaborative perception and mapping on multiple UAV devices respectively, so that each UAV device can independently complete the mapping, localization and recognition tasks; The initial synchronization module is used to perform SLAM initial synchronization and Rviz visualization operations on multiple UAV devices. Specifically, it includes: S310, each UAV device performs single-machine initialization at the same location to build a local two-dimensional grid scene; S320, the Rviz visualization tool is started and initialized, the built local two-dimensional grid scene is displayed in the Rviz visualization interface, and command interaction is performed. The visualization control module is used for each of the drone devices to perform cross-device scene sharing and collaborative perception through the Rviz visualization tool. Specifically, it includes: S410, using the command interaction button in the Rviz visualization tool to transmit the scene and location data of each of the drone devices to other drone devices; S420, after receiving the scene and location data, the other drone devices perform image relocation on the scene for the current image frame. The coordinate transformation module is used to calculate the coordinate transformation relationship between the scenes of each UAV device, unify the local scenes and poses constructed by each UAV device into the coordinate system of the global scene, and display the position of each UAV device in real time in the global scene. Specifically, it includes: S510, calculating the coordinate system transformation relationship between the local scene and the scenes of other devices based on the fact that the local device has the same position in the world coordinate system, continuously processing 60 frames of images, obtaining 60 coordinate transformation matrices and averaging them to obtain the repositioning coordinate system transformation relationship, and recording it in the multi-machine scene coordinate system management node; S520, transforming the scenes of other devices through the repositioning coordinate system transformation relationship to obtain a unified global scene; S530, transforming the position information of other devices through the repositioning coordinate system transformation relationship to obtain the current position and historical trajectory of each UAV device, and using different colors to represent them in the Rviz visualization tool interface; The target detection module is used to detect targets when the UAV devices are moving. Multiple UAV devices perform multi-source target fusion perception and visualization, specifically including: S610, when the UAV devices are moving, visual target detection is performed on target objects within the camera's field of view, the target objects are outlined with a rectangle to obtain object labels, and the position of the target object in the relative coordinate system corresponding to the local scene is calculated based on the size and position of the rectangle; S620, the absolute position of the target object in the global scene is calculated based on the position of the UAV devices, and the object labels are classified; S630, when multiple devices observe the same target, position optimization and data fusion are performed based on the object labels, and the target is uniformly displayed in the interface of the Rviz visualization tool. The data update module is used to re-share data and update the global scene and target data when multiple drone devices meet again during their forward movement, until the task is completed and the user exits. Specifically, it includes: S710, when a drone device is close to other drone devices during its forward movement, determining whether it has entered the data sharing range; S720, if it has entered the data sharing range, then synchronously updating the data of the global scene and the target object, updating based on the latest global scene and the repositioning coordinate system transformation relationship, until the task is completed and the user exits.

Images