Cluster robot cooperative search and rescue processing method, system and platform based on multi-modal fusion and NB-IoT communication and storage medium

Abstract

The invention discloses a cluster robot cooperative search and rescue processing method, system and platform based on multi-modal fusion and NB-IoT communication and a storage medium. According to the scheme of the invention, through a bionic quadruped robot platform, adaptive motion control based on a central pattern generator (CPG), a tight coupling multi-modal SLAM front end and cooperative communication and map fusion based on NB-IoT, the cooperative search and rescue processing of the cluster robot is realized; and efficient, stable and collaborative operation of the robot in an uncertain environment is realized. That is to say, according to the scheme of the invention, through deep fusion of adaptive motion control of the bionic quadruped robot, laser-vision-IMU tight coupling SLAM front-end optimization and NB-IoT low-power-consumption wide-area communication technology, the core pain points of large sensing blind area, poor motion stability and difficult multi-machine cooperative communication of a single robot in an uncertain environment after a disaster are effectively solved; high-robustness autonomous positioning, environment modeling and cooperative search and rescue of multiple robots in an extreme environment are realized, and the search and rescue efficiency and success rate are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the fields of robotics, artificial intelligence and Internet of Things communication processing technology, and specifically relates to a method, system, platform and storage medium for collaborative search and rescue of cluster robots based on multimodal fusion and NB-IoT communication. Background Technology

[0002] Following natural disasters such as earthquakes and landslides, the post-disaster environment exhibits high uncertainty in three aspects: physical structure, sensing conditions, and communication links. The complex and rugged ruins, containing obstacles, slopes, and unstable surfaces, pose a significant challenge to robot mobility. Simultaneously, interference from smoke, darkness, and dust can drastically reduce the reliability of single-sensor perception (such as pure LiDAR or pure vision), making traditional single-robot SLAM (Simultaneous Localization and Mapping) techniques prone to failure. Furthermore, ruins strongly shield wireless signals, rendering traditional communication methods like WiFi and 4G prone to weak signal coverage and high power consumption, hindering stable collaboration and data transmission among multiple robots.

[0003] Currently, although there are studies using multiple robots, they are mostly limited to collaboration in ideal environments or use loosely coupled data fusion methods, failing to fully address the three core issues of motion adaptability, perception robustness, and communication reliability in extremely uncertain environments.

[0004] Therefore, in order to address the technical problems and deficiencies mentioned above, there is an urgent need to design and develop a collaborative search and rescue method, system, platform, and storage medium for cluster robots based on multimodal fusion and NB-IoT communication. Summary of the Invention

[0005] To overcome the shortcomings and difficulties of the existing technologies, the present invention aims to provide a collaborative search and rescue processing method, system, platform and storage medium for swarm robots based on multimodal fusion and NB-IoT communication. This method can deeply integrate biomimetic motion control, tightly coupled multimodal perception and low-power wide-area communication technologies to improve the search and rescue efficiency and success rate of swarm robots in post-disaster environments.

[0006] The first objective of this invention is to provide a collaborative search and rescue processing method for swarm robots based on multimodal fusion and NB-IoT communication; the second objective of this invention is to provide a collaborative search and rescue processing system for swarm robots based on multimodal fusion and NB-IoT communication; the third objective of this invention is to provide a collaborative search and rescue processing platform for swarm robots based on multimodal fusion and NB-IoT communication; and the fourth objective of this invention is to provide a computer-readable storage medium.

[0007] The first objective of this invention is achieved by the method comprising:

[0008] Generate and acquire first data corresponding to the post-disaster environment; wherein, the first data is multimodal data, including laser point cloud data, visual image data and IMU data;

[0009] By combining a tightly coupled multimodal SLAM front-end processing algorithm, the first data is spatiotemporally aligned and fused separately to construct a first map corresponding to the post-disaster environment in real time, and generate second data corresponding to the first map; wherein, the first map is local environment sub-map data; and the second data is its own pose data;

[0010] Based on the first map and the second data, and combined with the map overlap detection algorithm, it is determined in real time whether there is an overlapping area between the first maps, and third data corresponding to the overlapping area is generated; wherein, the third data is a relative pose transformation relationship;

[0011] Based on the third data, the first map is processed by real-time coordinate transformation and stitching, and a second map corresponding to the first map is generated. At the same time, based on the second map and combined with the NB-IoT network, the robot is controlled to perform collaborative search and rescue operations. The second map is a complete environmental map that corresponds to the post-disaster environment and has been optimized.

[0012] The second objective of this invention is achieved by providing a system for implementing the collaborative search and rescue method for swarm robots based on multimodal fusion and NB-IoT communication, wherein the system comprises:

[0013] The data generation and acquisition unit is used to generate and acquire first data corresponding to the post-disaster environment; wherein, the first data is multimodal data, including laser point cloud data, visual image data and IMU data;

[0014] The first data processing and generation unit is used to combine a tightly coupled multimodal SLAM front-end processing algorithm to perform spatiotemporal alignment and fusion processing on the first data, construct a first map corresponding to the post-disaster environment in real time, and generate second data corresponding to the first map; wherein, the first map is local environment sub-map data; and the second data is its own pose data.

[0015] The second data processing and generation unit is used to determine in real time whether there is an overlapping area between the first maps based on the first map and the second data, combined with a map overlap detection algorithm, and to generate third data corresponding to the overlapping area; wherein, the third data is a relative pose transformation relationship.

[0016] The third data processing and generation unit is used to perform real-time coordinate transformation and stitching processing on the first map based on the third data, and generate a second map corresponding to the first map. At the same time, based on the second map and in conjunction with the NB-IoT network, it controls the robot to perform collaborative search and rescue operations. The second map is a complete environmental map that corresponds to the post-disaster environment and has been optimized.

[0017] The third objective of this invention is achieved as follows: it includes a processor, a memory, and a control program for a swarm robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication; wherein the processor executes the control program for the swarm robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication, the control program for the swarm robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication is stored in the memory, and the control program for the swarm robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication implements the swarm robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication.

[0018] The fourth objective of this invention is achieved as follows: the computer-readable storage medium stores a control program for a cluster robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication, and the control program for the cluster robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication implements the cluster robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication.

[0019] This invention generates and acquires first data corresponding to a post-disaster environment through a method; wherein the first data is multimodal data, including laser point cloud data, visual image data, and IMU data; combining a tightly coupled multimodal SLAM front-end processing algorithm, the first data is spatiotemporally aligned and fused to construct a first map corresponding to the post-disaster environment in real time, and second data corresponding to the first map is generated; wherein the first map is local environment sub-map data; the second data is its own pose data; based on the first map and the second data, and combined with a map overlap detection algorithm, it is determined in real time whether there is an overlapping area between the first maps, and third data corresponding to the overlapping area is generated; its In this method, the third data represents the relative pose transformation relationship. Based on the third data, the first map is processed by real-time coordinate transformation and stitching, and a second map corresponding to the first map is generated. Simultaneously, based on the second map and combined with the NB-IoT network, the robot is controlled to perform collaborative search and rescue operations. The second map is a complete environmental map corresponding to the post-disaster environment and optimized. The method also includes a corresponding system, platform, and storage medium. Through a biomimetic quadruped robot platform, adaptive motion control based on a central pattern generator (CPG), a tightly coupled multimodal SLAM front-end, and collaborative communication and map fusion based on NB-IoT, the robot can achieve efficient, stable, and collaborative operation in uncertain environments.

[0020] In other words, the present invention effectively solves the core pain points of large perception blind spots, poor motion stability, and difficulty in multi-robot collaborative communication in uncertain post-disaster environments by deeply integrating adaptive motion control of biomimetic quadruped robots, laser-vision-IMU tightly coupled SLAM front-end optimization, and NB-IoT low-power wide-area communication technology. It achieves highly robust autonomous localization, environmental modeling, and collaborative search and rescue of multiple robots in extreme environments, significantly improving search and rescue efficiency and success rate. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the biomimetic robot structure of a swarm robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication according to the present invention.

[0023] Figure 2This is a schematic diagram of the adaptive motion method based on rhythmic gait, which is a collaborative search and rescue processing method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0024] Figure 3 This is a schematic diagram of the architecture of a multimodal SLAM-based collaborative search and rescue method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0025] Figure 4 This is a schematic diagram of the core chip of the communication module of a collaborative search and rescue processing method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0026] Figure 5 This is a schematic diagram of the NB-IoT module chip in the collaborative search and rescue processing method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0027] Figure 6 This is a schematic diagram of the SIM card slot circuit for a collaborative search and rescue processing method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0028] Figure 7 This is a circuit diagram of the signal receiving module of a cluster robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication according to the present invention.

[0029] Figure 8 This is a schematic diagram of the multimodal information fusion method of the collaborative search and rescue processing method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention;

[0030] Figure 9 This is a comparative diagram of the pitch and roll angles of the fuselage under different modes in a collaborative search and rescue method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0031] Figure 10 This is a schematic diagram illustrating the collaborative construction of a cluster robot collaborative search and rescue method based on multimodal fusion and NB-IoT communication according to the present invention.

[0032] Figure 10 (a) is a schematic diagram of the robot A surrounding scene in a cluster robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication according to the present invention;

[0033] Figure 10 (b) is a schematic diagram of the robot B surrounding the scene in the collaborative search and rescue processing method of cluster robots based on multimodal fusion and NB-IoT communication according to the present invention;

[0034] Figure 10(c) is a schematic diagram of the multi-robot fusion construction of a cluster robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication according to the present invention;

[0035] Figure 11 This is a schematic diagram of the process steps of a collaborative search and rescue method for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0036] Figure 12 This is a schematic diagram of the architecture of a cluster robot collaborative search and rescue processing system based on multimodal fusion and NB-IoT communication according to the present invention;

[0037] Figure 13 This is a schematic diagram of the collaborative search and rescue processing platform architecture for cluster robots based on multimodal fusion and NB-IoT communication according to the present invention.

[0038] Figure 14 This is a schematic diagram of a computer-readable storage medium architecture in one embodiment of the present invention. Detailed Implementation

[0039] To facilitate a clearer understanding of the objectives, technical solutions, and advantages of this invention, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Those skilled in the art can easily understand other advantages and effects of this invention from the content disclosed in this specification.

[0040] This invention can also be implemented or applied through other different specific examples, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of this invention.

[0041] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0042] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Secondly, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0043] Preferably, the collaborative search and rescue method for swarm robots based on multimodal fusion and NB-IoT communication of the present invention is applied in one or more terminals or servers. The terminal is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0044] The terminal can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal can interact with the customer via a keyboard, mouse, remote control, touchpad, or voice control device.

[0045] This invention provides a method, system, platform, and storage medium for collaborative search and rescue of cluster robots based on multimodal fusion and NB-IoT communication.

[0046] like Figure 11 The diagram shown is a flowchart of a collaborative search and rescue method for cluster robots based on multimodal fusion and NB-IoT communication provided in an embodiment of the present invention.

[0047] In this embodiment, the cluster robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication can be applied to a terminal with display function or a fixed terminal. The terminal is not limited to personal computers, smartphones, tablets, desktop computers or all-in-one computers with cameras, etc.

[0048] The swarm robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication can also be applied to a hardware environment consisting of a terminal and a server connected to the terminal via a network. The network includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The swarm robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication in this embodiment of the invention can be executed by the server, by the terminal, or by both the server and the terminal.

[0049] For example, for a swarm robot collaborative search and rescue processing terminal that requires multimodal fusion and NB-IoT communication, the swarm robot collaborative search and rescue processing function based on multimodal fusion and NB-IoT communication provided by the method of this invention can be directly integrated into the terminal, or a client for implementing the method of this invention can be installed. Alternatively, the method provided by this invention can also run on servers or other devices in the form of a Software Development Kit (SDK), providing an interface for the swarm robot collaborative search and rescue processing function based on multimodal fusion and NB-IoT communication in the form of an SDK. Terminals or other devices can then implement the swarm robot collaborative search and rescue processing function based on multimodal fusion and NB-IoT communication through the provided interface. The invention will be further described below with reference to the accompanying drawings.

[0050] like Figures 1-11 As shown, this invention provides a collaborative search and rescue method for cluster robots based on multimodal fusion and NB-IoT communication. The method includes the following steps:

[0051] S01. Generate and acquire first data corresponding to the post-disaster environment; wherein, the first data is multimodal data, including laser point cloud data, visual image data and IMU data;

[0052] S02. Combining the tightly coupled multimodal SLAM front-end processing algorithm, the first data is spatiotemporally aligned and fused to construct a first map corresponding to the post-disaster environment in real time, and generate second data corresponding to the first map; wherein, the first map is local environment sub-map data; the second data is its own pose data;

[0053] S03. Based on the first map and the second data, and combined with the map overlap detection algorithm, determine in real time whether there is an overlapping area between the first maps, and generate third data corresponding to the overlapping area; wherein, the third data is a relative pose transformation relationship;

[0054] S04. Based on the third data, the first map is processed by real-time coordinate transformation and stitching, and a second map corresponding to the first map is generated. At the same time, the robot is controlled to perform collaborative search and rescue operations based on the second map and in conjunction with the NB-IoT network. The second map is a complete environmental map that corresponds to the post-disaster environment and has been optimized.

[0055] The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes:

[0056] S021. By minimizing the residual terms of laser, vision, and IMU, the second data corresponding to the tightly coupled multimodal processing is calculated and generated; wherein, the calculation formula is:

[0057] (2)

[0058] In the formula, Let be the rigid body transformation matrix to be optimized, which belongs to a special three-dimensional Euclidean group; The number of laser observations; The number of visual observations; The number of observations by the IMU; It is laser detection residual; It is visual residual; It is the IMU residual; It is the Maharanobi norm.

[0059] The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes:

[0060] S022. Construct a first model corresponding to the first map; wherein, the first model is a lidar residual model; the calculation formula is as follows:

[0061] (3)

[0062] In the formula, For the current frame number One laser point, sensor coordinate system; For the corresponding point on the map, in the world coordinate system; for Map normal vector at that location; The pose transformation to be optimized; The symbol represents the change operation. .

[0063] The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes:

[0064] S023. Construct a second model corresponding to the first map; wherein, the second model is a visual reprojection residual model; the calculation formula is as follows:

[0065] (5)

[0066] In the formula, For the first 3D landmarks, world coordinate system; These are the 2D pixel coordinates observed in the image; For camera projection function:

[0067] (6)

[0068] In the formula, To transform the world coordinates to the camera coordinate system.

[0069] The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes:

[0070] S024. Construct a third model corresponding to the first map; wherein, the third model is an IMU pre-integration residual model; the calculation formula is as follows:

[0071] (8)

[0072] In the formula, For IMU pre-integral quantity from arrive ; For the attitude, speed, and position at any given moment; Let gravitational force be the vector, in world coordinate system; To map the SO(3) matrix to the so(3) Lie algebra

[0073] Specifically, in this embodiment of the invention, a quadruped robot [1] In post-disaster search and rescue scenarios, its biomimetic design and multi-degree-of-freedom motion capabilities enable it to adapt to the complex and uncertain post-disaster environment. Structurally, such as... Figure 1 As shown, each foot is equipped with three servo joints, enabling omnidirectional gait. This structure boasts high load scalability, with a load-to-weight ratio of up to 1:1, and features standardized interfaces on the back, such as ROS2 communication, DC power interface, and USB interface, allowing for the integration of LiDAR, robotic arms, cameras, sensors, wireless communication modules, and more.

[0074] This study employs multi-sensor data sensing, which is then used to develop an adaptive motion method based on rhythmic gait. This method addresses issues requiring comprehensive consideration of body posture, reachability space constraints, foot trajectory planning, and stable foot placement during movement. The overall approach based on rhythmic gait is as follows: Figure 2 As shown.

[0075] There are two core aspects: gait switching via the central pattern generator and adaptive environmental adjustment. An improved Matsuoka oscillator network is employed to generate phase-coupled rhythmic signals through nonlinear differential equations. The formula for calculating the rhythmic signal is as follows:

[0076] (1)

[0077] The symbol definitions and physical meanings are shown in Table 1.

[0078] Table 1. Definitions and Physical Meanings of Rhythm Signal Formulas

[0079]

[0080] Mapping technology employs SLAM (Simultaneous Localization and Mapping), currently using either single-lidar SLAM or single-vision SLAM. However, these methods have limitations in unpredictable post-disaster environments. Therefore, this solution proposes a multimodal information fusion data acquisition method. For example... Figure 3 As shown, the flow structure diagram of environmental data collected by LiDAR, binocular cameras, and sensors (such as inertial measurement units, IMU) is used to finally draw a multimodal map through multi-party spatiotemporal alignment, front-end tight coupling, and back-end optimization.

[0081] The core key technology lies in optimizing the tight coupling of multimodal front-ends. The key steps and pseudocode are as follows:

[0082] Step 1: System Initialization

[0083] (1) Align the timestamps of all sensors.

[0084] (2) Make a rough estimate of IMU bias and gravity direction.

[0085] (3) Initialize an initial map and pose using binocular vision or laser SLAM.

[0086] Step 2: Front-end processing, preparing data for back-end optimization. The pseudocode is as follows:

[0087] When a new sensor data packet (image + point cloud + IMU) arrives: / / Pseudocode: Front-end data processing FUNCTION processFrame(images, point_cloud, imu_buffer) / / 1. Visual feature extraction and tracking visual_features = extractAndTrackORBFeatures(images) / / 2. Laser point cloud processing filtered_point_cloud = removeGroundAndDownsample(point_cloud) current_lidar_frame = extractEdgeAndPlanarFeatures(filtered_point_cloud) / / Extract line and surface features / / 3. IMU pre-integration (from the previous keyframe to the current frame) IF last_keyframe_exists THEN delta_p,delta_v,delta_q,covariance=imuPreintegration(last_keyframe.time, current_time, imu_buffer) END IF / / 4. Keyframe determination IF isKeyFrame(visual_features, current_pose_estimate) THEN new_keyframe = createNewKeyFrame(current_pose_estimate, `visual_features, current_lidar_frame, imu_preintegrated) addToSlidingWindow(new_keyframe) triggerBackendOptimization() / / Trigger backend tight coupling optimization ENDIFEND FUNCTION`

[0088] Step 3: Backend tight coupling optimization, the pseudocode is as follows:

[0089] Optimization is triggered when the number of keyframes in the sliding window reaches the limit or when a new keyframe is added. / / Pseudocode: Backend Tightly Coupled Optimization FUNCTION tightlyCoupledOptimization(sliding_window) / / Construct the optimization problem problem = CeresOptimizationProblem() / / First add a prior factor (if the window edge-dives old keyframes) IF has_prior THENproblem.addPriorFactor(prior_error) END IF / / Iterate through all keyframes in the sliding window FOR i = 0 TO N-1 IN sliding_window / / 1. Add IMU factor IF i>0 THEN / / Get the IMU pre-integrated value from keyframe i-1 to i imu_factor = createIMUFactor(states[i-1], states[i], imu_preintegrated[i])problem.addResidualBlock(imu_factor) END IF / / 2. Add visual factor FOREACH visual_feature IN keyframe[i].features / / Find other keyframes that can also observe this feature point co_visible_frames = findFramesObservingSameFeature(visual_feature.id) / / Create visual reprojection error factor visual_factor = createReprojectionErrorFactor(states[i], ... , co_visible_frames,visual_feature) problem.addResidualBlock(visual_factor) END FOR / / 3. Add laser factor / / Match the point cloud features of the current keyframe with the local map constructed from other keyframes in the window local_map = buildLocalMapFromWindow(sliding_window,exclude_frame=i) lidar_factor = createLidarEdgePlanarFactor(states[i], keyframe[i].lidar_features, local_map) problem.addResidualBlock(lidar_factor) END FOR / / Set parameter block (states to be estimated) FOR i = 0 TON-1 IN sliding_window problem.addParameterBlock(states[i].position) problem.addParameterBlock(states[i].orientation) / / Use quaternions or SO(3) manifold problem.addParameterBlock(states[i].velocity) problem.addParameterBlock(states[i].imu_bias) END FOR / / Perform optimization options.max_solver_time = 0.05 / / Control optimization time to ensure real-time solver.summary = problem.solve(options) / / 4. Marginalization / / If the window is full, marginalize the oldest keyframe and convert its information into a prior factor to apply to the remaining window IF sliding_window.isFull() THEN prior_factor = marginalizeOldestKeyFrame(sliding_window) setPrior(prior_factor) END IF / / Update the state of the sliding window: updateStatesInSlidingWindow(problem.getOptimizedStates()) END FUNCTION.

[0090] Step 4: Map Building

[0091] The optimized keyframe poses and point clouds / features were used to update the global map. / / Pseudocode: Map Building FUNCTION updateGlobalMap(optimized_sliding_window) FOR EACH optimized_keyframe INoptimized_sliding_window / / Convert the point cloud in the optimized pose to world coordinates and add it to the global map global_map.update(optimized_keyframe.pose, optimized_keyframe.point_cloud) END FOR / / Map trimming and voxel filtering may be needed to control the scale global_map.voxelFilter() END FUNCTION

[0092] Key technical points:

[0093] (1) The core of tight coupling: In an optimization problem, minimize the IMU pre-integration residual, visual reprojection residual and laser matching residual at the same time.

[0094] (2) IMU pre-integration: This is the key technology to achieve tight coupling of VIO. It decouples IMU dynamics from pose estimation, which greatly improves efficiency.

[0095] (3) Sliding window and marginalization: In order to balance accuracy and computational cost, a sliding window must be used. Marginalization technology ensures that the information of old frames that are removed from the window is not lost, thus maintaining the consistency of the system.

[0096] (4) Robustness handling: In post-disaster environments, sensors may fail (e.g., vision encounters smoke, laser encounters glass). Adding robust kernel functions (such as Huber Loss) and performing chi-square tests on factors to eliminate outliers is crucial in the pseudocode.

[0097] (5) Representative systems: Open source projects such as LIO-SAM, VINS-Mono, and ORB-SLAM3 have adopted some or all of the above ideas. The solution can be seen as an extension and combination of them, especially in the deep integration of laser and vision features at the front end.

[0098] The optimal pose transformation matrix T* is solved by minimizing the sum of the residuals from laser, vision, and IMU, as shown in the following formula: (2)

[0099] Explanation of formula symbols:

[0100] Table 2. Mathematical Definitions and Physical Meanings of Tightly Coupled Multi-Factor Graph Optimization Formulas

[0101]

[0102] Detailed modeling of the residual term, where elidar,k is the lidar residual, is shown in the following mathematical model, i.e., the lidar residual mathematical model formula:

[0103] (3)

[0104] Variable descriptions: pk∈R3: the kth laser point in the current frame (sensor coordinate system); qk∈R3: the corresponding point in the map (world coordinate system); nk∈R3: the map normal vector (unit vector) at qk; T∈SE(3): the pose transformation to be optimized; ∘ symbol: transformation operation, T∘p=Rp+t;

[0105] Physical meaning: Minimize the vertical distance between the current laser point pk after transformation and the map plane (with qk as the base point and normal vector nk).

[0106] covariance matrix Σ l The construction of the covariance matrix, i.e., the formula for constructing the covariance matrix:

[0107] (4)

[0108] Variable descriptions: σd: ranging noise along the normal direction (typically 0.01~0.05m); σp: position noise in the tangential plane (affected by beam angle);

[0109] The mathematical model for the visual reprojection residual is shown below, i.e., the formula for the visual reprojection residual mathematical model:

[0110] (5)

[0111] Variable descriptions: Xm∈R3: the m-th 3D landmark (world coordinate system); zm∈R2: the observed 2D pixel coordinates in the image; π(⋅): camera projection function (camera projection function matrix).

[0112] (6)

[0113] T⋅X: Transforms world coordinates to camera coordinates.

[0114] Physical meaning: Minimize the deviation between the projected position of the 3D landmark point and the measured pixel position.

[0115] Covariance matrix Σv, visual reprojection residual covariance matrix:

[0116] (7)

[0117] EIMU is the IMU pre-integration residual. The mathematical model matrix formula for the IMU pre-integration residual is shown below:

[0118] (8)

[0119] Variable descriptions: ΔRn,n+1,Δvn,n+1,Δpn,n+1 Δ: IMU pre-integral (from tn to tn+1); attitude, velocity, and position at times Rn,vn,pn; g: gravity vector (world coordinate system); Log(⋅): mapping the SO(3) matrix to the so(3) Lie algebra.

[0120] Physical meaning: Rotation residual: the difference between the relative rotation predicted by the IMU and the pose estimate; Velocity residual: the deviation between the velocity change predicted by the IMU and the gravity correction; Position residual: the deviation between the displacement predicted by the IMU and the kinematic model; Covariance matrix Σi: obtained through IMU noise parameters (gyroscope angle random walk σg, accelerometer random walk σa) and pre-integration theory propagation (mathematical formula for IMU covariance matrix):

[0121] (9)

[0122] Variable Explanation: F is the Jacobian matrix of pre-integration error versus IMU noise; a multimodal information fusion method based on NB-IoT communication module technology was adopted. An NB-IoT module was installed on the robot. NB-IoT technology was used for communication. The PCB layout of the NB-IoT communication module includes the core circuitry:

[0123] like Figure 4 As shown, the STM32L151C8T6D chip is used in the communication module, serving as the main control chip for the NB-IoT module. Employing a low-power MCU, the NB-IoT terminal module has a standby current of only 5μA and a continuous operating power consumption of approximately 100mW, significantly reducing power consumption compared to 4G modules. In this robot, this can extend battery life by approximately 30% to 50%.

[0124] like Figure 5 As shown, this is an NB-IoT module, using Lierda's NB861 chip. Its features include ultra-low power consumption, stronger penetration than mainstream wireless communication technologies like WiFi, and lower power consumption.

[0125] like Figure 6 The diagram shows the SIM card slot circuit, which allows SIM cards to be inserted into the NB-IoT module. Each SIM card corresponds to a robot's communication module, enabling multimodal information fusion of cluster robots.

[0126] like Figure 7 As shown, this is a signal receiving module used for acquiring data using different communication protocols. This module has an analog-to-digital converter (ADC) that can acquire analog data. It has an RS485 standard interface and digital input / output interfaces.

[0127] like Figure 8 As shown, by using NB-IoT technology, multiple robots can be connected, and through multimodal information fusion methods, the local environment maps constructed by the SLAM technology of multiple robots are stitched together into a complete map. Multiple image modalities, such as visible light images, infrared images, multispectral images, and depth images, are fused to obtain more accurate target detection, recognition, and classification results.

[0128] An overlap detection mechanism was implemented, enabling environmental map fusion when overlapping areas are detected. A single robot fuses geometric features from LiDAR with textural features from visual sensors to create a unified "multimodal map signature," constructing a multimodal map. Furthermore, multiple robots can also detect overlapping areas. These overlapping detections, along with shared landmark points and poses, are then combined using a global optical speed adjustment optimization process, with each robot's sub-maps stitched together to reduce detection errors.

[0129] Multi-robot visual SLAM map fusion framework such as Figure 7 As shown, each robot first executes its own SLAM, establishing its bag-of-words vector set and sub-map, and continuously performs map overlap detection algorithms. Once map overlap detection is successful, the PnP algorithm result is used as the initial pose, and the ICP algorithm is employed for accurate relative pose calculation. Finally, coordinate transformation and map stitching are used to achieve multi-robot environmental map fusion.

[0130] This invention utilizes the VREP simulation platform to build a quadruped robot model. Using MATLAB in conjunction with VREP simulation, the proposed adaptive motion control method is experimentally verified. The quadruped robot achieves body position and attitude tracking, enabling it to complete complex climbing and traversing rugged terrain, achieving the expected motion effects. By comparing the changes in body pose during normal gait and adaptive gait traversing rugged terrain, such as... Figure 9 As shown, under normal gait, the maximum pitch angle of the fuselage reached 2.6303° and the maximum roll angle reached 2.8628°; while under adaptive gait, the maximum pitch angle of the fuselage was only 0.7014° and the roll angle was 0.6061°, which shows that adaptive motion can effectively stabilize the fuselage.

[0131] The core idea of ​​adaptive algorithms is to upgrade the control loop from simple trajectory tracking to an intelligent adaptive system that includes "perception-decision-execution".

[0132] Top-level control loop pseudocode / / Pseudocode: Main control loop FUNCTION mainControlLoop() WHILE robotIsRunning: / / 1. Perception layer: Get multimodal state information current_body_pose, current_body_velocity = getBodyIMU() foot_contact_state = [False, False, False,False] / / Initialize quadrupedal contact state foot_force = [0.0, 0.0, 0.0, 0.0] / / Initialize quadrupedal force sensor data FOR i IN range(4): / / Traverse the four legs foot_force[i]=getFootForceSensor(i) / / Innovation point 1: Force-based contact state judgment is more robust than simple position judgment IF foot_force[i]>CONTACT_FORCE_THRESHOLD: foot_contact_state[i] = True END IF END FOR / / 2. Decision layer: State evaluation and gait phase decision desired_gait_phase = planDefaultGaitPhase() / / Plan the default gait phase (such as CPG) / / Innovation 2: Gait adjustment based on body state deviation body_pose_error = calculatePoseError(current_body_pose, desired_body_pose) adaptive_gait_phase = adaptGaitBasedOnBodyPose(desired_gait_phase, body_pose_error, foot_contact_state) / / 3. Execution layer: Adaptive foot trajectory generation FOR i INrange(4): / / Innovation 3: Generate foot trajectory in real time based on the adjusted gait and contact state desired_foot_position[i]=generateAdaptiveSwingTrajectory(i,adaptive_gait_phase, body_pose_error) END FOR / / 4.Underlying control: Whole-body control WBC joint_torques = wholeBodyController(desired_body_pose, desired_foot_position, current_body_pose, foot_contact_state) sendTorquesToActuators(joint_torques) sleep(control_cycle_time) ENDWHILE END FUNCTION.

[0133] Innovative Detail 1: Gait Phase Adaptation Based on Contact State and Airframe Attitude

[0134] This is the core of achieving stability. When the fuselage tilts unexpectedly, instead of waiting for the entire gait cycle to end, the gait is immediately fine-tuned. / / Pseudocode: Gait Adaptation Based on Body Attitude FUNCTION adaptGaitBasedOnBodyPose(nominal_phase, body_pose_error, foot_contact_state) / / Input: Nominal gait phase, body attitude error (pitch / roll), foot contact state / / Output: Adjusted adaptive gait phase adaptive_phase = nominal_phase / / Initialize to nominal phase / / Strategy 1: Emergency Attitude Stabilization - If the attitude error exceeds the threshold, prioritize stabilization IF abs(body_pose_error.pitch)>MAX_PITCH_ERROR OR abs(body_pose_error.roll)>MAX_ROLL_ERROR: / / Innovative Details: End the swing phase early and enter the support phase to form a stable support polygon FORi IN range(4): / / If the leg is swinging (not in contact with the ground) and the current attitude requires it to provide support IF NOT foot_contact_state[i] AND isLegInSupportPolygon(i,body_pose_error): / / Force the leg's phase to be rapidly advanced to the support phase. adaptive_phase.leg_phase[i] = min(nominal_phase.leg_phase[i] + PHASE_ADVANCE_GAIN, 1.0) END IF END FOR END IF / / Strategy 2: Diagonal Support Enhancement - Actively Adjust Support Mode in Rough Terrain / / For example, if the fuselage is detected to tilt to the right (roll angle is positive) IF body_pose_error.roll > ROLL_TILT_THRESHOLD: / / Innovative Details: Lengthen the support phase of the left leg and shorten the support phase of the right leg to generate a righting torque. adaptive_phase = reinforceDiagonalSupport(adaptive_phase, body_pose_error.roll) ENDIF RETURN adaptive_phase END FUNCTION

[0135] Innovation Detail 2: Adaptive Foot Trajectory Generation

[0136] Traditional methods use a fixed swing trajectory. This method dynamically adjusts the trajectory height and landing point based on real-time conditions. / / Pseudocode: Adaptive Swing Trajectory Generation FUNCTION generateAdaptiveSwingTrajectory(leg_id, gait_phase, body_pose_error) / / Input: Leg ID, adaptive gait phase, fuselage attitude error / / Output: Expected foot position of this leg (in body coordinates) nominal_foot_pos = getNominalFootPosition(leg_id, gait_phase) / / If this leg is in the swing phase IF isLegInSwingPhase(gait_phase, leg_id): / / Innovative Detail 1: Terrain-Predictive Foot Lift - Predict the terrain ahead based on fuselage attitude and increase the leg lift height / / For example, if the fuselage pitch angle is positive (head up, tail down), it means that it is going uphill, and the front leg needs to be lifted higher terrain_clearance =calculateTerrainClearance(leg_id, body_pose_error) nominal_foot_pos.z = nominal_foot_pos.z + terrain_clearance / / Innovative Detail 2: Foot Placement Compensation - Actively adjusts the foot placement to maintain stability based on the robot's speed and tilt. / / The goal is to ensure that the foot's velocity relative to the ground is zero upon landing (smooth landing). body_velocity_compensation = calculateBodyVelocityCompensation(body_velocity) body_tilt_compensation = calculateBodyTiltCompensation(body_pose_error) compensated_foot_pos = nominal_foot_pos + body_velocity_compensation + body_tilt_compensation / / Ensures the adjusted foot placement is within the robot's kinematic limits. IFISFootPositionReachable(compensated_foot_pos, leg_id): RETURNcompensated_foot_pos ELSE: RETURN getReachableNearestPoint(compensated_foot_pos, leg_id) END IF ELSE: / / Support phase: The primary task of the foot is to provide support force, maintaining a relatively fixed position, but allowing for compliant control. RETURN nominal_foot_posEND IFEND FUNCTION.

[0137] A single robot sub-map, through map overlap detection and map fusion algorithms, can ultimately yield a complete map of a complex scene. For example... Figure 10 The image shows the results of multi-robot collaborative mapping. Figure 10 (a) Create a map of the scene around robot A. Figure 10 (b) Create a map for robot B navigating the pipeline. Figure 10 (c) shows the result of multi-robot map fusion.

[0138] Example: Multiple quadruped robots of this invention are deployed at the disaster site. Robot A enters through the entrance, and Robot B enters through the ventilation duct. Each robot utilizes its own sensors and SLAM algorithm to construct a local map. Figure 10 (-a, 10-b), and sends data packets containing feature points, key frames, and sub-map point clouds to the fusion server in the rear command center via the NB-IoT network.

[0139] The server detects overlapping areas (such as a shared room) in the maps of robots A and B, and then calculates the pose transformation matrix T of robot B relative to A. Using matrix T, the entire sub-map of robot B is transformed into the coordinate system of robot A. The two point clouds are merged, and duplicate points in the overlapping areas are eliminated through optimization, ultimately generating a complete environment map. Figure 10 -c).

[0140] Commanders can view the overall map through the monitoring terminal to locate the trapped personnel and then issue instructions to the nearest robot A via the communication network to guide it to the target location for further confirmation and rescue.

[0141] To achieve the above objectives, the present invention also provides a collaborative search and rescue system for swarm robots based on multimodal fusion and NB-IoT communication, such as... Figure 12 As shown, the system is applied to the collaborative search and rescue method for swarm robots based on multimodal fusion and NB-IoT communication. The system includes:

[0142] The data generation and acquisition unit is used to generate and acquire first data corresponding to the post-disaster environment; wherein, the first data is multimodal data, including laser point cloud data, visual image data and IMU data;

[0143] The first data processing and generation unit is used to combine a tightly coupled multimodal SLAM front-end processing algorithm to perform spatiotemporal alignment and fusion processing on the first data, construct a first map corresponding to the post-disaster environment in real time, and generate second data corresponding to the first map; wherein, the first map is local environment sub-map data; and the second data is its own pose data.

[0144] The second data processing and generation unit is used to determine in real time whether there is an overlapping area between the first maps based on the first map and the second data, combined with a map overlap detection algorithm, and to generate third data corresponding to the overlapping area; wherein, the third data is a relative pose transformation relationship.

[0145] The third data processing and generation unit is used to perform real-time coordinate transformation and stitching processing on the first map based on the third data, and generate a second map corresponding to the first map. At the same time, based on the second map and in conjunction with the NB-IoT network, it controls the robot to perform collaborative search and rescue operations. The second map is a complete environmental map that corresponds to the post-disaster environment and has been optimized.

[0146] The first data processing and generation unit further includes:

[0147] The first generation module is used to calculate and generate second data corresponding to the tightly coupled multimodal processing by combining the residual terms of laser, vision, and IMU data with minimization; the calculation formula is as follows:

[0148] (2)

[0149] In the formula, Let be the rigid body transformation matrix to be optimized, which belongs to a special three-dimensional Euclidean group; The number of laser observations; The number of visual observations; The number of observations by the IMU; It is the Maharanobi norm;

[0150] The first construction module is used to construct a first model corresponding to the first map; wherein, the first model is a lidar residual model; the calculation formula is as follows:

[0151] (3)

[0152] In the formula, For the current frame number One laser point, sensor coordinate system; For the corresponding point on the map, in the world coordinate system; for Map normal vector at that location; The pose transformation to be optimized; The symbol represents the change operation. ;

[0153] The second construction module is used to construct a second model corresponding to the first map; wherein, the second model is a visual reprojection residual model; the calculation formula is as follows:

[0154] (5)

[0155] In the formula, For the first 3D landmarks, world coordinate system; These are the 2D pixel coordinates observed in the image; For camera projection function:

[0156] (6)

[0157] In the formula, To transform world coordinates to camera coordinates;

[0158] The third construction module is used to construct a third model corresponding to the first map; wherein, the third model is an IMU pre-integration residual model; the calculation formula is as follows:

[0159] (8)

[0160] In the formula, For IMU pre-integral quantity from arrive ; For the attitude, speed, and position at any given moment; Let gravitational force be the vector, in world coordinate system; To map the SO(3) matrix to the so(3) Lie algebra

[0161] The system also includes multiple quadruped robot platforms, a multimodal information fusion processing center, and a remote monitoring terminal;

[0162] The quadruped robot platform includes a robot body, a main control module, a motion control module, a sensor module, and an NB-IoT communication module. The robot body has a biomimetic quadruped structure, with three servo joints on each leg. The robot's back is equipped with a standardized expansion interface for mounting the sensor module and the robotic arm. The standardized expansion interface includes a ROS2 communication interface, a DC power interface, and a USB interface.

[0163] The sensor module includes at least a lidar, a binocular camera, and an inertial measurement unit;

[0164] The NB-IoT communication module is used to establish low-power, high-penetration communication connections between robots and between robots and the multimodal information fusion processing center, transmitting multimodal sensor data, local map data, and status information; wherein, the multimodal information fusion processing center is used to receive data from each robot, perform multi-robot SLAM map fusion, and generate a globally consistent map.

[0165] The remote monitoring terminal is used to receive and display the globally consistent map and robot status, and to send control commands to the robot.

[0166] In the system solution embodiment of the present invention, the specific details of the method steps involved in the collaborative search and rescue of cluster robots based on multimodal fusion and NB-IoT communication have been described above. That is to say, the functional modules in the system are used to implement the steps or sub-steps in the above method embodiment, which will not be repeated here.

[0167] To achieve the above objectives, the present invention also provides a collaborative search and rescue processing platform for swarm robots based on multimodal fusion and NB-IoT communication, such as... Figure 13 As shown, the device includes a processor, a memory, and a control program for a swarm robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication. The processor executes the control program, which is stored in the memory. This control program implements the steps of the swarm robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication. Additionally, an apparatus for analyzing and calculating the safe thickness of pipe roof grouting in the disturbed zone of a tunnel face is provided. This apparatus includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the steps of the pipe roof grouting safe thickness analysis method. The specific details of these steps have been described above and will not be repeated here.

[0168] In this embodiment of the invention, the processor built into the swarm robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication can be composed of integrated circuits. For example, it can be composed of a single packaged integrated circuit, or multiple integrated circuits packaged with the same or different functions. This includes combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor connects to various components using various interfaces and lines, and executes programs or units stored in memory, as well as calling data stored in memory, to perform various functions and process data for swarm robot collaborative search and rescue based on multimodal fusion and NB-IoT communication.

[0169] The memory is used to store program code and various data. It is installed in the collaborative search and rescue processing platform for swarm robots based on multimodal fusion and NB-IoT communication, and enables high-speed and automatic access to programs or data during operation. The memory includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0170] To achieve the above objectives, the present invention also provides a computer-readable storage medium, such as... Figure 14 As shown, the computer-readable storage medium stores a control program for a cluster robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication. This control program implements the steps of the cluster robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication. Specifically, it provides a computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the computer program implements the step of analyzing the safe thickness of the pipe roof grouting. The specific details of these steps have been described above and will not be repeated here.

[0171] In the description of embodiments of the present invention, it should be noted that any process or method description in the flowcharts or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0172] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, a “computer-readable medium” can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0173] This invention generates and acquires first data corresponding to a post-disaster environment through a method; wherein the first data is multimodal data, including laser point cloud data, visual image data, and IMU data; combining a tightly coupled multimodal SLAM front-end processing algorithm, the first data is spatiotemporally aligned and fused to construct a first map corresponding to the post-disaster environment in real time, and second data corresponding to the first map is generated; wherein the first map is local environment sub-map data; the second data is its own pose data; based on the first map and the second data, and combined with a map overlap detection algorithm, it is determined in real time whether there is an overlapping area between the first maps, and third data corresponding to the overlapping area is generated; its In this method, the third data represents the relative pose transformation relationship. Based on the third data, the first map is processed by real-time coordinate transformation and stitching, and a second map corresponding to the first map is generated. Simultaneously, based on the second map and combined with the NB-IoT network, the robot is controlled to perform collaborative search and rescue operations. The second map is a complete environmental map corresponding to the post-disaster environment and optimized. The method also includes a corresponding system, platform, and storage medium. Through a biomimetic quadruped robot platform, adaptive motion control based on a central pattern generator (CPG), a tightly coupled multimodal SLAM front-end, and collaborative communication and map fusion based on NB-IoT, the robot can achieve efficient, stable, and collaborative operation in uncertain environments.

[0174] In other words, the present invention effectively solves the core pain points of large perception blind spots, poor motion stability, and difficulty in multi-robot collaborative communication in uncertain post-disaster environments by deeply integrating adaptive motion control of biomimetic quadruped robots, laser-vision-IMU tightly coupled SLAM front-end optimization, and NB-IoT low-power wide-area communication technology. It achieves highly robust autonomous localization, environmental modeling, and collaborative search and rescue of multiple robots in extreme environments, significantly improving search and rescue efficiency and success rate.

[0175] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A collaborative search and rescue method for swarm robots based on multimodal fusion and NB-IoT communication, characterized in that, The method includes: Generate and acquire first data corresponding to the post-disaster environment; wherein, the first data is multimodal data, including laser point cloud data, visual image data and IMU data; By combining a tightly coupled multimodal SLAM front-end processing algorithm, the first data is spatiotemporally aligned and fused separately to construct a first map corresponding to the post-disaster environment in real time, and generate second data corresponding to the first map; wherein, the first map is local environment sub-map data; and the second data is its own pose data; Based on the first map and the second data, and combined with the map overlap detection algorithm, it is determined in real time whether there is an overlapping area between the first maps, and third data corresponding to the overlapping area is generated; wherein, the third data is a relative pose transformation relationship; Based on the third data, the first map is processed by real-time coordinate transformation and stitching, and a second map corresponding to the first map is generated. At the same time, based on the second map and combined with the NB-IoT network, the robot is controlled to perform collaborative search and rescue operations. The second map is a complete environmental map that corresponds to the post-disaster environment and has been optimized.

2. The method for collaborative search and rescue of swarm robots based on multimodal fusion and NB-IoT communication according to claim 1, characterized in that, The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes: By minimizing the residuals from laser, vision, and IMU, second data corresponding to the tightly coupled multimodal processing is calculated; the calculation formula is as follows: （2） In the formula, Let be the rigid body transformation matrix to be optimized, which belongs to a special three-dimensional Euclidean group; The number of laser observations; The number of visual observations; The number of observations by the IMU; It is the Maharanobi norm.

3. The method for collaborative search and rescue of swarm robots based on multimodal fusion and NB-IoT communication according to claim 1 or 2, characterized in that, The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes: Construct a first model corresponding to the first map; wherein, the first model is a lidar residual model; the calculation formula is as follows: （3） In the formula, For the current frame number One laser point, sensor coordinate system; For the corresponding point on the map, in the world coordinate system; for Map normal vector at that location; The pose transformation to be optimized; The symbol represents the change operation. .

4. A collaborative search and rescue method for swarm robots based on multimodal fusion and NB-IoT communication according to claim 1 or 2, characterized in that, The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes: Construct a second model corresponding to the first map; wherein, the second model is a visual reprojection residual model; the calculation formula is as follows: （5） In the formula, For the first 3D landmarks, world coordinate system; These are the 2D pixel coordinates observed in the image; For camera projection function: （6） In the formula, To transform the world coordinates to the camera coordinate system.

5. A collaborative search and rescue method for swarm robots based on multimodal fusion and NB-IoT communication according to claim 1 or 2, characterized in that, The combined tightly coupled multimodal SLAM front-end processing algorithm, which performs spatiotemporal alignment and fusion processing on the first data to construct a first map corresponding to the post-disaster environment in real time and generate second data corresponding to the first map, further includes: Construct a third model corresponding to the first map; wherein, the third model is an IMU pre-integration residual model; the calculation formula is as follows: （8） In the formula, For IMU pre-integral quantity from arrive ; For the attitude, speed, and position at any given moment; Let gravitational force be the vector, in world coordinate system; To map the SO(3) matrix to the so(3) Lie algebra 6. A collaborative search and rescue system for swarm robots based on multimodal fusion and NB-IoT communication, characterized in that, The system is applied to the collaborative search and rescue method for swarm robots based on multimodal fusion and NB-IoT communication as described in any one of claims 1 to 5, and the system includes: The data generation and acquisition unit is used to generate and acquire first data corresponding to the post-disaster environment; wherein, the first data is multimodal data, including laser point cloud data, visual image data and IMU data; The first data processing and generation unit is used to combine a tightly coupled multimodal SLAM front-end processing algorithm to perform spatiotemporal alignment and fusion processing on the first data, construct a first map corresponding to the post-disaster environment in real time, and generate second data corresponding to the first map; wherein, the first map is local environment sub-map data; and the second data is its own pose data. The second data processing and generation unit is used to determine in real time whether there is an overlapping area between the first maps based on the first map and the second data, combined with a map overlap detection algorithm, and to generate third data corresponding to the overlapping area; wherein, the third data is a relative pose transformation relationship. The third data processing and generation unit is used to perform real-time coordinate transformation and stitching processing on the first map based on the third data, and generate a second map corresponding to the first map. At the same time, based on the second map and in conjunction with the NB-IoT network, it controls the robot to perform collaborative search and rescue operations. The second map is a complete environmental map that corresponds to the post-disaster environment and has been optimized.

7. The collaborative search and rescue system for cluster robots based on multimodal fusion and NB-IoT communication according to claim 6, characterized in that, The first data processing and generation unit further includes: The first generation module is used to calculate and generate second data corresponding to the tightly coupled multimodal processing by combining the residual terms of laser, vision, and IMU data with minimization; the calculation formula is as follows: （2） In the formula, Let be the rigid body transformation matrix to be optimized, which belongs to a special three-dimensional Euclidean group; The number of laser observations; The number of visual observations; The number of observations by the IMU; It is the Maharanobi norm; The first construction module is used to construct a first model corresponding to the first map; wherein, the first model is a lidar residual model; the calculation formula is as follows: （3） In the formula, For the current frame number One laser point, sensor coordinate system; For the corresponding point on the map, in the world coordinate system; for Map normal vector at that location; The pose transformation to be optimized; The symbol represents the change operation. ; The second construction module is used to construct a second model corresponding to the first map; wherein, the second model is a visual reprojection residual model; the calculation formula is as follows: （5） In the formula, For the first 3D landmarks, world coordinate system; These are the 2D pixel coordinates observed in the image; For camera projection function: （6） In the formula, To transform world coordinates to camera coordinates; The third construction module is used to construct a third model corresponding to the first map; wherein, the third model is an IMU pre-integration residual model; the calculation formula is as follows: （8） In the formula, For IMU pre-integral quantity from arrive ; For the attitude, speed, and position at any given moment; Let gravitational force be the vector, in world coordinate system; To map the SO(3) matrix to the so(3) Lie algebra 8. A collaborative search and rescue system for swarm robots based on multimodal fusion and NB-IoT communication as described in claim 6 or 7, characterized in that, The system also includes multiple quadruped robot platforms, a multimodal information fusion processing center, and a remote monitoring terminal; The quadruped robot platform includes a robot body, a main control module, a motion control module, a sensor module, and an NB-IoT communication module. The robot body has a biomimetic quadruped structure, with three servo joints on each leg. The robot's back is equipped with a standardized expansion interface for mounting the sensor module and the robotic arm. The standardized expansion interface includes a ROS2 communication interface, a DC power interface, and a USB interface. The sensor module includes at least a lidar, a binocular camera, and an inertial measurement unit; The NB-IoT communication module is used to establish low-power, high-penetration communication connections between robots and between robots and the multimodal information fusion processing center, transmitting multimodal sensor data, local map data, and status information; wherein, the multimodal information fusion processing center is used to receive data from each robot, perform multi-robot SLAM map fusion, and generate a globally consistent map. The remote monitoring terminal is used to receive and display the globally consistent map and robot status, and to send control commands to the robot.

9. A collaborative search and rescue platform for swarm robots based on multimodal fusion and NB-IoT communication, characterized in that, The system includes a processor, a memory, and a control program for a swarm robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication. The processor executes the control program, which is stored in the memory. The control program implements the swarm robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a control program for a cluster robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication. The control program for the cluster robot collaborative search and rescue processing platform based on multimodal fusion and NB-IoT communication implements the cluster robot collaborative search and rescue processing method based on multimodal fusion and NB-IoT communication as described in any one of claims 1 to 5.

Images