A multimodal inspection robot swarm control method

Abstract

This invention relates to the field of robot control technology, specifically a multimodal inspection robot swarm control method. The method includes: collecting heterogeneous environmental data such as laser point cloud sequences and visible light video streams through multiple sensing terminals; performing spatiotemporal alignment and feature-level fusion on these data to construct a unified spatiotemporal perception map containing information such as a semantically defined 3D environmental model; based on this map, using an improved distributed model predictive control algorithm with rolling optimization based on an online learning interactive cost function, generating and executing local control command sequences for each robot; and receiving incremental perception data returned by the robots and asynchronously updating the unified spatiotemporal perception map and the interactive cost function. This method improves the comprehensiveness of swarm environmental perception and control accuracy, and enhances the coordination and dynamic adaptability of robot swarm collaborative operations.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method for controlling a multimodal inspection robot swarm. Background Technology

[0002] Multimodal inspection robot swarms are widely used in industrial inspection, park security and other fields. Existing swarm control methods mostly collect environmental data through a single or a few types of sensing terminals, failing to achieve collaborative utilization of multiple heterogeneous environmental data. Control algorithms mostly adopt traditional distributed control strategies, lacking dynamic optimization mechanisms based on real-time sensing data, and have not built a unified spatiotemporal sensing framework to support swarm collaborative control.

[0003] Existing methods collect environmental data of limited variety, lacking spatiotemporal alignment and feature-level fusion, thus failing to comprehensively reflect the environmental status of the inspection area and resulting in a lack of accurate environmental support for cluster control. Traditional distributed control algorithms do not incorporate online learning interaction cost functions, cannot perform rolling optimization based on inter-robot interactions and environmental changes, exhibit poor adaptability of local control command sequences, and do not utilize incremental sensing data returned by robots to dynamically update the control basis.

[0004] The inability to effectively integrate heterogeneous environmental data, the lack of a unified spatiotemporal perception framework, insufficient dynamic optimization capabilities of control algorithms, and inadequate utilization of incremental perception data result in low precision in collaborative control of robot clusters, poor coordination of action execution, inability to adapt to dynamic changes in complex inspection environments, and difficulty in achieving efficient and accurate cluster inspection operations. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal inspection robot cluster control method.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal inspection robot swarm control method, comprising: Heterogeneous environmental data is collected by various sensing terminals deployed in the inspection area. The heterogeneous environmental data includes laser point cloud sequences, visible light video streams, thermal images, and multi-source radio frequency signals. The collected heterogeneous environmental data is spatiotemporally aligned and feature-level fused to construct a unified spatiotemporal perception map, which includes a semantic three-dimensional model of the environmental structure, dynamic target trajectory, and radio frequency signal intensity distribution field. Based on the unified spatiotemporal perception map, an improved distributed model predictive control algorithm is used to generate local control command sequences for each multimodal inspection robot in the cluster. The improved distributed model predictive control algorithm is based on online learning interactive cost function for rolling optimization. The local control command sequence is sent to the corresponding multimodal inspection robot to drive the multimodal inspection robot to perform movement, observation and data acquisition actions; The system receives incremental perception data returned by each multimodal inspection robot after it performs an action, and uses the incremental perception data to asynchronously update the unified spatiotemporal perception map and the interaction cost function of the online learning.

[0007] As a further aspect of the present invention, the collected heterogeneous environmental data is spatiotemporally aligned and feature-level fused to construct a unified spatiotemporal perception map of the unified spatiotemporal perception map, including: For each frame of the laser point cloud sequence, its timestamp and sensor spatial pose are extracted, and each frame of laser point cloud is registered to the global coordinate system to form a dense three-dimensional point cloud map. Keyframe extraction and feature point matching are performed on the visible light video stream. A texture map is generated through visual synchronous positioning and mapping technology. The texture map is then projected and fused with the dense 3D point cloud map to generate a preliminary 3D model with texture. Using the thermal image, temperature anomaly areas on the surface of the preliminary 3D model are identified, and the temperature values ​​are associated with the preliminary 3D model as an independent layer. Time difference of arrival (TDOA) and signal fingerprint analysis are performed on the multi-source radio frequency signals to calculate the location and signal strength of the radio frequency signal sources and generate the radio frequency signal strength distribution field. The moving target contour indicated by the dynamic target trajectory is superimposed on the textured preliminary three-dimensional model in real time, and the radio frequency signal intensity distribution field is associated with the global coordinate system in the form of a field map, together forming the unified spatiotemporal perception map.

[0008] As a further aspect of the present invention, the improved distributed model predictive control algorithm performs rolling optimization based on an online-learned interactive cost function, including: At the beginning of each optimization cycle, the real-time status information and local targets of each multimodal inspection robot are obtained, and the dynamic environment sub-map around the multimodal inspection robot is extracted from the unified spatiotemporal perception map. Define an initial interaction cost function, which is composed of a weighted sum of a path tracking error term, a collision risk term, and an energy consumption term; During the rolling optimization process, a behavior interaction predictor based on a neural network model is introduced. The behavior interaction predictor receives predicted trajectory segments from other robots and outputs the predicted interaction conflict intensity value. The predicted interaction conflict intensity value is used as a dynamic weighting coefficient to adjust the collision risk term in the initial interaction cost function in real time, forming the online learning interaction cost function available in the current optimization cycle. Using the interactive cost function of the online learning as the objective, distributed rolling optimization is performed on the trajectory sequence and control sequence of each multimodal inspection robot to obtain the local control command sequence of each robot, ensuring that the optimization process converges within a finite number of iterations.

[0009] As a further aspect of the present invention, the behavior interaction predictor receives predicted trajectory segments from other robots and outputs predicted interaction conflict intensity values, including: Before each rolling optimization solution, the predicted trajectory fragments broadcast by neighboring multimodal inspection robots in the previous optimization cycle are collected from the communication link. The predicted trajectory fragments contain the planned position and attitude of the robots in the future finite time steps. Align fragments of the local robot's planned trajectory with fragments of all neighboring robot trajectories in the spatiotemporal domain to form a multidimensional spatiotemporal relation tensor; The multidimensional spatiotemporal relation tensor is input into a pre-trained convolutional recurrent neural network model, which analyzes the spatiotemporal correlation patterns between multiple robot trajectories. The output of the convolutional recurrent neural network model is a scalar conflict intensity coefficient between zero and one. The scalar conflict intensity coefficient represents the probability assessment value of spatiotemporal overlap or intrusion of the safe distance between robots within the time window covered by the predicted trajectory segment. The scalar conflict intensity coefficient is output as the predicted interaction conflict intensity value for the current optimization step, and used to adjust the cost function.

[0010] As a further aspect of the present invention, the local control command sequence is sent to the corresponding multimodal inspection robot to drive the multimodal inspection robot to perform movement, observation, and data acquisition actions, including: The local control command sequence containing velocity and angular velocity sequences is converted into the underlying drive signals for each joint motor. During the movement of the multimodal inspection robot, the data from its odometer and inertial measurement unit are read in real time and compared with the expected motion state of the control command to generate a motion compensation signal. According to the pre-set observation task nodes in the local control command sequence, the gimbal camera, lidar, thermal imager and radio frequency receiver on the multimodal inspection robot are triggered to perform a collaborative scanning action; The collaborative scanning action includes: the gimbal camera continuously zooms to capture images of the target area pointed to by the observation task node; the lidar performs synchronous rotational scanning; the thermal imager collects infrared radiation information of the corresponding area; and the radio frequency receiver performs full-band signal reception of a specific frequency band. All execution status logs and raw sensor data generated during the driving, motion compensation, scanning, and detection processes are time-stamped, encapsulated together with task node identifiers, and prepared for back transmission as the incremental sensing data.

[0011] As a further aspect of the present invention, the step of receiving incremental sensing data returned by each multimodal inspection robot after performing an action, and asynchronously updating the unified spatiotemporal perception map and the online learning interaction cost function using the incremental sensing data, includes: Receive data packets from various multimodal inspection robots and extract the raw sensor data with time stamps and observation task node information from them; The laser scanning data in the original sensing data is registered and fused with the three-dimensional point cloud map in the unified spatiotemporal perception map in real time to correct the holes or pose drift errors in the map. From the visible light video stream and thermal imaging image in the original sensing data, new dynamic targets or static targets whose state has changed are extracted, and the dynamic target trajectory layer and temperature anomaly layer in the unified spatiotemporal perception map are updated. The radio frequency signal intensity distribution field is updated using the data detected by the radio frequency receiver; The observation task node information, the actual executed control command sequence, and the final pose of the multimodal inspection robot are stored as a state transition tuple in the experience replay pool for periodically retraining the neural network model parameters on which the online learning interaction cost function depends.

[0012] As a further aspect of the present invention, the observation task node information, the actual executed control command sequence, and the final pose of the multimodal inspection robot are stored as a state transition tuple in an experience replay pool for periodically retraining the neural network model parameters on which the online learning interaction cost function depends, including: Multiple historical state transition tuples are extracted in batches from the experience replay pool using a uniform sampling method; For each state transition tuple, extract the environmental sub-image segment corresponding to the observation task node, the starting segment of the actual executed control command sequence, and the final multimodal inspection robot pose. An evaluation network is constructed using a feedforward neural network as the main body. The input of the evaluation network is the environmental sub-image segment and the starting segment of the control command sequence, and the output is the overall cost evaluation value of the decision sequence. Using supervised learning, the evaluation network is trained with the actual cost implied by the pose change between state transition tuples in adjacent time steps as the supervision signal, so that the evaluation value approximates the actual cost. After training converges, the network parameters from the input layer to the intermediate feature layer in the evaluation network are transferred to the dynamic weight adjustment module of the online learning interaction cost function, replacing the original model parameters of the behavior interaction predictor.

[0013] As a further aspect of the present invention, the method further includes the step of handling dynamic changes in cluster topology: Continuously monitor the heartbeat signal and network connection quality of each multimodal inspection robot, and build and maintain the cluster's communication topology in real time; When a new multimodal inspection robot is detected joining the cluster, or when a robot is detected to be offline due to a malfunction, the communication topology map is updated and broadcast to all online robots. Upon receiving the updated communication topology map, the robot automatically adjusts the optimization range of its distributed model predictive control algorithm to ensure that the set of new neighboring robots considered during optimization decisions is consistent with the communication topology map. The newly added robot obtains a snapshot of the unified spatiotemporal perception map at the current moment from the neighboring robots, and initializes its own local optimization problem based on this snapshot.

[0014] As a further aspect of the present invention, the newly added robot obtains a snapshot of the unified spatiotemporal perception map at the current moment from neighboring robots, and initializes its own local optimization problem based on this snapshot, including: Newly joined robots send map snapshot request messages to neighboring robots; After receiving the map snapshot request message, the neighboring robot extracts the sub-map data covering the local area where the new robot is located from the current version of the unified spatiotemporal perception map from its local storage, and encapsulates the sub-map data and its global timestamp into a snapshot data packet and sends it to the newly joined robot. The newly added robot receives and unpacks the snapshot data packet, loads the subgraph data into its own memory, and uses it as a valid representation of the current environment; The newly added robot immediately collects data from its surroundings using its onboard sensors, and then quickly matches and locates the collected data with the sub-map data to determine its initial pose in the unified spatiotemporal perception map. Based on the initial pose, within the boundary of the subgraph data, a reachable initial target point is set, and with the initial target point as the destination, the improved distributed model predictive control algorithm is started to generate the first local control command sequence of the newly added robot.

[0015] As a further aspect of the present invention, the method further includes a global task reassignment step: Set up a task priority queue to store various inspection tasks issued by external systems or discovered autonomously by robots. The inspection tasks include the geographical scope of the task, the task type, and the task deadline. Regularly assess the urgency of each task in the task queue, and evaluate the current workload, remaining battery power, and estimated time to reach the task area for each robot in the cluster. Taking task urgency, robot workload, robot remaining battery power, and robot arrival time as inputs, a global task allocation optimization calculation is run. The goal of the calculation is to maximize the total value of the tasks expected to be completed within the specified time. The global task allocation optimization calculation assigns a multimodal inspection robot that is most suitable for performing the inspection task to each unassigned inspection task, and calculates an optimal path for the assigned robot to reach the task area. The assignment relationship, task details, and optimal path are sent to the assigned robot. After completing the current local control command sequence, or immediately interrupting the interruptible non-urgent task upon receiving the task, the assigned robot inserts the newly received inspection task into its local optimization target and begins to move according to the optimal path.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: By collecting heterogeneous environmental data such as laser point cloud sequences, visible light video streams, thermal images, and multi-source radio frequency signals through various sensing terminals deployed in the inspection area, spatiotemporal alignment and feature-level fusion are performed on the collected heterogeneous environmental data to construct a unified spatiotemporal sensing map that includes a semantic 3D model of the environmental structure, dynamic target trajectories, and radio frequency signal intensity distribution fields. The comprehensive collection of multiple heterogeneous data can compensate for the limitations of single-data sensing, and spatiotemporal alignment and feature-level fusion can eliminate spatiotemporal deviations and redundancies between data. This allows the unified spatiotemporal sensing map to comprehensively and accurately reflect the environmental structure and dynamic changes of the inspection area, providing more comprehensive environmental support for cluster control. Compared with single-data sensing, it can improve the cluster's ability to perceive complex environments.

[0017] Based on a unified spatiotemporal perception map, an improved distributed model predictive control algorithm using an online-learning-based interactive cost function for rolling optimization is employed to generate local control command sequences for each multimodal inspection robot in the cluster. Simultaneously, incremental perception data is received from each robot after its actions, and this incremental data is used to asynchronously update the unified spatiotemporal perception map and the online-learning interactive cost function. The online-learning interactive cost function allows the control algorithm to adapt to the interaction relationships between robots and environmental changes in real time. The locally controlled command sequences generated by rolling optimization better match the actual operational needs of each robot. The asynchronous update mechanism enables the perception map and cost function to dynamically adjust along with the inspection process, avoiding control lag, enhancing the dynamic adaptability of cluster control, and improving the coordination and accuracy of robot cluster action execution. Attached Figure Description

[0018] Figure 1 This is a flowchart of a multimodal inspection robot cluster control method according to the present invention; Figure 2 Flowchart for constructing a unified spatiotemporal perception map; Figure 3 A flowchart for the rolling optimization of the improved distributed model predictive control algorithm. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] See Figure 1 This invention provides a multimodal inspection robot swarm control method, the specific method including: Heterogeneous environmental data is collected by multiple sensing terminals deployed in the inspection area. This heterogeneous environmental data includes laser point cloud sequences, visible light video streams, thermal images, and multi-source radio frequency signals. The collected heterogeneous environmental data is spatiotemporally aligned and feature-level fused to construct a unified spatiotemporal perception map. This unified spatiotemporal perception map includes a semantic 3D model of the environmental structure, dynamic target trajectories, and radio frequency signal intensity distribution fields. Based on the unified spatiotemporal perception map, an improved distributed model predictive control algorithm is used to generate a rolling sequence of local control commands for each multimodal inspection robot in the cluster. This improved distributed model predictive control algorithm is based on an online-learned interactive cost function for rolling optimization. The local control command sequence is then sent to the corresponding multimodal inspection robot, driving it to perform movement, observation, and data acquisition actions. Incremental perception data returned by each multimodal inspection robot after performing its actions is received, and the unified spatiotemporal perception map and the online-learned interactive cost function are asynchronously updated using this incremental perception data.

[0022] In one embodiment of the present invention, see [reference] Figure 2 For each frame of the laser point cloud sequence, its timestamp and sensor spatial pose are extracted, and each frame of laser point cloud is registered to the global coordinate system to form a dense three-dimensional point cloud map. Keyframe extraction and feature point matching are performed on the visible light video stream, and a texture map is generated through visual synchronous positioning and mapping technology. The texture map is then projected and fused with the dense three-dimensional point cloud map to generate a textured preliminary three-dimensional model. The thermal image is used to identify temperature anomaly areas on the surface of the preliminary three-dimensional model, and the temperature values ​​are associated with the preliminary three-dimensional model as an independent layer. The time difference of arrival (TDOA) and signal fingerprint analysis are performed on the multi-source radio frequency (RF) signals to calculate the location and signal strength of the RF signal sources, generating the RF signal strength distribution field. The moving target contour indicated by the dynamic target trajectory is superimposed onto the textured preliminary three-dimensional model in real time, and the RF signal strength distribution field is associated with the global coordinate system in the form of a field map to jointly constitute the unified spatiotemporal perception map.

[0023] In specific implementations, for each frame of the laser point cloud sequence, its timestamp and sensor spatial pose are extracted, and each frame of laser point cloud is registered to the global coordinate system to form a dense 3D point cloud map. In specific implementations, keyframe extraction and feature point matching are performed on the visible light video stream. Texture maps are generated using visual synchronous positioning and mapping techniques, and the texture maps are projected and fused with the dense 3D point cloud map to generate a textured preliminary 3D model. In some embodiments, thermal imaging is used to identify temperature anomaly areas on the surface of the preliminary 3D model, and temperature values ​​are associated and labeled as an independent layer with the preliminary 3D model. Time difference of arrival (TDOA) localization and signal fingerprint analysis are performed on multi-source radio frequency (RF) signals to calculate the location and signal strength of the RF signal sources, generating an RF signal intensity distribution field. The moving target contour indicated by the dynamic target trajectory is superimposed onto the textured preliminary 3D model in real time, and the RF signal intensity distribution field is associated with the global coordinate system in the form of a field map, together forming a unified spatiotemporal perception map.

[0024] In practice, the registration process of the laser point cloud sequence involves pose transformation. For a point in the k-th frame of the laser point cloud... Its coordinates in the global coordinate system It can be calculated using the following formula: in: The homogeneous transformation matrix of the k-th frame lidar relative to the global coordinate system is derived from the timestamp and the sensor's spatial pose. This represents the local coordinate vector of the i-th point in the laser point cloud of the k-th frame; This represents the homogeneous coordinate vector of the point in the global coordinate system after transformation. Optionally, keyframe extraction of the visible light video stream is based on image feature richness and inter-frame difference threshold. Feature point matching adopts the scale-invariant feature transformation method. Visual synchronous localization and mapping technology simultaneously estimates the camera pose and constructs a sparse feature point map. The projection fusion of texture mapping and 3D point cloud map maps pixels to a 3D point surface based on camera intrinsic and extrinsic parameters. It can be understood that the identification of temperature anomaly areas is based on the coordinate calibration of thermal images and visible light images. The overheated pixel areas in the thermal images are mapped to the corresponding triangular faces of the preliminary 3D model, and the average temperature value of each face is recorded in an independent layer. In some embodiments, the time difference of arrival (TDOA) localization of multi-source radio frequency signals uses at least three receiving stations to measure the TDOA of the same signal source, and combines the signal propagation speed to establish a system of equations to solve for the spatial coordinates of the signal source; signal fingerprint analysis constructs a fingerprint database by collecting signal strength at different locations, uses a matching algorithm to infer the possible location of unknown signal sources, and combines the results of both to generate a radio frequency signal strength distribution field covering the global coordinate system.

[0025] In one embodiment of the present invention, see [reference] Figure 3 At the beginning of each optimization cycle, the real-time state information and local targets of each multimodal inspection robot are acquired, and a dynamic environment sub-graph around the multimodal inspection robot is extracted from the unified spatiotemporal perception map. An initial interaction cost function is defined, which is composed of a weighted sum of path tracking error, collision risk, and energy consumption terms. During the rolling optimization process, a behavior interaction predictor based on a neural network model is introduced. The behavior interaction predictor receives predicted trajectory fragments from other robots and outputs the predicted interaction conflict intensity value. Before each rolling optimization solution, predicted trajectory fragments broadcast by neighboring multimodal inspection robots in the previous optimization cycle are collected from the communication link. These predicted trajectory fragments contain the robot's planned position and attitude within a finite future time step. The fragments of the local planned trajectory are aligned with the trajectory fragments of all neighboring robots in the spatiotemporal domain to form a multidimensional spatiotemporal relation tensor. The multidimensional spatiotemporal relation tensor is input into a pre-trained convolutional recurrent neural network. In the model, the convolutional recurrent neural network model analyzes the spatiotemporal correlation patterns between the trajectories of multiple robots. The output of the convolutional recurrent neural network model is a scalar conflict intensity coefficient between zero and one, which represents the probability assessment value of spatiotemporal overlap or intrusion of the safety distance between robots within the time window covered by the predicted trajectory segment. The scalar conflict intensity coefficient is used as the predicted interaction conflict intensity value output in the current optimization step to adjust the cost function. The predicted interaction conflict intensity value is used as a dynamic weight coefficient to adjust the collision risk term in the initial interaction cost function in real time, forming the online learning interaction cost function available in the current optimization cycle. With the online learning interaction cost function as the target, distributed rolling optimization is performed on the trajectory sequence and control sequence of each multimodal inspection robot to obtain the local control command sequence of each robot, ensuring that the optimization process converges within a finite number of iterations.

[0026] In practical implementation, at the beginning of each optimization cycle, the real-time state information and local targets of each multimodal inspection robot are acquired, and a dynamic environment sub-graph around the multimodal inspection robot is extracted from the unified spatiotemporal perception map. An initial interaction cost function is defined, consisting of a weighted sum of path tracking error, collision risk, and energy consumption terms. During the rolling optimization process, a behavior interaction predictor based on a neural network model is introduced. This predictor receives predicted trajectory fragments from other robots and outputs predicted interaction conflict intensity values. Before each rolling optimization solution, predicted trajectory fragments broadcast by neighboring multimodal inspection robots in the previous optimization cycle are collected from the communication link. These predicted trajectory fragments contain the robot's planned position and attitude within a finite future time step. The fragments of the local planned trajectory are aligned with the trajectory fragments of all neighboring robots in the spatiotemporal domain, forming a multidimensional spatiotemporal relation tensor. This multidimensional spatiotemporal relation tensor is input into a pre-trained convolutional recurrent neural network model, which analyzes the spatiotemporal correlation patterns between the trajectories of multiple robots. The output of the convolutional recurrent neural network model is a scalar conflict intensity coefficient between zero and one. This coefficient represents the probability assessment of spatiotemporal overlap or safety distance intrusion between robots within the time window covered by the predicted trajectory segment. The scalar conflict intensity coefficient is used as the predicted interaction conflict intensity value for the current optimization step, which is then used to adjust the cost function. The predicted interaction conflict intensity value is used as a dynamic weighting coefficient to adjust the collision risk term in the initial interaction cost function in real time, forming an online-learnable interaction cost function usable in the current optimization cycle. Using the online-learned interaction cost function as the objective, distributed rolling optimization is performed on the trajectory sequence and control sequence of each multimodal inspection robot to obtain the local control command sequence for each robot, ensuring that the optimization process converges within a finite number of iterations.

[0027] Optionally, the rolling optimization process involves solving a finite-time optimal control problem within each optimization cycle, based on the robot's current state and the environmental subgraph. This means finding the control sequence that minimizes the interaction cost function of online learning, and using the first control variable of this sequence as the actual control command issued. It can be understood that the convolutional recurrent neural network model used in the behavior interaction predictor extracts spatial proximity patterns between multiple trajectories through its convolutional layers, captures the trend of trajectory evolution over time through its recurrent layers, and finally maps the spatiotemporal features to a single scalar conflict intensity coefficient.

[0028] In some embodiments, the time length of the predicted trajectory segment is consistent with the prediction time domain of the distributed model predictive control algorithm to ensure that the interactive prediction covers the entire optimization interval. The dimensions of the multidimensional spatiotemporal relation tensor include the number of time steps, the number of robots participating in the prediction, and the feature dimensions required to describe the trajectory. The features may include the robot's planar coordinates, heading angle, and velocity components. Optionally, the calculation of the scalar conflict intensity coefficient considers not only the Euclidean distance between robots but also the robot's size radius and safety buffer distance. When the predicted minimum relative distance is less than the safety threshold, the conflict intensity coefficient approaches 1. It can be understood that applying the predicted interactive conflict intensity value as a dynamic weight coefficient to the collision risk term means that when a high-intensity interactive conflict is predicted, the weight of the collision risk term in the initial interactive cost function increases, forcing the optimization algorithm to generate a more conservative collision avoidance trajectory.

[0029] In one embodiment of the present invention, the local control command sequence containing velocity and angular velocity sequences is converted into low-level drive signals for each joint motor. During the movement of the multimodal inspection robot, data from its odometer and inertial measurement unit are read in real time and compared with the expected motion state of the control commands to generate motion compensation signals. According to the preset observation task nodes in the local control command sequence, the gimbal camera, lidar, thermal imager, and radio frequency receiver mounted on the multimodal inspection robot are triggered to perform a collaborative scanning action. The collaborative scanning action includes the gimbal camera performing continuous zoom shooting of the target area pointed to by the observation task node, the lidar performing synchronous rotation scanning, the thermal imager collecting infrared radiation information of the corresponding area, and the radio frequency receiver performing full-band signal reception of a specific frequency band. All execution status logs and raw sensor data generated during the driving, motion compensation, scanning, and reception processes are time-stamped, encapsulated together with the task node identifier, and prepared for transmission back as the incremental sensing data.

[0030] In practical implementation, the local control command sequence, including velocity and angular velocity sequences, is converted into low-level drive signals for each joint motor. During the movement of the multimodal inspection robot, data from its odometry and inertial measurement unit are read in real time and compared with the expected motion state of the control commands to generate motion compensation signals. Based on the pre-set observation task nodes in the local control command sequence, the gimbal camera, LiDAR, thermal imager, and RF receiver on the multimodal inspection robot are triggered to perform coordinated scanning actions. The coordinated scanning actions include the gimbal camera continuously zooming and capturing images of the target area pointed to by the observation task node, the LiDAR simultaneously performing a rotational scan, the thermal imager acquiring infrared radiation information of the corresponding area, and the RF receiver performing full-band signal reception in a specific frequency band. All execution status logs and raw sensor data generated during the driving, motion compensation, scanning, and reception processes are time-stamped, encapsulated together with the task node identifier, and prepared for feedback as incremental sensing data.

[0031] In some embodiments, when the multimodal inspection robot is in a power tunnel scenario, it receives a local control command sequence containing a control quantity sequence with a forward speed of 0.5 meters per second and a turning angular velocity of 0.1 radians per second. The robot chassis control system decomposes these control quantities into armature voltage and current commands for the left and right drive wheels through inverse kinematics calculations, and drives the motors to rotate via pulse width modulation. The robot's built-in odometer and nine-axis inertial measurement unit sample at a frequency of 100 Hz to monitor the robot's linear and angular displacement deviations in real time. If there is an accumulated error between the actual displacement fed back by the odometer and the expected displacement of the control command, the motion controller generates a compensation pulse based on the error integral term and adjusts the motor speed to achieve trajectory following with millimeter-level accuracy.

[0032] Optionally, observation task node information is embedded at specific time indices in the local control command sequence. When the robot reaches the designated point, the central processing unit triggers the coordinated operation of the multimodal sensors. The gimbal camera adjusts its pitch and yaw angles to lock onto the target power distribution cabinet, and the lens focal length is adaptively adjusted according to the target distance to ensure image clarity. The lidar performs a 360-degree horizontal scan at a frequency of 10 Hz to acquire high-resolution point cloud data of the surrounding environment. The thermal imager is aimed at the power distribution cabinet door area to capture thermal radiation images of the surface temperature distribution. The radio frequency receiver initiates spectrum scanning in the 2.4 GHz and 5.8 GHz bands to detect the signal strength and envelope characteristics of the wireless sensor nodes. The timing of multi-sensor data acquisition is aligned using hardware synchronization signals to ensure that all types of data have a consistent microsecond-level time reference.

[0033] It is understandable that the execution status log records actuator status variables such as motor drive current, actual travel speed, and gimbal rotation angle. The raw sensor data includes point cloud frames from the lidar, compressed image data streams from the camera, the radiation temperature matrix from the thermal imager, and the power spectral density array from the RF receiver. Time stamping uses a combination of GPS timing and an onboard high-precision clock to assign a unique absolute timestamp to each data record. Task node identifiers use a globally unique numbering rule to mark the inspection task and observation point to which the data belongs. The final encapsulated data packet uses a binary encoding format, including packet header information, a timestamp field, a task identifier, and a payload data segment, facilitating network transmission and backend parsing. Refer to Table 1, which shows a typical multimodal sensor collaborative scanning parameter configuration: Table 1: Multimodal Sensor Cooperative Scanning Parameter Configuration Table In practical implementation, the generation of motion compensation signals depends on the calculation of the deviation between the robot's kinematic model and actual measured values. For a two-wheeled differential drive robot, its instantaneous turning radius... It can be derived from the following formula: in: The instantaneous linear velocity scalar value representing the center point of the robot. This represents the instantaneous angular velocity scalar value of the robot around its vertical axis. When there is a significant difference between the actual turning radius measured by the odometry and the expected turning radius calculated based on the control commands, it indicates that the robot is slipping or experiencing uneven ground friction. In this case, the motion compensation signal will generate additional torque commands to correct the wheel speed and restore the expected motion trajectory.

[0034] In one embodiment of the present invention, data packets from various multimodal inspection robots are received, and time-stamped raw sensor data and observation task node information are parsed from them; the laser scanning data in the raw sensor data is registered and fused in real time with the 3D point cloud map in the unified spatiotemporal perception map to correct holes or pose drift errors in the map; new dynamic targets or static targets with changing states are extracted from the visible light video stream and thermal imaging in the raw sensor data, and the dynamic target trajectory layer and temperature anomaly layer in the unified spatiotemporal perception map are updated; the radio frequency signal intensity distribution field is updated using the data detected by the radio frequency receiver; the observation task node information, the actual executed control command sequence, and the final pose of the multimodal inspection robot are stored as a state transition tuple in the experience replay pool for periodically retraining the interaction cost function of the online learning based on the neural network. The network model parameters are used to extract multiple historical state transition tuples from the experience replay pool using uniform sampling. For each state transition tuple, the environmental sub-image segment corresponding to the observed task node, the starting segment of the actual executed control command sequence, and the final multimodal inspection robot pose are extracted. An evaluation network is constructed based on a feedforward neural network. The input of the evaluation network is the environmental sub-image segment and the starting segment of the control command sequence, and the output is the overall cost evaluation value of the decision sequence. Supervised learning is used, with the actual cost implied by the pose change between state transition tuples at adjacent time steps as the supervision signal, to train the evaluation network so that the evaluation value approximates the actual cost. After training convergence, the network parameters from the input layer to the intermediate feature layer in the evaluation network are transferred to the dynamic weight adjustment module of the online learning interaction cost function, replacing the original model parameters of the behavior interaction predictor.

[0035] In practice, data packets from various multimodal inspection robots are received, and time-stamped raw sensor data and observation task node information are parsed from them. Laser scan data from the raw sensor data is registered and fused in real time with the 3D point cloud map in the unified spatiotemporal perception map to correct holes or pose drift errors in the map. New dynamic targets or static targets with changing states are extracted from the visible light video stream and thermal imaging images in the raw sensor data, and the dynamic target trajectory layer and temperature anomaly layer in the unified spatiotemporal perception map are updated. The radio frequency signal intensity distribution field is updated using data detected by the radio frequency receiver. The observation task node information, the actual executed control command sequence, and the final pose of the multimodal inspection robot are stored as a state transition tuple in the experience replay pool for periodically retraining the neural network model parameters upon which the online learning interaction cost function depends.

[0036] In some embodiments, multiple historical state transition tuples are batched from the experience replay pool using uniform sampling. For each state transition tuple, the environmental sub-image segment corresponding to the observed task node, the starting segment of the actual executed control command sequence, and the final multimodal inspection robot pose are extracted. An evaluation network is constructed primarily using a feedforward neural network. The input to the evaluation network is the environmental sub-image segment and the starting segment of the control command sequence, and the output is the overall cost evaluation value of the decision sequence. Using supervised learning, the evaluation network is trained with the actual cost implied by the pose changes between state transition tuples at adjacent time steps as the supervision signal, so that the evaluation value approximates the actual cost. After training convergence, the network parameters from the input layer to the intermediate feature layer in the evaluation network are transferred to the dynamic weight adjustment module of the online learning interaction cost function, replacing the original model parameters of the behavior interaction predictor.

[0037] In practical implementation, when the multimodal inspection robot transmits newly collected laser scan data from the corridor area, the data processing end calls the iterative nearest-point algorithm to register the new point cloud with the existing 3D point cloud map in the unified spatiotemporal perception atlas. If the registration error exceeds a set threshold, pose drift is determined, and the pose correction is calculated using the least squares method. The point cloud coordinates of the corresponding area in the map are then rigidly transformed and updated. For data-free blank areas in the laser point cloud, the newly scanned non-overlapping point cloud is directly incorporated into the map to fill the gaps in the original map. In the substation scenario, the newly appearing outlines of maintenance personnel in the visible light video stream are extracted by the target detection algorithm to generate motion vectors for dynamic targets, thereby updating the personnel movement path in the dynamic target trajectory layer. In the thermal imaging image, areas where the temperature of the transformer heat sink rises from 60 degrees Celsius to 75 degrees Celsius are marked as new temperature anomaly areas, and the temperature anomaly layer updates the patch attribute values ​​of this area.

[0038] Optionally, the RF receiver detects a newly added 433 MHz wireless sensor network signal, with signal strength increasing at the end of the corridor. The data processing unit recalculates the power distribution of the RF signal source based on this, overlaying the new signal strength grid data onto the existing RF signal strength distribution field. The state transition tuple is stored using a fixed-length data structure, and the experience replay pool employs a first-in, first-out queue management mechanism, automatically discarding the oldest historical data when the pool is full. See Table 2 for the data structure design of the state transition tuple: Table 2: Definition of State Transition Tuple Data Structure In some embodiments, the hidden layers of the feedforward neural network are designed as three fully connected layers, with ReLU nonlinear units as the activation function and a linear activation function for the output layer, directly regressing the overall cost evaluation value. The computation of the supervision signal relies on continuous pose data. If discontinuous tuple sequences exist in the experience replay pool, the training of that batch of samples is skipped to avoid noise interference. The parameter transfer operation only copies the weights and biases of the feature extraction layer of the evaluation network, retaining the original output layer logic in the interactive cost function of online learning, ensuring that the dynamic weight adjustment module can reuse the feature representation capabilities of offline training.

[0039] In one embodiment of the present invention, the heartbeat signal and network connection quality of each multimodal inspection robot are continuously monitored to construct and maintain the cluster's communication topology in real time. When a new multimodal inspection robot is detected joining the cluster, or when a robot is detected to be offline due to a fault, the communication topology is updated and broadcast to all online robots. Robots receiving the updated communication topology automatically adjust the optimization range of their distributed model predictive control algorithm to ensure that the set of new neighboring robots considered in their optimization decisions is consistent with the communication topology. Newly joined robots obtain a snapshot of the unified spatiotemporal perception map at the current moment from neighboring robots and initialize their local optimization problem based on this snapshot. Newly joined robots send a map snapshot request message to neighboring robots. Upon receiving the map snapshot request message, neighboring robots... The system extracts sub-map data covering the local area where the new robot is located from the current version of the unified spatiotemporal awareness map from local storage, and encapsulates the sub-map data and its global timestamp into a snapshot data packet and sends it to the newly joined robot. The newly joined robot receives and unpacks the snapshot data packet, loads the sub-map data into its own memory as a valid representation of the current environment, and immediately performs a data acquisition of the surrounding environment based on its own sensors. It then quickly matches and locates the acquired data with the sub-map data to determine its initial pose in the unified spatiotemporal awareness map. Based on the initial pose, it sets a reachable initial target point within the boundary of the sub-map data, and uses the initial target point as the destination to start the improved distributed model predictive control algorithm to generate the first local control command sequence for the newly joined robot. A task priority queue is set up to store various inspection tasks issued by external systems or discovered autonomously by robots. Each inspection task includes its geographical scope, type, and deadline. The urgency of each task in the queue is periodically assessed, along with the current workload, remaining battery power, and estimated arrival time of each robot in the cluster. Using task urgency, robot workload, remaining battery power, and estimated arrival time as input, a global task allocation optimization calculation is run. The goal of this calculation is to maximize the total value of tasks expected to be completed within a specified time. The global task allocation optimization calculation assigns a multimodal inspection robot best suited to perform each unassigned inspection task and calculates an optimal path for the assigned robot to reach the task area. The assignment relationship, task details, and optimal path are sent to the assigned robot. After completing the current local control command sequence, or immediately interrupting interruptible non-urgent tasks upon receiving the new task, the assigned robot inserts the newly received inspection task into its local optimization objective and begins moving along the optimal path.

[0040] In practice, the heartbeat signal and network connection quality of each multimodal inspection robot are continuously monitored, and the cluster's communication topology is constructed and maintained in real time. When a new multimodal inspection robot is detected joining the cluster, or a robot is detected going offline due to a fault, the communication topology is updated and broadcast to all online robots. Robots receiving the updated communication topology automatically adjust the optimization range of their distributed model predictive control algorithm to ensure that the set of new neighboring robots considered in their optimization decisions is consistent with the communication topology. The newly joined robot obtains a snapshot of the unified spatiotemporal awareness map at the current moment from neighboring robots and initializes its local optimization problem based on this snapshot. The newly joined robot sends a map snapshot request message to neighboring robots. Upon receiving the map snapshot request message, the neighboring robots extract the subgraph data covering the local area where the new robot is located from the current version of the unified spatiotemporal awareness map from their local storage, encapsulate the subgraph data and its global timestamp into a snapshot data packet, and send it to the newly joined robot. The newly joined robot receives and unpacks the snapshot data packet, loads the subgraph data into its own memory, and uses it as a valid representation of the current environment. The newly added robot immediately collects data from its surroundings using its onboard sensors. This data is then quickly matched and localized with submap data to determine its initial pose within the unified spatiotemporal perception map. Based on this initial pose, a reachable initial target point is set within the boundaries of the submap data. Using this initial target point as the destination, an improved distributed model predictive control algorithm is initiated to generate the robot's first local control command sequence. A task priority queue is set up to store various inspection tasks issued by external systems or discovered autonomously by the robots. Each inspection task includes its geographical scope, type, and deadline. The urgency of each task in the task queue is periodically assessed, along with the current workload, remaining battery power, and estimated arrival time of each robot in the cluster. Using task urgency, robot workload, remaining battery power, and estimated arrival time as input, a global task allocation optimization calculation is run. The goal of this calculation is to maximize the total value of tasks expected to be completed within a specified time. The global task allocation optimization calculation assigns the most suitable multimodal inspection robot to each unassigned inspection task and calculates an optimal path for the assigned robot to reach the task area. The assignment relationship, task details, and optimal path are sent to the assigned robot. After completing the current local control command sequence, or immediately interrupting the interruptible non-urgent task upon receiving the task, the assigned robot inserts the newly received inspection task into its local optimization target and begins to move according to the optimal path.

[0041] In some embodiments, the objective function for global task allocation optimization calculation adopts an integer linear programming model, and its expression is: in: Represents the total number of inspection tasks to be assigned; This represents the total number of multimodal inspection robots available in the cluster; The representative will carry out the task Assigned to robots The scalar value of the utility score that can be obtained is determined by the urgency of the task, the matching degree of robot capabilities, and the arrival time. It is a binary decision variable, when the task Assigned to robots The value is 1 if it is true, and 0 otherwise.

[0042] Optionally, the cluster communication topology uses an adjacency matrix data structure, where rows and columns correspond to registered robot identifiers, and matrix elements record the communication latency and link stability between corresponding robots. When a robot loses three consecutive heartbeat packets, it is determined to be offline, the system removes it from the adjacency matrix, and broadcasts a topology change notification to the entire network. Newly added robots actively send registration messages after connecting to the local area network. The central scheduler verifies their identity and adds them to the matrix, and the updated adjacency matrix is ​​then distributed to all nodes. Robots receiving the update adjust their neighbor lists and, in subsequent interactions with the distributed model predictive control algorithm, only exchange predicted trajectory information with neighbors defined in the topology graph.

[0043] Understandably, the task priority queue is maintained using a max-heap data structure. Task urgency is calculated backward from the deadline, with tasks closer to the deadline having higher priority. The robot's workload is calculated by counting the number of tasks it currently handles and the estimated remaining execution time. Remaining battery power is represented by the state of charge percentage reported by the battery management system. The estimated time to reach the task area is estimated based on a routing algorithm using the current pose and the task's geographical coordinates. Global task allocation optimization calculations are run periodically. Before each calculation, a snapshot of the current task and robot state is frozen. Once the optimal assignment scheme is obtained, it is immediately issued for execution. The assigned robot adds new task points to the local optimization objective, discretizing the optimal path into a series of intermediate waypoints, which are then used as tracking targets for the local control command sequence.

[0044] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for controlling a multimodal inspection robot swarm, characterized in that, include: Heterogeneous environmental data is collected by various sensing terminals deployed in the inspection area. The heterogeneous environmental data includes laser point cloud sequences, visible light video streams, thermal images, and multi-source radio frequency signals. The collected heterogeneous environmental data is spatiotemporally aligned and feature-level fused to construct a unified spatiotemporal perception map, which includes a semantic three-dimensional model of the environmental structure, dynamic target trajectory, and radio frequency signal intensity distribution field. Based on the unified spatiotemporal perception map, an improved distributed model predictive control algorithm is used to generate local control command sequences for each multimodal inspection robot in the cluster. The improved distributed model predictive control algorithm is based on online learning interactive cost function for rolling optimization. The local control command sequence is sent to the corresponding multimodal inspection robot to drive the multimodal inspection robot to perform movement, observation and data acquisition actions; The system receives incremental perception data returned by each multimodal inspection robot after it performs an action, and uses the incremental perception data to asynchronously update the unified spatiotemporal perception map and the interaction cost function of the online learning.

2. The multimodal inspection robot swarm control method according to claim 1, characterized in that, The collected heterogeneous environmental data are spatiotemporally aligned and feature-level fused to construct a unified spatiotemporal perception map, including: For each frame of the laser point cloud sequence, its timestamp and sensor spatial pose are extracted, and each frame of laser point cloud is registered to the global coordinate system to form a dense three-dimensional point cloud map. Keyframe extraction and feature point matching are performed on the visible light video stream. A texture map is generated through visual synchronous positioning and mapping technology. The texture map is then projected and fused with the dense 3D point cloud map to generate a preliminary 3D model with texture. Using the thermal image, temperature anomaly areas on the surface of the preliminary 3D model are identified, and the temperature values ​​are associated with the preliminary 3D model as an independent layer. Time difference of arrival (TDOA) and signal fingerprint analysis are performed on the multi-source radio frequency signals to calculate the location and signal strength of the radio frequency signal sources and generate the radio frequency signal strength distribution field. The moving target contour indicated by the dynamic target trajectory is superimposed on the textured preliminary three-dimensional model in real time, and the radio frequency signal intensity distribution field is associated with the global coordinate system in the form of a field map, together forming the unified spatiotemporal perception map.

3. The multimodal inspection robot swarm control method according to claim 1, characterized in that, The improved distributed model predictive control algorithm performs rolling optimization based on an online-learned interactive cost function, including: At the beginning of each optimization cycle, the real-time status information and local targets of each multimodal inspection robot are obtained, and the dynamic environment sub-map around the multimodal inspection robot is extracted from the unified spatiotemporal perception map. Define an initial interaction cost function, which is composed of a weighted sum of a path tracking error term, a collision risk term, and an energy consumption term; During the rolling optimization process, a behavior interaction predictor based on a neural network model is introduced. The behavior interaction predictor receives predicted trajectory segments from other robots and outputs the predicted interaction conflict intensity value. The predicted interaction conflict intensity value is used as a dynamic weighting coefficient to adjust the collision risk term in the initial interaction cost function in real time, forming the online learning interaction cost function available in the current optimization cycle. Using the interactive cost function of the online learning as the objective, distributed rolling optimization is performed on the trajectory sequence and control sequence of each multimodal inspection robot to obtain the local control command sequence of each robot, ensuring that the optimization process converges within a finite number of iterations.

4. The multimodal inspection robot swarm control method according to claim 3, characterized in that, The behavior interaction predictor receives predicted trajectory segments from other robots and outputs predicted interaction conflict intensity values, including: Before each rolling optimization solution, the predicted trajectory fragments broadcast by neighboring multimodal inspection robots in the previous optimization cycle are collected from the communication link. The predicted trajectory fragments contain the planned position and attitude of the robots in the future finite time steps. Align fragments of the local robot's planned trajectory with fragments of all neighboring robot trajectories in the spatiotemporal domain to form a multidimensional spatiotemporal relation tensor; The multidimensional spatiotemporal relation tensor is input into a pre-trained convolutional recurrent neural network model, which analyzes the spatiotemporal correlation patterns between multiple robot trajectories. The output of the convolutional recurrent neural network model is a scalar conflict intensity coefficient between zero and one. The scalar conflict intensity coefficient represents the probability assessment value of spatiotemporal overlap or intrusion of the safe distance between robots within the time window covered by the predicted trajectory segment. The scalar conflict intensity coefficient is output as the predicted interaction conflict intensity value for the current optimization step, and used to adjust the cost function.

5. The multimodal inspection robot swarm control method according to claim 1, characterized in that, The local control command sequence is sent to the corresponding multimodal inspection robot to drive the multimodal inspection robot to perform movement, observation, and data acquisition actions, including: The local control command sequence containing velocity and angular velocity sequences is converted into the underlying drive signals for each joint motor. During the movement of the multimodal inspection robot, the data from its odometer and inertial measurement unit are read in real time and compared with the expected motion state of the control command to generate a motion compensation signal. According to the pre-set observation task nodes in the local control command sequence, the gimbal camera, lidar, thermal imager and radio frequency receiver on the multimodal inspection robot are triggered to perform a collaborative scanning action; The collaborative scanning action includes: the gimbal camera continuously zooms to capture images of the target area pointed to by the observation task node; the lidar performs synchronous rotational scanning; the thermal imager collects infrared radiation information of the corresponding area; and the radio frequency receiver performs full-band signal reception of a specific frequency band. All execution status logs and raw sensor data generated during the driving, motion compensation, scanning, and detection processes are time-stamped, encapsulated together with task node identifiers, and prepared for back transmission as the incremental sensing data.

6. The multimodal inspection robot swarm control method according to claim 5, characterized in that, The step of receiving incremental sensing data returned after each multimodal inspection robot performs an action, and asynchronously updating the unified spatiotemporal perception map and the online learning interaction cost function using the incremental sensing data, includes: Receive data packets from various multimodal inspection robots and extract the raw sensor data with time stamps and observation task node information from them; The laser scanning data in the original sensing data is registered and fused with the three-dimensional point cloud map in the unified spatiotemporal perception map in real time to correct the holes or pose drift errors in the map. From the visible light video stream and thermal imaging image in the original sensing data, new dynamic targets or static targets whose state has changed are extracted, and the dynamic target trajectory layer and temperature anomaly layer in the unified spatiotemporal perception map are updated. The radio frequency signal intensity distribution field is updated using the data detected by the radio frequency receiver; The observation task node information, the actual executed control command sequence, and the final pose of the multimodal inspection robot are stored as a state transition tuple in the experience replay pool for periodically retraining the neural network model parameters on which the online learning interaction cost function depends.

7. The multimodal inspection robot swarm control method according to claim 6, characterized in that, The observation task node information, the actual executed control command sequence, and the final pose of the multimodal inspection robot are stored as a state transition tuple in the experience replay pool for periodically retraining the neural network model parameters on which the online learning interaction cost function depends, including: Multiple historical state transition tuples are extracted in batches from the experience replay pool using a uniform sampling method; For each state transition tuple, extract the environmental sub-image segment corresponding to the observation task node, the starting segment of the actual executed control command sequence, and the final multimodal inspection robot pose. An evaluation network is constructed using a feedforward neural network as the main body. The input of the evaluation network is the environmental sub-image segment and the starting segment of the control command sequence, and the output is the overall cost evaluation value of the decision sequence. Using supervised learning, the evaluation network is trained with the actual cost implied by the pose change between state transition tuples in adjacent time steps as the supervision signal, so that the evaluation value approximates the actual cost. After training converges, the network parameters from the input layer to the intermediate feature layer in the evaluation network are transferred to the dynamic weight adjustment module of the online learning interaction cost function, replacing the original model parameters of the behavior interaction predictor.

8. The multimodal inspection robot swarm control method according to claim 1, characterized in that, The method also includes a step for handling dynamic changes in cluster topology: Continuously monitor the heartbeat signal and network connection quality of each multimodal inspection robot, and build and maintain the cluster's communication topology in real time; When a new multimodal inspection robot is detected joining the cluster, or when a robot is detected to be offline due to a malfunction, the communication topology map is updated and broadcast to all online robots. Upon receiving the updated communication topology map, the robot automatically adjusts the optimization range of its distributed model predictive control algorithm to ensure that the set of new neighboring robots considered during optimization decisions is consistent with the communication topology map. The newly added robot obtains a snapshot of the unified spatiotemporal perception map at the current moment from the neighboring robots, and initializes its own local optimization problem based on this snapshot.

9. A multimodal inspection robot swarm control method according to claim 8, characterized in that, The newly added robot obtains a snapshot of the unified spatiotemporal perception map at the current moment from neighboring robots, and initializes its own local optimization problem based on this snapshot, including: Newly joined robots send map snapshot request messages to neighboring robots; After receiving the map snapshot request message, the neighboring robot extracts the sub-map data covering the local area where the new robot is located from the current version of the unified spatiotemporal perception map from its local storage, and encapsulates the sub-map data and its global timestamp into a snapshot data packet and sends it to the newly joined robot. The newly added robot receives and unpacks the snapshot data packet, loads the subgraph data into its own memory, and uses it as a valid representation of the current environment; The newly added robot immediately collects data from its surroundings using its onboard sensors, and then quickly matches and locates the collected data with the sub-map data to determine its initial pose in the unified spatiotemporal perception map. Based on the initial pose, within the boundary of the subgraph data, a reachable initial target point is set, and with the initial target point as the destination, the improved distributed model predictive control algorithm is started to generate the first local control command sequence of the newly added robot.

10. A multimodal inspection robot swarm control method according to claim 1, characterized in that, The method also includes a global task reallocation step: Set up a task priority queue to store various inspection tasks issued by external systems or discovered autonomously by robots. The inspection tasks include the geographical scope of the task, the task type, and the task deadline. Regularly assess the urgency of each task in the task queue, and evaluate the current workload, remaining battery power, and estimated time to reach the task area for each robot in the cluster. Taking task urgency, robot workload, robot remaining battery power, and robot arrival time as inputs, a global task allocation optimization calculation is run. The goal of the calculation is to maximize the total value of the tasks expected to be completed within the specified time. The global task allocation optimization calculation assigns a multimodal inspection robot that is most suitable for performing the inspection task to each unassigned inspection task, and calculates an optimal path for the assigned robot to reach the task area. The assignment relationship, task details, and optimal path are sent to the assigned robot. After completing the current local control command sequence, or immediately interrupting the interruptible non-urgent task upon receiving the task, the assigned robot inserts the newly received inspection task into its local optimization target and begins to move according to the optimal path.

Images