Multi-type robot collaborative operation method based on dynamic environment perception modeling

Abstract

The present invention discloses a multi-type robot collaborative operation method based on dynamic environment perception modeling. Through multimodal data acquisition and fusion technology, combined with three-dimensional environment modeling and multi-robot collaborative communication mechanism, optimization and dynamic adjustment of task allocation are realized, and a complete solution is formed; multimodal data fusion improves the comprehensiveness and real-time nature of environmental perception, dynamic environment modeling ensures the accuracy and updating efficiency of the model, and multi-robot communication and task collaboration optimize resource allocation and execution effect; the present invention improves the problems of information isolation, single task allocation and insufficient flexibility of collaborative operation in traditional robot systems, and the proposed technical solution realizes efficient collaboration and resource optimization among multiple robots in complex environments, significantly improves the system's perception ability, task response speed and overall operation efficiency, and is widely applicable to complex scenarios such as inspection, rescue and industrial production.

Description

Technical Field

[0001] The present invention belongs to the field of robot control technology, and specifically relates to a multi-type robot collaborative operation method based on dynamic environment perception modeling. Background Art

[0002] Multi-robot systems are widely used in complex scenarios such as inspection, rescue, and industrial production. Their collaborative working capabilities are of great significance to improving task execution efficiency and environmental adaptability. However, current multi-robot systems have significant deficiencies in environmental perception, modeling accuracy, and task collaboration efficiency. In existing technologies, environmental perception often relies on single-modal sensor data (such as lidar or cameras), which makes it difficult to fully obtain multi-dimensional information in complex dynamic environments. The differences in performance and data formats between sensors make the fusion of multimodal data a technical difficulty, and the inconsistency of multimodal information further limits the accuracy and comprehensiveness of environmental perception.

[0003] Traditional environmental modeling methods are mostly based on static data and cannot adapt to the dynamic changes of obstacles, target objects or task conditions in the environment. This static modeling method easily leads to lag or lack of accuracy in the environmental model, thus affecting the effectiveness of task planning. In complex application scenarios, environmental modeling methods that lack dynamic update mechanisms are difficult to meet real-time requirements.

[0004] Multi-robot systems also have deficiencies in task allocation and resource scheduling. Existing task allocation methods usually adopt preset static strategies, which lack the flexibility to adjust to environmental changes or equipment status. This strategy easily leads to problems such as resource waste, task conflicts, and low operating efficiency. At the same time, the communication mechanisms and protocols between different types of robots lack unified standards, and the efficiency of data sharing and collaborative operations is limited, making it difficult to give full play to the collaborative capabilities of multi-robot systems. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a multi-type robot collaborative operation method based on dynamic environment perception modeling, by designing a comprehensive multimodal data perception mechanism, establishing a real-time dynamic environmental model, and introducing efficient communication and collaborative operation strategies, to provide support for the efficient operation of multi-robot systems in complex environments.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-type robot collaborative operation method based on dynamic environment perception modeling includes the following steps:

[0008] Step 1: Multimodal perception data collection

[0009] 11) Data collection: Use sensors to collect multimodal data;

[0010] 12) Data preprocessing: After preprocessing the collected multimodal data;

[0011] 13) Feature extraction: extract features from preprocessed multimodal data;

[0012] Step 2: Multimodal data fusion

[0013] 21) Data-level fusion: A cascade operation is used to combine the features of all modalities into a preliminary feature set in a specific order;

[0014] 22) Feature-level fusion: Nonlinear mapping is used on the preliminary feature set to capture the interaction between the modalities; the cross-modal fusion mechanism based on Transformer realizes the deep fusion of multi-modal features and dynamically adjusts the importance of each modal feature to obtain cross-modal fusion features;

[0015] Step 3: Environment Modeling

[0016] 31) Generate 3D point cloud model: Extract key information in the environment from cross-modal fusion features and generate a 3D point cloud model;

[0017] 32) Dynamically update point cloud model: Based on time series data, dynamically update the 3D point cloud model so that it can reflect real-time changes in the environment;

[0018] Step 4: Multi-robot collaborative communication

[0019] The collaborative communication of multiple types of robots is realized through a unified communication model to support information sharing, task collaboration and data standardization between devices, thus improving the overall efficiency and flexibility of the system.

[0020] Step 5: Task allocation and collaborative work

[0021] 51) Construct a task allocation model: Assign patrol tasks to multiple robots of different types with minimizing the total cost as the objective function to construct a task allocation model;

[0022] 52) Collaborative task adjustment: Dynamically adjust task allocation based on changes in the environment and tasks.

[0023] Furthermore, in step 21), the sensor set is assumed to be:

[0024]

[0025] in: Indicates Class sensor, Indicates the type of sensor;

[0026] No. Sensor Collected data stream It is expressed as:

[0027]

[0028] in: Represents three-dimensional point cloud coordinate data; Represents image plane coordinate data; Represents the inertial measurement unit data, is the angular velocity; is the acceleration; Represents temperature and humidity sensor data, is temperature; for humidity; For time.

[0029] Furthermore, in step 12), the collected multimodal data is subjected to denoising and standardization processing:

[0030] For 3D point cloud coordinate data , use Gaussian noise removal model for denoising:

[0031]

[0032] in: Represents the three-dimensional point cloud coordinate data after denoising; represents the noise component;

[0033] For image data , denoising is performed by subtracting noise:

[0034]

[0035] in: Represents the image data after denoising; Indicates noise;

[0036] To ensure the time alignment of multimodal data, synchronization operations are performed:

[0037]

[0038] in: is the synchronized multimodal data; Represents the alignment mechanism.

[0039] Furthermore, the method for extracting features from the preprocessed multimodal data is as follows:

[0040] Geometric features of point cloud data Include normal vector and curvature ; Among them: normal vector It is used to represent the directional characteristics of the point cloud surface, and , represents the gradient, Indicates the point cloud position; curvature Describes the local shape characteristics of the point cloud surface and can reflect the degree of surface change;

[0041] The visual features of image data are extracted through deep learning models, and the feature vector It is expressed as:

[0042]

[0043] in: Represents a deep learning model;

[0044] Kinematic characteristics of inertial measurement unit data Through the angular velocity and acceleration The integral calculation of:

[0045]

[0046] in: Represents acceleration and angular velocity as a function of time, and the integral result reflects the cumulative effect of motion;

[0047] Physical quantity characteristics of environmental data Including temperature and humidity :

[0048]

[0049] in: represents the temperature-time function; Represents humidity as a function of time.

[0050] Furthermore, in step 21), the preliminary feature set It is expressed as:

[0051]

[0052] in: Represent point cloud features; Represents image features; Represents IMU characteristics; Represents environmental data characteristics; Represents a feature concatenation operation.

[0053] Furthermore, in step 22), the method of using nonlinear mapping on the preliminary feature set to capture the interaction relationship between the modes is:

[0054]

[0055] in: Represents the fused multimodal features; represents the weight matrix; is the bias term; is the ReLU activation function;

[0056] The method of realizing deep fusion of multimodal features based on the Transformer cross-modal fusion mechanism is:

[0057]

[0058]

[0059] in: Indicates the first Attention score of the individual; Represents the embedded representation of the input sequence; , and are the weight matrices of the query, key, and value corresponding to the header, respectively; is the dimension of the key vector;

[0060] The outputs of multiple heads are concatenated and linearly mapped to obtain cross-modal fusion features:

[0061]

[0062] in: It is the cross-modal fusion feature; is the number of heads; is the output weight matrix.

[0063] Furthermore, in step 31), the generated three-dimensional point cloud model is:

[0064]

[0065] in: Represents the spatial coordinates of a point; The semantic annotation of the representation point is determined by the visual data related information in the cross-modal fusion feature, which is used to describe the category information of a specific point in the point cloud; is the total number of points contained in the point cloud model;

[0066] In step 32), the dynamic updating method of the three-dimensional point cloud model is:

[0067]

[0068] in: express 3D point cloud model at the moment; express 3D point cloud model at the moment; To fuse features across modalities The generated incremental model represents the newly added or updated points.

[0069] Furthermore, in step 4, the communication data between multiple devices is represented as:

[0070]

[0071] in: Indicates the device and equipment Communication data between For equipment Status; For equipment Status; A shared 3D point cloud model; Mapping functions for communication protocols;

[0072] The data normalization process is expressed as:

[0073]

[0074] in: Represents the standardized data; is the original data; for the target agreement; is the protocol conversion function;

[0075] When a third-party system does not support the standard protocol, the data format conversion is completed through adaptive protocol conversion, which is expressed as:

[0076]

[0077] in: is the standardized data after transformation; is the original input data; for the target agreement; It is an adaptive conversion function used to adapt to the intercommunication requirements of multiple protocols;

[0078] All devices are connected to the ring network switch and connected to the host computer through the ground monitoring station. The network model of device communication is expressed as:

[0079]

[0080] in: Represents the network communication topology; A communication node for the device; It is the service interface of the ring network switch; Defines the network configuration method for the device.

[0081] Further, in step 51), the task allocation model is:

[0082]

[0083]

[0084] in: Represents a robot Execute the task Costs; Represented by 3D point cloud model The robot you get With the task The distance between Indicates the device and equipment Communication data between α represents the proportionality coefficient, which is used to adjust the impact of communication data on the task allocation cost; Indicates the number of patrol tasks; is the number of robots.

[0085] Furthermore, in step 52), the method for dynamically adjusting task allocation is:

[0086]

[0087] in: Indicates the new task allocation plan; Indicates the previous task allocation plan; Represents task adjustments based on real-time feedback updates.

[0088] The beneficial effects of the present invention are:

[0089] The present invention is based on a multi-type robot collaborative operation method based on dynamic environment perception modeling. Through multimodal data acquisition and fusion technology, combined with three-dimensional environment modeling and multi-robot collaborative communication mechanism, it realizes the optimization and dynamic adjustment of task allocation and forms a complete solution; multimodal data fusion improves the comprehensiveness and real-time nature of environmental perception, dynamic environment modeling ensures the accuracy and updating efficiency of the model, and multi-robot communication and task collaboration optimize resource allocation and execution effect; the present invention improves the problems of information isolation, single task allocation and insufficient flexibility of collaborative operation in traditional robot systems. The proposed technical solution realizes efficient collaboration and resource optimization among multiple robots in complex environments, significantly improves the system's perception ability, task response speed and overall operation efficiency, and is widely applicable to complex scenarios such as inspection, rescue and industrial production.

[0090] Specifically, the multi-type robot collaborative operation method based on dynamic environment perception modeling of the present invention has the following technical effects:

[0091] 1) Multimodal data fusion improves environmental perception accuracy: By fusing data from different sensors (such as point clouds, images, IMU data, and environmental data), comprehensive perception of complex dynamic environments is achieved, overcoming the limitations of a single sensor modality; multimodal data fusion improves the accuracy and real-time performance of environmental perception and helps generate more accurate environmental models;

[0092] 2) Dynamically updated environmental modeling: By dynamically updating the 3D point cloud model, it can reflect environmental changes in real time, which is particularly suitable for complex and dynamic task environments. By incrementally updating the newly collected data, the environmental model is synchronized with the actual situation, effectively improving the timeliness and accuracy of environmental modeling.

[0093] 3) Efficient multi-robot collaborative operation: By designing a multi-robot collaborative communication mechanism, real-time data sharing and task collaboration between robots are achieved, effectively avoiding resource waste and task conflicts; unified communication protocols and flexible collaborative mechanisms improve the collaborative efficiency of multi-robot systems in complex scenarios and enhance the overall effect of task execution;

[0094] 4) Optimized task allocation and dynamic adjustment: The task allocation strategy based on the environment model and multi-robot status can dynamically adjust the task allocation according to the real-time environment and equipment status; by minimizing the total cost function, it can reasonably schedule task resources, optimize task allocation, and improve task response speed and execution efficiency;

[0095] 5) Adaptable protocol support: Adopts an adaptive protocol conversion mechanism to support intercommunication between a variety of different devices and communication protocols, ensuring efficient and stable data transmission;

[0096] In summary, the multi-type robot collaborative operation method based on dynamic environment perception modeling in the present invention provides support for the efficient operation of multi-robot systems in complex environments by designing a comprehensive multimodal data perception mechanism, establishing a real-time dynamic environmental model, and introducing efficient communication and collaborative operation strategies. It can adapt to various complex environments and collaborative operations of multiple types of equipment, has strong compatibility and scalability, and is widely used in various application scenarios such as inspection, rescue, and industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0098] Figure 1 A flowchart of a multi-type robot collaborative operation method based on dynamic environment perception modeling of the present invention;

[0099] Figure 2 Flowchart for multimodal sensing data collection;

[0100] Figure 3 Flowchart of multi-robot collaborative communication. DETAILED DESCRIPTION

[0101] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0102] like Figure 1 As shown, the multi-type robot collaborative operation method based on dynamic environment perception modeling in this embodiment includes the following steps.

[0103] Step 1: Multimodal perception data collection.

[0104] In this embodiment, the process of multimodal sensing data collection is as follows: Figure 2 As shown, the following steps are included.

[0105] 11) Data collection: Use sensors to collect multimodal data.

[0106] Specifically, in this embodiment, the sensor set is assumed to be:

[0107]

[0108] in: Indicates Class sensor, Indicates the type of sensor.

[0109] And sensor Collected data stream It is expressed as:

[0110]

[0111] in: Represents three-dimensional point cloud coordinate data; Represents image plane coordinate data; Represents the inertial measurement unit data, is the angular velocity; is the acceleration; Represents temperature and humidity sensor data, is temperature; for humidity; For time.

[0112] 12) Data preprocessing: After preprocessing the collected multimodal data.

[0113] In this embodiment, in order to ensure data quality during multimodal sensing data collection, the original data needs to be standardized and denoised.

[0114] For 3D point cloud coordinate data , use Gaussian noise removal model for denoising:

[0115]

[0116] in: Represents the three-dimensional point cloud coordinate data after denoising; Represents the noise component.

[0117] For image data , denoising is performed by subtracting noise:

[0118]

[0119] in: Represents the image data after denoising; Indicates noise.

[0120] To ensure the time alignment of multimodal data, synchronization operations are performed:

[0121]

[0122] in: is the synchronized multimodal data; Represents the alignment mechanism.

[0123] 13) Feature extraction: Extract features from preprocessed multimodal data.

[0124] Extract key features from multimodal data to provide a basis for subsequent modeling and analysis. Specifically, the method for extracting features from preprocessed multimodal data is as follows.

[0125] Geometric features of point cloud data Include normal vector and curvature ; Among them: normal vector It is used to represent the directional characteristics of the point cloud surface, and , represents the gradient, Indicates the point cloud position; curvature Describes the local shape characteristics of the point cloud surface and can reflect the degree of surface change.

[0126] The visual features of image data are extracted through deep learning models, and the feature vector It is expressed as:

[0127]

[0128] in: Represents a deep learning model.

[0129] Motion characteristics of inertial measurement unit (IMU) data Through the angular velocity and acceleration The integral calculation of:

[0130]

[0131] in: Represents acceleration and angular velocity as a function of time. The integral reflects the cumulative effect of motion.

[0132] Physical quantity characteristics of environmental data Including temperature and humidity :

[0133]

[0134] in: represents the temperature-time function; Represents humidity as a function of time.

[0135] The features extracted from multimodal data cover multiple dimensions such as space, vision, motion and environment, providing a complete information basis for the fusion of multimodal data.

[0136] Step 2: Multimodal data fusion.

[0137] 21) Data-level fusion: In data-level fusion, a cascade operation is used to combine the features of all modalities into a preliminary feature set in a specific order. Specifically, in this embodiment, the preliminary feature set It is expressed as:

[0138]

[0139] in: Represent point cloud features; Represents image features; Represents IMU characteristics; Represents environmental data characteristics; Represents a feature concatenation operation, which combines data features of multiple modalities into an overall feature in a specific order.

[0140] 22) Feature-level fusion: In feature-level fusion, nonlinear mapping is used on the preliminary feature set to capture the interaction between the modalities. The Transformer-based cross-modal fusion mechanism realizes the deep fusion of multi-modal features and dynamically adjusts the importance of each modal feature to obtain cross-modal fusion features.

[0141] Specifically, the method of using nonlinear mapping on the preliminary feature set to capture the interaction relationship between the modes is:

[0142]

[0143] in: Represents the fused multimodal features; represents the weight matrix; is the bias term; is the ReLU activation function.

[0144] In order to achieve deep fusion of multimodal features and dynamically adjust the importance of each modal feature, a Transformer-based cross-modal fusion mechanism is introduced. The Transformer architecture can effectively capture the complex associations and dependencies between different modal features.

[0145] The Transformer encoder consists of multiple multi-head self-attention layers and feed-forward neural network layers. In the multi-head self-attention layer, by calculating the attention scores of different heads, each feature in the input sequence is weighted and summed, thereby fusing the information of different modal features. For the hth head in the multi-head self-attention layer, the formula for calculating the attention score is:

[0146]

[0147]

[0148] in: Indicates the first Attention score of the individual; Represents the embedded representation of the input sequence; , and They are the weight matrices of the query, key, and value of the corresponding header respectively; is the dimension of the key vector.

[0149] The outputs of multiple heads are concatenated and linearly mapped to obtain cross-modal fusion features:

[0150]

[0151] in: It is the cross-modal fusion feature; is the number of heads; is the output weight matrix; Represents a feature concatenation operation.

[0152] As the embedded representation of the input sequence X, the final cross-modal fusion feature is generated through the multi-head attention mechanism Finally, the output after the cross-modal fusion of Transformer is , which is used for subsequent application tasks. Feature-level fusion completes the integration and optimization of multimodal features and is the final output of data fusion.

[0153] Step 3: Environmental modeling.

[0154] 31) Generate 3D point cloud model: Extract key information in the environment from cross-modal fusion features and generate a 3D point cloud model.

[0155] In this embodiment, the generated three-dimensional point cloud model is:

[0156]

[0157] in: Represents the spatial coordinates of a point; The semantic annotation of the representation point is determined by combining the visual data related information in the cross-modal fusion feature with the deep learning model to describe the category information of a specific point in the point cloud, such as "obstacle", "target object", etc. The total number of points contained in the point cloud model.

[0158] feature The included multimodal fusion information provides multi-dimensional data support for point cloud generation, such as providing positional relationships through visual features, reflecting geometric properties through point cloud features, and associating dynamic change information through IMU features.

[0159] 32) Dynamically update point cloud model: Based on time series data, dynamically update the 3D point cloud model so that it can reflect real-time changes in the environment.

[0160] In this embodiment, the dynamic update method of the three-dimensional point cloud model is:

[0161]

[0162] in: express 3D point cloud model at the moment; express 3D point cloud model at the moment; To fuse features across modalities The generated incremental model represents the newly added or updated points.

[0163] Fusion Features It plays a core role in the dynamic update process, generating new point cloud increments Δ by integrating the latest data from multimodal sensors. Mt , thereby updating the environmental model in real time to ensure its dynamic adaptability.

[0164] Step 4: Multi-robot collaborative communication.

[0165] like Figure 3 As shown, the collaborative communication of multiple types of robots (such as mobile robots, drones, cameras, etc.) is implemented through a unified communication model to support information sharing, task collaboration, and data standardization processing between devices, thereby improving the overall efficiency and flexibility of the system.

[0166] In this embodiment, the communication data between multiple devices is represented as:

[0167]

[0168] in: Indicates the device and equipment Communication data between For equipment Status (such as location, task status, etc.); For equipment Status; A shared 3D point cloud model; Mapping functions to communication protocols.

[0169] Devices use multiple protocols to communicate with each other to meet the requirements of different scenarios for transmission speed and stability. For example, robots and drones use TCP protocol for reliable data transmission, video and voiceprint monitoring systems combine TCP / UDP protocol to balance transmission efficiency and stability, primary and secondary equipment monitoring systems communicate with patrol hosts via UDP protocol, and intelligent analysis hosts and upper-level systems use TCP and FTPS protocols to ensure data transmission security.

[0170] In order to achieve standardized data processing, it is necessary to support multiple protocols and interfaces. Industrial protocols (such as OPC, Modbus, and CAN) are used for low-latency device communication, IoT protocols (such as MQTT) are used for efficient connection between cloud platforms and edge devices, and protocols such as WebAPI, SQL, and RTSP are used for video streaming and database interaction. In this embodiment, the data standardization process is expressed as:

[0171]

[0172] in: Represents the standardized data; is the original data; for the target agreement; Protocol conversion function.

[0173] When a third-party system does not support standard protocols (such as OPC UA), the data format conversion is completed through the adaptive protocol conversion component. The process is expressed as:

[0174]

[0175] in: is the standardized data after transformation; is the original input data; for the target agreement; It is an adaptive conversion function used to adapt to the intercommunication requirements of multiple protocols.

[0176] All devices are connected to the ring network switch through the Ethernet optical port or electrical port, and are connected to the host computer through the ground monitoring station. Specifically, the network model of device communication is expressed as:

[0177]

[0178] in: Represents the network communication topology; A communication node for the device; It is the service interface of the ring network switch; Defines the network configuration method for the device.

[0179] Through the above unified communication model and protocol design, the system realizes data sharing and collaboration among multiple types of devices, which not only improves the stability and real-time performance of transmission, but also provides reliable support for device collaborative operations in complex scenarios. This whole set of mechanisms ensures the efficiency, compatibility and flexibility of the system.

[0180] Step 5: Task allocation and collaborative work.

[0181] In the intelligent patrol system, task allocation and collaborative work are the core parts to ensure the efficient operation of the system. Combining the results of multimodal data fusion, environmental modeling and multi-robot collaborative communication mentioned above, the realization of task allocation and collaborative work requires the use of a shared environmental model. , the fused features And communication data between multiple devices To carry out reasonable task planning and resource scheduling.

[0182] 51) Construct a task allocation model: Assign patrol tasks to multiple robots of different types with the objective function of minimizing the total cost, and construct a task allocation model. Specifically, the goal of task allocation is to assign patrol tasks to multiple robots of different types. T 1, T 2,…, Tm Assign to multiple robots R 1, R 2,…, Rk The task allocation model can be optimized by minimizing the total cost. The task allocation model constructed in this embodiment is:

[0183]

[0184]

[0185] in: Represents a robot Execute the task Costs (such as time, energy, etc.); Represented by 3D point cloud model The robot you get With the task The distance between Indicates the device and equipment Communication data between them to ensure that task allocation is consistent with the shared environment model; α Represents the proportionality coefficient, which is used to adjust the impact of communication data on the task allocation cost Indicates the number of patrol tasks; is the number of robots.

[0186] By minimizing the cost function, task allocation can be optimized according to the matching of each robot and the task, thereby improving inspection efficiency and resource utilization.

[0187] 52) Collaborative task adjustment: Dynamically adjust task allocation based on changes in the environment and tasks.

[0188] In the actual inspection process, due to changes in the environment and tasks, task allocation may need to be dynamically adjusted. and fusion features , the system can update the task allocation in real time. The robots communicate with each other through data Share information about environmental changes, such as changes in obstacle locations and target object locations, to achieve dynamic adjustment of tasks.

[0189] In this embodiment, the method for dynamically adjusting task allocation is:

[0190]

[0191] in: Indicates the new task allocation plan; Indicates the previous task allocation plan; Represents task adjustments based on real-time feedback updates.

[0192] This process enables task allocation to flexibly respond to environmental changes and equipment status changes, improving the system's responsiveness and inspection efficiency.

[0193] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A multi-type robot collaborative operation method based on dynamic environment perception modeling, characterized by: The steps include: Step 1: Multimodal perception data collection 11) Data collection: Use sensors to collect multimodal data; 12) Data preprocessing: After preprocessing the collected multimodal data; 13) Feature extraction: extract features from preprocessed multimodal data; Step 2: Multimodal Data Fusion 21) Data-level fusion: A cascade operation is used to combine the features of all modalities into a preliminary feature set in a specific order; Preliminary feature set It is expressed as: in: Represent point cloud features; Represents image features; Represents IMU characteristics; Represents environmental data characteristics; Represents feature concatenation operation; 22) Feature-level fusion: Nonlinear mapping is used on the preliminary feature set to capture the interaction between the modalities; the cross-modal fusion mechanism based on Transformer realizes the deep fusion of multi-modal features and dynamically adjusts the importance of each modal feature to obtain cross-modal fusion features; The method of using nonlinear mapping on the preliminary feature set to capture the interaction between the modes is: in: Represents the fused multimodal features; represents the weight matrix; is the bias term; is the ReLU activation function; The method of realizing deep fusion of multimodal features based on the Transformer cross-modal fusion mechanism is: in: represents the first Attention score of the individual; Represents the embedded representation of the input sequence; , and are the weight matrices of the query, key, and value corresponding to the header, respectively; is the dimension of the key vector; The outputs of multiple heads are concatenated and linearly mapped to obtain cross-modal fusion features: in: It is a cross-modal fusion feature; is the number of heads; is the output weight matrix; Step 3: Environment Modeling 31) Generate 3D point cloud model: Extract key information in the environment from cross-modal fusion features and generate a 3D point cloud model; 32) Dynamically update point cloud model: Based on time series data, dynamically update the 3D point cloud model so that it can reflect real-time changes in the environment; Step 4: Multi-robot collaborative communication The collaborative communication of multiple types of robots is realized through a unified communication model to support information sharing, task collaboration and data standardization between devices, thus improving the overall efficiency and flexibility of the system. Step 5: Task allocation and collaborative work 51) Construct a task allocation model: Assign patrol tasks to multiple robots of different types with minimizing the total cost as the objective function to construct a task allocation model; 52) Collaborative task adjustment: Dynamically adjust task allocation based on changes in the environment and tasks.

2. The multi-type robot collaborative operation method based on dynamic environment perception modeling according to claim 1 is characterized by: In step 21), the sensor set is: in: Indicates Class sensor, Indicates the type of sensor; No. Sensor Collected data stream It is expressed as: in: Represents three-dimensional point cloud coordinate data; Represents image plane coordinate data; Represents the inertial measurement unit data, is the angular velocity; is the acceleration; Represents temperature and humidity sensor data, is temperature; for humidity; For time.

3. The multi-type robot collaborative operation method based on dynamic environment perception modeling according to claim 2 is characterized by: In the step 12), the collected multimodal data is subjected to denoising and standardization processing: For 3D point cloud coordinate data , use Gaussian noise removal model for denoising: in: Represents the three-dimensional point cloud coordinate data after denoising; represents the noise component; For image data , denoising is performed by subtracting noise: in: Represents the image data after denoising; Indicates noise; To ensure the time alignment of multimodal data, synchronization operations are performed: in: is the synchronized multimodal data; Represents the alignment mechanism.

4. The multi-type robot collaborative operation method based on dynamic environment perception modeling according to claim 3 is characterized by: The method for feature extraction of preprocessed multimodal data is: Geometric features of point cloud data Include normal vector and curvature ; Among them: normal vector It is used to represent the directional characteristics of the point cloud surface, and , represents the gradient, Indicates the point cloud position; curvature Describes the local shape characteristics of the point cloud surface and can reflect the degree of surface change; The visual features of image data are extracted through deep learning models, and the feature vector It is expressed as: in: Represents a deep learning model; Kinematic characteristics of inertial measurement unit data Through the angular velocity and acceleration The integral calculation of: in: Represents acceleration and angular velocity as a function of time, and the integral result reflects the cumulative effect of motion; Physical quantity characteristics of environmental data Including temperature and humidity : in: represents the temperature-time function; Represents humidity as a function of time.

5. The multi-type robot collaborative operation method based on dynamic environment perception modeling according to claim 1 is characterized by: In step 31), the generated three-dimensional point cloud model is: in: Represents the spatial coordinates of a point; The semantic annotation of the representation point is determined by the visual data related information in the cross-modal fusion feature, which is used to describe the category information of a specific point in the point cloud; is the total number of points contained in the point cloud model; In step 32), the dynamic updating method of the three-dimensional point cloud model is: in: express 3D point cloud model at the moment; express 3D point cloud model at the moment; To fuse features across modalities The generated incremental model represents the newly added or updated points.

6. The multi-type robot collaborative operation method based on dynamic environment perception modeling according to claim 1 is characterized by: In step 4, the communication data between multiple devices is represented as: in: Indicates the device and equipment Communication data between For equipment Status; For equipment Status; A shared 3D point cloud model; Mapping function for communication protocol; The data normalization process is expressed as: in: Represents the standardized data; is the original data; for the target agreement; is the protocol conversion function; When a third-party system does not support the standard protocol, the data format conversion is completed through adaptive protocol conversion, which is expressed as: in: is the standardized data after transformation; is the original input data; for the target agreement; It is an adaptive conversion function used to adapt to the intercommunication requirements of multiple protocols; All devices are connected to the ring network switch and connected to the host computer through the ground monitoring station. The network model of device communication is expressed as: in: Represents the network communication topology; A communication node for the device; It is the service interface of the ring network switch; Defines the network configuration method for the device.

7. The multi-type robot collaborative operation method based on dynamic environment perception modeling according to claim 1 is characterized by: In step 51), the task allocation model is: in: Represents a robot Execute the task Costs; Represented by 3D point cloud model The robot you get With the task The distance between Indicates the device and equipment Communication data between α represents the proportionality coefficient, which is used to adjust the impact of communication data on the task allocation cost; Indicates the number of patrol tasks; is the number of robots.

8. The multi-type robot collaborative operation method based on dynamic environment perception modeling according to claim 7 is characterized by: In step 52), the method for dynamically adjusting task allocation is: in: Indicates the new task allocation plan; Indicates the previous task allocation plan; Represents task adjustments based on real-time feedback updates.

Images