Multi-mode large model-based computing power center multi-robot collaborative operation and maintenance system and method

Abstract

The invention relates to the field of computing power center operation and maintenance, in particular to a computing power center multi-robot collaborative operation and maintenance system and method based on a multi-modal large model, and the method comprises the steps: collecting equipment operation data in real time; carrying out data preprocessing on the acquired multi-modal data; the preprocessed single-mode features are input into a multi-mode large model, a unified multi-mode feature vector is generated through an adaptive feature fusion algorithm, the equipment fault type, position and occurrence probability are predicted based on the fused features, and a multi-robot cooperative operation and maintenance strategy is generated; the cooperative control center distributes tasks to the optimal robot combination through a Hungary algorithm according to an operation and maintenance strategy; a reinforcement learning algorithm is adopted to plan a collision-free path for each robot, and real-time dynamic adjustment is carried out to cope with environmental changes; and the operation and maintenance robot executes inspection, fault positioning and basic maintenance tasks, and uploads execution data to the cooperative control center in real time. The complex operation and maintenance requirements of the computing power center are effectively met, and the operation and maintenance efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of operation and maintenance of computing power centers, and specifically to a multi-robot collaborative operation and maintenance system and method for computing power centers based on a multimodal large model. Background Art

[0002] With the rapid development of information technology, computing centers, as core hubs for data processing and storage, are expanding in scale, with increasing numbers of devices and exponentially increasing operational complexity. Traditional computing center operations and maintenance rely primarily on manual inspections and simple monitoring systems, which present numerous limitations.

[0003] On the one hand, manual inspections are not only inefficient but also difficult to achieve real-time, comprehensive monitoring. Computing centers have numerous and widely distributed devices. Manual inspections cannot promptly detect subtle anomalies in equipment operation, easily missing the optimal opportunity for fault warnings. This can lead to escalating failures, impacting the normal operation of the computing center, and even causing serious consequences such as data loss and business interruption. On the other hand, simple monitoring systems typically only collect data from a single modality, such as basic parameters like device temperature and voltage, and cannot fully reflect the operating status of the equipment. Moreover, these monitoring systems lack intelligent analytical capabilities, making it difficult to conduct in-depth mining and correlation analysis of the collected data. This makes it impossible to accurately predict the occurrence of equipment failures, let alone formulate scientific and reasonable operation and maintenance strategies based on equipment status and operation and maintenance needs.

[0004] Furthermore, with the continuous development of artificial intelligence technology, multimodal large models have demonstrated powerful capabilities in data processing and analysis. However, in the field of computing power center operations and maintenance, how to combine multimodal large models with multi-robot collaborative operations and fully utilize multimodal data to achieve intelligent operations and maintenance remains a pressing issue. Existing operations and maintenance systems fail to fully utilize the rich information contained in multimodal data, making it difficult to accurately predict equipment failures and achieve intelligent multi-robot collaboration, thus failing to meet the requirements of efficient and intelligent operations and maintenance in modern computing power centers. Summary of the Invention

[0005] In response to the problem that existing systems cannot meet the needs of efficient and intelligent operation and maintenance of modern computing power centers, the present invention provides a multi-robot collaborative operation and maintenance system and method for computing power centers based on a multimodal large model.

[0006] In a first aspect, the technical solution of the present invention provides a multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model, comprising: Multimodal data acquisition modules are deployed at various monitoring points in the computing center to collect real-time equipment operation data and use adaptive sampling technology to dynamically adjust the sampling frequency based on the equipment's operating status. The data preprocessing module performs data preprocessing on the collected multimodal data and extracts features from different modal data respectively; The multimodal large model receives pre-processed multimodal feature data and implements intelligent operation and maintenance through the following functions: Perform feature fusion based on an adaptive feature fusion algorithm and predict equipment fault type, fault location, and occurrence probability based on the fusion results; An operation and maintenance robot cluster, consisting of at least two robots with autonomous navigation capabilities. Each robot is equipped with high-precision sensors and actuators to receive instructions and perform equipment inspections, fault location, and basic maintenance operations, and upload data to the collaborative control center in real time. Collaborative control center, equipped with: The dynamic task allocation module is used to generate the optimal task allocation plan using the Hungarian algorithm based on the fault prediction results output by the multimodal large model, combined with equipment layout, task urgency, and real-time robot status, and supports dynamic adjustment. The multi-robot path planning module uses a reinforcement learning algorithm to generate a collision-free optimal path for each robot and adjust the path in real time to cope with environmental changes; The human-computer interaction interface provides a visual interface to display device status, robot task progress and fault warning information, and supports operation and maintenance personnel to manually adjust task priority or path planning.

[0007] The system collects multimodal data and fuses features using deep learning technology. This multimodal data fusion approach provides a more comprehensive reflection of equipment operating status and improves the accuracy of fault diagnosis. Operation and maintenance robots are equipped with high-precision sensors and intelligent algorithms, enabling them to pinpoint the location and cause of equipment anomalies. This reduces the time and workload for on-site troubleshooting by maintenance personnel and improves the accuracy and reliability of operations. The multi-robot path planning module utilizes a dynamic obstacle avoidance algorithm based on reinforcement learning to generate an optimal collision-free path for each robot and adjust the path in real time to accommodate environmental changes. This intelligent path planning approach ensures that robots can complete tasks efficiently and safely in complex environments, further enhancing the accuracy of operations and maintenance.

[0008] As a further limitation of the technical solution of the present invention, the data preprocessing module performs the following processing on the collected multimodal data: Sliding window filtering is used for time series data, and fast Fourier transform is performed on audio signals for noise reduction; Use convolutional neural network to extract indicator light color / morphological features from image data; Long short-term memory network is used to extract trend features from time series data; Mel-frequency cepstral coefficients are used to extract the frequency characteristics of abnormal noises from the equipment. Based on the timestamp synchronization algorithm, different modal features are aligned to microsecond accuracy; Add anomaly labels to data points that exceed the normal operation threshold of the device, and record the timestamp and anomaly level of the anomaly; Priority encoding is performed on processed data based on the anomaly level.

[0009] As a further limitation of the technical solution of the present invention, the multimodal data acquisition module adopts adaptive sampling technology to dynamically adjust the sampling frequency according to the operating status of the device, specifically including: When the device is in normal operation, the preset low-frequency threshold sampling is used; When an abnormality occurs in the device, it automatically switches to the preset high-frequency threshold sampling and triggers the data preprocessing module to perform data cleaning and feature extraction. The data processing module inputs the extracted features into the multimodal large model.

[0010] The multimodal data acquisition module uses adaptive sampling technology to dynamically adjust the sampling frequency based on the device's operating status. This not only improves data acquisition efficiency but also enables timely capture of device status changes, enhancing the system's adaptability to dynamic environments.

[0011] As a further limitation of the technical solution of the present invention, the feature fusion process of the multimodal large model includes: For unimodal feature vectors of images, time series, and audio, a dynamic weight allocation algorithm is used to calculate cross-modal correlation; Generate a unified multimodal feature vector and input it into a fully connected neural network for multi-objective fault prediction.

[0012] As a further limitation of the technical solution of the present invention, the task dynamic allocation module is specifically used to: Constructing the task-robot cost matrix, elements of the cost matrix Including movement time, skill matching, power consumption and urgency factors; The Hungarian algorithm is used to solve the allocation plan that minimizes the total cost, and the task is assigned to the robot that is closest and in the best state first; Monitor the robot status in real time. When the task priority changes or a new fault occurs, the cost matrix is ​​locally updated and redistribution is triggered.

[0013] As a further limitation of the technical solution of the present invention, the dynamic obstacle avoidance mechanism of the multi-robot path planning module includes: Pre-generate collision-free initial paths based on the trained reinforcement learning model; Real-time environment perception through high-precision sensors, dynamic path adjustment using speed barrier method or fast expansion random tree algorithm; Robots avoid path conflicts within the cluster by sharing real-time position and speed information.

[0014] As a further limitation of the technical solution of the present invention, the training process of the reinforcement learning model includes: Model the physical environment of the computing center, including equipment layout, obstacle locations, and dynamically changing environmental factors. Use the robot's initial position, target position, and environmental state as the initial input to the reinforcement learning algorithm. Use reinforcement learning algorithms to train robot path planning to generate a reinforcement learning model, including: Define the state space as the robot's current position and environment information; Define the action space as the directions and speeds in which the robot can move; Design a reward function and optimize it based on path length, obstacle avoidance effect, and task completion time to maximize the reward value.

[0015] As a further limitation of the technical solution of the present invention, the system also includes a real-time monitoring module, which continuously collects the robot's motion data and environmental feedback information during the path execution process, and feeds it back to the multi-robot path planning module to iteratively update the reinforcement learning model.

[0016] While performing its tasks, the robot uses high-precision sensors to perceive changes in its surroundings in real time and feeds this information back to the path planning module. This real-time perception and feedback mechanism enables the robot to flexibly adjust its path in dynamic environments, enhancing the system's flexibility. The multimodal large model and reinforcement learning model continuously collect data and iteratively update during operation. This continuous learning mechanism enables the system to continuously optimize its operation and maintenance strategies, adapting to new equipment conditions and environmental changes, further enhancing the system's flexibility and adaptability.

[0017] The system automates the entire process, from fault prediction and task allocation to path planning, reducing reliance on manual labor and lowering labor costs. Through dynamic task allocation and multi-robot collaboration, the system rationally allocates O&M resources, avoiding waste and duplication of work, further reducing O&M costs. Predictive maintenance can proactively identify potential equipment failures, enabling timely maintenance and preventing escalation of failures, thereby extending equipment life and reducing replacement costs.

[0018] The system provides an intuitive human-machine interface that displays device status, robot task progress, and fault warning information. This interface allows operations personnel to gain real-time insights into operations and make manual adjustments, improving the convenience and transparency of operations management. A multimodal large model analyzes and predicts based on extensive historical and real-time data, providing a scientific basis for operations decision-making. This data-driven decision-making approach helps operations personnel better formulate operations strategies and improve operations management. The system optimizes the overall operations process through multi-robot collaboration, ensuring rapid and effective decision-making in the complex and ever-changing computing center environment, promptly identifying and addressing potential issues, effectively reducing operations risks, and improving overall operations management.

[0019] In a second aspect, the technical solution of the present invention further provides a multi-robot collaborative operation and maintenance method for a computing center based on a multimodal large model, comprising the following steps: Through the sensor network deployed at various monitoring points in the computing power center, equipment operation data is collected in real time, and adaptive sampling technology is used to dynamically adjust the sampling frequency according to the equipment operation status; Perform data preprocessing on the collected multimodal data and extract features for different modal data respectively; The pre-processed single-modal features are input into the multi-modal large model, and a unified multi-modal feature vector is generated through an adaptive feature fusion algorithm. Based on the fused features, the equipment fault type, location, and probability of occurrence are predicted, and a multi-robot collaborative operation and maintenance strategy is generated. The collaborative control center uses the Hungarian algorithm to assign tasks to the optimal robot combination based on the operation and maintenance strategy. It also uses a reinforcement learning algorithm to plan collision-free paths for each robot and dynamically adjusts them in real time to respond to environmental changes. The operation and maintenance robots perform inspections, fault location, and basic maintenance tasks, while uploading execution data to the collaborative control center in real time; Collect robot execution results and feedback from operation and maintenance personnel, update the training data of the multimodal large model, regularly perform incremental training on the model, and optimize the accuracy of fault prediction.

[0020] As a further limitation of the technical solution of the present invention, the steps of dynamically adjusting the sampling frequency according to the operating status of the device using the adaptive sampling technology include: Determine the equipment operating status based on the collected values ​​of the equipment operating parameters; When the device is in normal operation, the preset low-frequency threshold sampling is used; When an abnormality occurs in the device, it automatically switches to the preset high-frequency threshold sampling and triggers the data preprocessing module to perform data cleaning and feature extraction.

[0021] As a further limitation of the technical solution of the present invention, the steps of preprocessing the collected multimodal data and extracting features from different modal data include: For data points marked as abnormal, frequency domain analysis is performed through fast Fourier transform to extract the eigenvectors corresponding to the abnormal frequency components; Align time series data with different sampling rates to ensure temporal consistency of cross-modal features; Different input priorities are assigned to preprocessed data based on the anomaly level, and high-risk data triggers model inference first.

[0022] As a further limitation of the technical solution of the present invention, the collaborative control center assigns tasks to the optimal robot combination based on the operation and maintenance strategy using the Hungarian algorithm; and uses the reinforcement learning algorithm to plan a collision-free path for each robot and dynamically adjusts it in real time to cope with environmental changes. The steps include: Construct a task-robot cost matrix that includes factors such as travel time, skill matching, power consumption, and urgency; The Hungarian algorithm is used to solve the allocation plan that minimizes the total cost; When a new fault occurs or the robot state changes, the cost matrix is ​​locally updated and tasks are reallocated; Pre-generate collision-free paths based on reinforcement learning models; Through real-time environment perception, the path is dynamically adjusted using the speed barrier method or the rapidly expanding random tree algorithm; Multiple robots share location information through a distributed communication protocol to avoid path conflicts.

[0023] As a further limitation of the technical solution of the present invention, the steps of collecting robot execution results and operation and maintenance personnel feedback and updating the training data of the multimodal large model include: Convert the operational staff's empirical solutions into structured knowledge graphs and integrate them with multimodal large models; Based on newly collected fault cases and processing results, incremental training is performed on the multimodal large model to update the fault prediction model; Model performance metrics are evaluated regularly, and a full model update is triggered when the metrics fall below a threshold.

[0024] It can be seen from the above technical solutions that the present application has the following advantages: through in-depth analysis of equipment operation data using a multimodal large model, the type, location and probability of equipment failure can be predicted in advance, thereby realizing predictive maintenance. This avoids the inefficiency of traditional passive maintenance, reduces equipment downtime, and improves operation and maintenance efficiency. The task dynamic allocation module adopts the Hungarian algorithm to generate the optimal task allocation plan based on fault prediction results, equipment layout, task urgency and real-time status of the robot. This dynamic allocation method can make rational use of robot resources, avoid resource waste, and further improve operation and maintenance efficiency. The system supports multi-robot collaborative work, coordinates actions through a master-slave allocation protocol, and ensures that multiple robots collaborate efficiently in complex tasks. This collaborative mechanism can effectively cope with the complex operation and maintenance needs of large-scale computing centers and improve overall operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A block diagram of a system provided in an embodiment of the present invention.

[0027] Figure 2 A flowchart of a method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the application objectives, features, and advantages of this application more obvious and easy to understand, the technical solutions protected by this application will be clearly and completely described below using specific embodiments and drawings. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] like Figure 1 As shown, the technical solution of the present invention provides a multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model, including: Multimodal data acquisition modules are deployed at various monitoring points in the computing center to collect real-time equipment operation data and use adaptive sampling technology to dynamically adjust the sampling frequency based on the equipment's operating status. For example, 9-axis sensor nodes (including temperature / humidity sensors and three-axis accelerometers) are deployed on the top of the computing center racks, with each rack installed at an interval of 2 meters; wide-angle cameras (resolution 1080P, frame rate 30fps) are installed on the front of the server cabinet to capture the status of device indicator lights; microphone arrays are deployed at key locations such as air-conditioning vents and power distribution cabinets to collect device operation audio.

[0030] The multimodal data acquisition module uses adaptive sampling technology to dynamically adjust the sampling frequency according to the device operating status, specifically including: When the device is in normal operation, the preset low-frequency threshold sampling is used; When an abnormality occurs in the device, it automatically switches to the preset high-frequency threshold sampling and triggers the data preprocessing module to perform data cleaning and feature extraction. The data processing module inputs the extracted features into the multimodal large model.

[0031] Normal status: The temperature sensor samples once every 5 minutes, the camera captures an image every 10 minutes, and the audio sensor records 10 seconds of background sound every hour; Abnormal trigger: When any sensor detects that a parameter exceeds a threshold (such as CPU temperature > 75°C, vibration acceleration > 5m / s²), high-frequency sampling is triggered: temperature once per second, camera 5 frames per second, and audio real-time streaming.

[0032] The data preprocessing module performs data preprocessing on the collected multimodal data and extracts features from different modal data respectively; The multimodal large model receives pre-processed multimodal feature data and implements intelligent operation and maintenance through the following functions: Perform feature fusion based on an adaptive feature fusion algorithm and predict equipment fault type, fault location, and occurrence probability based on the fusion results; An operation and maintenance robot cluster, consisting of at least two robots with autonomous navigation capabilities. Each robot is equipped with high-precision sensors and actuators to receive instructions and perform equipment inspections, fault location, and basic maintenance operations, and upload data to the collaborative control center in real time. Collaborative control center, equipped with: The dynamic task allocation module is used to generate the optimal task allocation plan using the Hungarian algorithm based on the fault prediction results output by the multimodal large model, combined with equipment layout, task urgency, and real-time robot status, and supports dynamic adjustment. The multi-robot path planning module uses a reinforcement learning algorithm to generate a collision-free optimal path for each robot and adjust the path in real time to cope with environmental changes; The human-computer interaction interface provides a visual interface to display device status, robot task progress and fault warning information, and supports operation and maintenance personnel to manually adjust task priority or path planning.

[0033] In some embodiments, the data preprocessing module performs the following processing on the collected multimodal data: (1) Use sliding window filtering on the time series data and perform fast Fourier transform noise reduction on the audio signal; remove the transient noise in the time series data and smooth the data. The specific processing steps are as follows: Define the size of the sliding window (e.g. the window length is N data points).

[0034] For each data point, take the 2 before and after N −1 data point (if N is an odd number).

[0035] Calculates the mean or median of the data points in the window and replaces the value of the current data point with that value.

[0036] Slide the window in sequence to filter the entire time series data.

[0037] (2) Use long short-term memory network to extract trend features from time series data; the specific steps are as follows: The filtered time series data is divided into multiple time series segments, and each segment is used as the input of the LSTM network.

[0038] Build an LSTM network model, including the input layer, LSTM layer, and output layer.

[0039] The LSTM model is trained using historical data to enable it to learn trend features in time series data.

[0040] Input the time series data to be processed into the trained LSTM model and extract the feature vector.

[0041] (3) Using Mel-frequency cepstral coefficients to extract the frequency characteristics of abnormal sound of the equipment; specifically including: Perform Fast Fourier Transform (FFT) on the collected audio signal to convert it from the time domain to the frequency domain.

[0042] Analyze frequency domain signals to identify and remove high-frequency noise components (usually the parts with less energy in the high-frequency band).

[0043] Perform inverse fast Fourier transform (IFFT) on the denoised frequency domain signal to convert it back to the time domain.

[0044] Perform pre-emphasis processing on the noise-reduced audio signal to enhance the high-frequency signal.

[0045] The pre-emphasized signal is divided into frames, with each frame typically being 20-40ms.

[0046] Perform fast Fourier transform (FFT) on each frame of signal to obtain the spectrum.

[0047] The spectrum is passed through the Mel filter bank to obtain the Mel spectrum.

[0048] Take the logarithm of the Mel spectrum and then perform discrete cosine transform (DCT) to extract the cepstral coefficients.

[0049] The first several cepstral coefficients are selected as audio features.

[0050] (4) Use convolutional neural network to extract the color / morphological features of indicator lights from image data; the specific processing steps are as follows: Preprocess the collected image data, including grayscale conversion, normalization and other operations.

[0051] Build a convolutional neural network (CNN) model, including convolutional layers, pooling layers, and fully connected layers.

[0052] The CNN model is trained using labeled image data (indicator lights of different colors and shapes). The specific training steps are implemented using existing methods.

[0053] The image to be processed is input into the trained CNN model to extract the feature vector.

[0054] (5) Based on the timestamp synchronization algorithm, different modal features are aligned to microsecond accuracy; specifically including: Add a timestamp to each modality data to ensure that the timestamp accuracy reaches the microsecond level.

[0055] Use a timestamp synchronization algorithm (such as interpolation or nearest neighbor method) to align the timestamps of data from different modalities.

[0056] For data points with inconsistent timestamps, adjustments are made through interpolation or the nearest neighbor method to make their timestamps consistent.

[0057] (6) Add an abnormal label to the data points that exceed the normal operation threshold of the equipment, and record the timestamp and abnormality level of the abnormality; The processed multimodal data is monitored in real time to determine whether the data points exceed the preset normal operation threshold.

[0058] If a data point exceeds the threshold, it is marked as an anomaly and the timestamp and anomaly level of the anomaly are recorded.

[0059] (7) Prioritize the processed data based on the anomaly level.

[0060] Categorize anomalous data points based on anomaly levels (e.g., low, medium, high).

[0061] Assign priority codes to abnormal data points of different levels, where a higher priority indicates a more serious anomaly.

[0062] The priority code is stored together with the data point to facilitate subsequent analysis and processing.

[0063] The data preprocessing module outputs the processed multimodal data and its features, anomaly labels and priority codes to the multimodal large model or other processing modules for subsequent fault prediction and operation and maintenance decision-making.

[0064] In some embodiments, the multimodal large model construction process is as follows: Input layer: receives image features (64 dimensions), time series features (statistics + LSTM output, 32 dimensions), and audio features (8 dimensions); Fusion layer: Multi-Head Attention mechanism is used to calculate cross-modal correlation. The formula is:

[0065] In the formula, Q is the time series feature vector, K is the image feature vector, V is the audio feature vector, is the dimension of the image feature vector.

[0066] Prediction layer: The fully connected neural network outputs 3 branches: Fault type (10 categories, such as power failure, fan abnormality); Fault location (rack number + device slot, such as Rack3-Unit5); Failure probability (0-1 continuous value).

[0067] Training data: We collected over 300,000 normal and faulty samples (covering 20 fault types) and used MixUp data augmentation to synthesize scarce fault scenarios. This allowed us to train a large multimodal model.

[0068] The multimodal large model uses a dynamic weight allocation algorithm to calculate the cross-modal correlation of single-modal feature vectors of images, time series, and audio; generates a unified multimodal feature vector and inputs it into a fully connected neural network for multi-target fault prediction.

[0069] In some embodiments, the task dynamic allocation module is specifically configured to: Constructing the task-robot cost matrix, elements of the cost matrix Including movement time, skill matching, power consumption and urgency factors; Task Collection :Indicates the tasks that need to be assigned. Each task Contains information such as task location and urgency.

[0070] Robot Collection :Indicates the available robots, each robot Contains information such as current location, battery level, skill configuration, etc.

[0071] Create one Cost Matrix ,in Represents a robot Processing tasks comprehensive cost.

[0072] Movement time calculation formula:

[0073] Computing Robot From current location to task Euclidean distance , according to the robot's moving speed , calculate the moving time according to the above formula.

[0074] Skill matching calculation formula:

[0075] Define the skill scoring function , evaluation robot Processing tasks The skill scores are normalized to the interval [0,1], where 1 indicates a perfect match and 0 indicates a complete mismatch.

[0076] Power loss calculation formula:

[0077] Estimation Robot Complete the task Power required , according to the current battery level of the robot , calculate the power loss ratio.

[0078] Urgency factor calculation formula

[0079] Defining tasks The urgency of (e.g., high, medium, low). Normalize the urgency to the interval [0,1], where 1 represents the highest urgency.

[0080] Comprehensive cost formula:

[0081] Where, The calculated comprehensive cost is , fill in the cost matrix middle.

[0082] The Hungarian algorithm is used to find the allocation plan that minimizes the total cost, giving priority to assigning tasks to the robot that is closest and in the best state; the details are as follows: For each row of the cost matrix , find the minimum value of the row , subtract from each element of the row , for each column of the cost matrix , find the minimum value of the column , subtract from each element of the column , generating the reduced cost matrix.

[0083] In the reduced cost matrix, mark all zero elements. Minimize the number of covering lines by drawing lines across all zero elements. Covering lines can be horizontal or vertical. Starting from each row, if there is a zero element in that row that is not covered, draw a horizontal line. Starting from each column, if there is a zero element in that column that is not covered, draw a vertical line. If the number of covering lines is equal to the number of rows (or columns) in the matrix, the optimal solution has been found. If the number of covering lines is less than the number of rows (or columns) in the matrix, the matrix needs to be adjusted.

[0084] Find the smallest uncovered element , subtracted from all uncovered elements , add the elements covering the intersection of the lines , repeat the above steps until the number of covered lines is equal to the number of rows (or columns) of the matrix.

[0085] Select zero elements from the reduced cost matrix to ensure that each task is assigned to only one robot and each robot only processes one task. Prioritize the robot with the closest distance and the best status. That is, when selecting zero elements, prioritize the allocation with the lowest overall cost.

[0086] According to the selected zero element, a task allocation plan is generated, and each task Assigned to the corresponding robot .

[0087] The cost matrix is ​​regularly fully updated to reflect the real-time status of the robots and dynamic changes in tasks. When a new fault occurs, only the rows and columns related to the new fault are partially updated to avoid full recalculation. A distributed locking mechanism prevents conflicts in multi-robot task allocation, ensuring correct and consistent task allocation. For tasks requiring multi-robot collaboration, a master-slave allocation protocol coordinates actions to ensure efficient execution of collaborative tasks.

[0088] In an embodiment of the present invention, the dynamic obstacle avoidance mechanism of the multi-robot path planning module includes: A collision-free initial path is pre-generated based on a trained reinforcement learning model; high-precision sensors are used to perceive the environment in real time, and the path is dynamically adjusted using the speed barrier method or the rapidly expanding random tree algorithm; robots avoid path conflicts within the cluster by sharing real-time position and speed information.

[0089] The training process of the reinforcement learning model includes: modeling the physical environment of the computing center, including equipment layout, obstacle locations, and dynamically changing environmental factors, and using the robot's initial position, target position, and environmental state as the initial input of the reinforcement learning algorithm; using the reinforcement learning algorithm to train the robot's path planning to generate a reinforcement learning model, specifically including: defining the state space as the robot's current position and environmental information; defining the action space as the robot's executable movement direction and speed; designing a reward function, and optimizing it based on path length, obstacle avoidance effect, and task completion time to maximize the reward value.

[0090] Use the Gazebo simulator to build a 3D model of the computing center, including racks, aisles, and temporary obstacles; robot coordinates (x, y), orientation θ, and obstacle distance within 5 meters (16-channel lidar data); forward / backward / left / right turns (0.5-meter step size, 15° angle); Reward function: R = −0.1 × path length −10 × collision risk + 5 × mission urgency Using the Proximal Policy Optimization (PPO) algorithm, which converged after 20,000 episodes, the average path length was 15% shorter than the A* algorithm. Robots use UWB positioning (10cm accuracy) to share their positions in real time. When the distance between two robots is less than 1.5 meters, the lower-priority robot triggers the Velocity Obstacle Method (VO) to bypass it. When encountering temporary obstacles (such as toolboxes placed by maintenance personnel), the RRT* algorithm is triggered through visual recognition (YOLOv5 model) to generate a temporary path.

[0091] The robot's execution results (task completion time, whether the fault handling was successful) are recorded by the operator on-site (e.g., cases where the actual fault type does not match the prediction). Incremental training is triggered daily at 2:00 AM, with new data pre-processed and injected into the model. Online learning algorithms (such as FTRL) are used to update the attention mechanism weights without retraining the entire model. If the same fault type is incorrectly predicted three times in a row, full training is automatically triggered, and a model performance report is generated.

[0092] While performing its tasks, the robot uses high-precision sensors to perceive changes in its surroundings in real time and feeds this information back to the path planning module. This real-time perception and feedback mechanism enables the robot to flexibly adjust its path in dynamic environments, enhancing the system's flexibility. The multimodal large model and reinforcement learning model continuously collect data and iteratively update during operation. This continuous learning mechanism enables the system to continuously optimize its operation and maintenance strategies, adapting to new equipment conditions and environmental changes, further enhancing the system's flexibility and adaptability.

[0093] The system provides an intuitive human-machine interface that displays device status, robot task progress, and fault warning information. This interface allows operations personnel to gain real-time insights into operations and make manual adjustments, improving the convenience and transparency of operations management. A multimodal large model analyzes and predicts based on extensive historical and real-time data, providing a scientific basis for operations decision-making. This data-driven decision-making approach helps operations personnel better formulate operations strategies and improve operations management. The system optimizes the overall operations process through multi-robot collaboration, ensuring rapid and effective decision-making in the complex and ever-changing computing center environment, promptly identifying and addressing potential issues, effectively reducing operations risks, and improving overall operations management.

[0094] like Figure 2 As shown, an embodiment of the present invention further provides a multi-robot collaborative operation and maintenance method for a computing center based on a multimodal large model, comprising the following steps: S1. Through the sensor network deployed at each monitoring point in the computing power center, the equipment operation data is collected in real time, and the sampling frequency is dynamically adjusted according to the equipment operation status using adaptive sampling technology; S2. Preprocess the collected multimodal data and extract features for different modal data respectively; S3. Input the pre-processed single-modal features into the multimodal large model, generate a unified multimodal feature vector through an adaptive feature fusion algorithm, predict the equipment fault type, location, and probability of occurrence based on the fused features, and generate a multi-robot collaborative operation and maintenance strategy; S4: The collaborative control center uses the Hungarian algorithm to assign tasks to the optimal robot combination based on the operation and maintenance strategy. It also uses a reinforcement learning algorithm to plan collision-free paths for each robot and dynamically adjusts them in real time to respond to environmental changes. S5, the operation and maintenance robot performs inspections, fault location, and basic maintenance tasks, while uploading execution data to the collaborative control center in real time; S6. Collect robot execution results and feedback from operation and maintenance personnel, update the training data of the multimodal large model, regularly perform incremental training on the model, and optimize the accuracy of fault prediction.

[0095] In an embodiment of the present invention, the steps of dynamically adjusting the sampling frequency according to the device operating status using the adaptive sampling technology include: S11, judging the equipment operating status according to the values ​​of the collected equipment operating parameters; S12. When the device is in normal operation, sampling is performed using a preset low-frequency threshold; S13. When an abnormality occurs in the device, it automatically switches to the preset high-frequency threshold sampling and triggers the data preprocessing module to perform data cleaning and feature extraction.

[0096] In an embodiment of the present invention, the steps of preprocessing the collected multimodal data and extracting features from different modal data include: S21. Perform frequency domain analysis on the data points marked as abnormal by fast Fourier transform, and extract the eigenvector corresponding to the abnormal frequency component; S22. Align time series data with different sampling rates to ensure temporal consistency of cross-modal features; S23. Assign different input priorities to the preprocessed data according to the anomaly level, and high-risk data will trigger model inference first.

[0097] In some embodiments, the collaborative control center assigns tasks to the optimal robot combination using the Hungarian algorithm based on the operation and maintenance strategy; and uses a reinforcement learning algorithm to plan a collision-free path for each robot and dynamically adjusts it in real time to cope with environmental changes. The steps include: S41. Construct a task-robot cost matrix that includes factors such as travel time, skill matching, power consumption, and urgency. S42. Use the Hungarian algorithm to solve the allocation plan that minimizes the total cost; S43, when a new fault occurs or the robot state changes, the cost matrix is ​​locally updated and tasks are reallocated; S44, pre-generating collision-free paths based on the reinforcement learning model; S45. Through real-time environmental perception, the speed barrier method or the rapidly expanding random tree algorithm is used to dynamically adjust the path; multiple robots share position information through a distributed communication protocol to avoid path conflicts.

[0098] In some embodiments, the steps of collecting robot execution results and operator feedback to update the training data of the multimodal large model include: Convert the operational staff's empirical solutions into structured knowledge graphs and integrate them with multimodal large models; Based on newly collected fault cases and processing results, incremental training is performed on the multimodal large model to update the fault prediction model; Model performance metrics are evaluated regularly, and a full model update is triggered when the metrics fall below a threshold.

[0099] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model, characterized by: include: Multimodal data acquisition modules are deployed at various monitoring points in the computing center to collect real-time equipment operation data and use adaptive sampling technology to dynamically adjust the sampling frequency based on the equipment's operating status. The data preprocessing module performs data preprocessing on the collected multimodal data and extracts features from different modal data respectively; The multimodal large model receives pre-processed multimodal feature data and implements intelligent operation and maintenance through the following functions: Perform feature fusion based on an adaptive feature fusion algorithm and predict equipment fault type, fault location, and occurrence probability based on the fusion results; An operation and maintenance robot cluster, consisting of at least two robots with autonomous navigation capabilities. Each robot is equipped with high-precision sensors and actuators to receive instructions and perform equipment inspections, fault location, and basic maintenance operations, and upload data to the collaborative control center in real time. Collaborative control center, equipped with: The dynamic task allocation module is used to generate the optimal task allocation plan using the Hungarian algorithm based on the fault prediction results output by the multimodal large model, combined with equipment layout, task urgency, and real-time robot status, and supports dynamic adjustment. The multi-robot path planning module uses a reinforcement learning algorithm to generate a collision-free optimal path for each robot and adjust the path in real time to cope with environmental changes; The human-computer interaction interface provides a visual interface to display device status, robot task progress and fault warning information, and supports operation and maintenance personnel to manually adjust task priority or path planning.

2. The multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model according to claim 1 is characterized in that: The data preprocessing module performs the following processing on the collected multimodal data: Sliding window filtering is used on time series data, and fast Fourier transform is performed on audio signals for noise reduction; Use convolutional neural network to extract indicator light color / morphological features from image data; Long short-term memory network is used to extract trend features from time series data; Mel-frequency cepstral coefficients are used to extract the frequency characteristics of abnormal noises from the equipment. Based on the timestamp synchronization algorithm, different modal features are aligned to microsecond accuracy; Add anomaly labels to data points that exceed the normal operation threshold of the device, and record the timestamp and anomaly level of the anomaly; Priority encoding is performed on processed data based on the anomaly level.

3. The multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model according to claim 2 is characterized in that: The multimodal data acquisition module uses adaptive sampling technology to dynamically adjust the sampling frequency according to the device operating status, specifically including: When the device is in normal operation, the preset low-frequency threshold sampling is used; When an abnormality occurs in the device, it automatically switches to the preset high-frequency threshold sampling and triggers the data preprocessing module to perform data cleaning and feature extraction. The data processing module inputs the extracted features into the multimodal large model.

4. The multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model according to claim 3 is characterized in that: The feature fusion process of the multimodal large model includes: For unimodal feature vectors of images, time series, and audio, a dynamic weight allocation algorithm is used to calculate cross-modal correlation; Generate a unified multimodal feature vector and input it into a fully connected neural network for multi-objective fault prediction.

5. The multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model according to claim 4 is characterized in that: The task dynamic allocation module is specifically used for: Constructing the task-robot cost matrix, elements of the cost matrix Including movement time, skill matching, power consumption and urgency factors; The Hungarian algorithm is used to solve the allocation plan that minimizes the total cost, and the task is assigned to the robot that is closest and in the best state first; Monitor the robot status in real time. When task priorities change or new faults occur, the cost matrix is ​​locally updated and redistribution is triggered.

6. The multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model according to claim 5 is characterized in that: The dynamic obstacle avoidance mechanism of the multi-robot path planning module includes: Pre-generate collision-free initial paths based on the trained reinforcement learning model; Real-time environment perception through high-precision sensors, dynamic path adjustment using speed barrier method or fast expansion random tree algorithm; Robots avoid path conflicts within the cluster by sharing real-time position and speed information.

7. The multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model according to claim 6 is characterized in that: The training process of the reinforcement learning model includes: Model the physical environment of the computing center, including equipment layout, obstacle locations, and dynamically changing environmental factors. Use the robot's initial position, target position, and environmental state as the initial input to the reinforcement learning algorithm. Use reinforcement learning algorithms to train robot path planning to generate a reinforcement learning model, including: Define the state space as the robot's current position and environment information; Define the action space as the directions and speeds in which the robot can move; Design a reward function and optimize it based on path length, obstacle avoidance effect, and task completion time to maximize the reward value.

8. The multi-robot collaborative operation and maintenance system for a computing center based on a multimodal large model according to claim 7 is characterized in that: The system also includes a real-time monitoring module, which continuously collects the robot's motion data and environmental feedback information during the path execution process, and feeds it back to the multi-robot path planning module to iteratively update the reinforcement learning model.

9. A multi-robot collaborative operation and maintenance method for a computing center based on a multimodal large model, characterized in that: The following steps are involved: Through the sensor network deployed at various monitoring points in the computing power center, equipment operation data is collected in real time, and adaptive sampling technology is used to dynamically adjust the sampling frequency according to the equipment operation status; Perform data preprocessing on the collected multimodal data and extract features for different modal data respectively; The pre-processed single-modal features are input into the multi-modal large model, and a unified multi-modal feature vector is generated through an adaptive feature fusion algorithm. Based on the fused features, the equipment fault type, location, and probability of occurrence are predicted, and a multi-robot collaborative operation and maintenance strategy is generated. The collaborative control center uses the Hungarian algorithm to assign tasks to the optimal robot combination based on the operation and maintenance strategy. It also uses a reinforcement learning algorithm to plan collision-free paths for each robot and dynamically adjusts them in real time to respond to environmental changes. The operation and maintenance robots perform inspections, fault location, and basic maintenance tasks, while uploading execution data to the collaborative control center in real time; Collect robot execution results and feedback from operation and maintenance personnel, update the training data of the multimodal large model, regularly perform incremental training on the model, and optimize the accuracy of fault prediction.

10. The multi-robot collaborative operation and maintenance method for a computing center based on a multimodal large model according to claim 9 is characterized in that: The steps for using adaptive sampling technology to dynamically adjust the sampling frequency based on the device operating status include: Determine the equipment operating status based on the collected values ​​of the equipment operating parameters; When the device is in normal operation, the preset low-frequency threshold sampling is used; When an abnormality occurs in the device, it automatically switches to the preset high-frequency threshold sampling and triggers the data preprocessing module to perform data cleaning and feature extraction.

Images