A building group collaboration method and system of embodied agent and multi-modal large model

Abstract

The application provides a building group coordination method of embodied agent and multi-modal large model, comprising the following steps: S1: multi-modal perception access; S2: physical-behavior dual-domain knowledge enhancement reasoning and candidate strategy generation; S3: building group distributed game deduction and equilibrium optimization; and S4: embodied execution and safety verification closed loop. The application provides a building group coordination method and system of embodied agent and multi-modal large model, which completely breaks through the space coverage limitation of fixed sensor network by taking a machine dog as a mobile perception node, and at the same time, the machine dog cross-checks the fixed sensor through the high-precision sensor of the machine dog, discovers sensor drift or failure in time, guarantees that the data basis of energy-saving decision is always reliable, and the embodied agent is not only a mobile sensor but also a near-field actuator, so that the short loop feedback of "perception-reasoning-execution" in seconds to minutes is realized for local incidental energy waste.

Description

Technical Field

[0001] This invention relates to the field of building cluster collaboration, and more particularly to a building cluster collaboration method and system using an embodied intelligent agent and a multimodal large model. Background Technology

[0002] The building sector is a major source of global energy consumption and carbon emissions. The proportion of energy consumption during the building operation phase in the total energy consumption of the whole society continues to rise. Driven by the national "dual carbon" strategy, building energy conservation is evolving from improving the energy efficiency of single equipment to intelligent operation and management of the entire building life cycle. Technologies such as IoT sensors, building automation systems, building information models, and digital twins are becoming increasingly mature, providing a foundation for the digitalization of building energy conservation.

[0003] In existing technologies, building energy-saving systems rely on building automation systems to collect data from fixed-location sensors and generate control commands for equipment such as air conditioning and lighting through methods such as PID control, model predictive control, or deep reinforcement learning. After the introduction of large models, existing solutions mainly use large models to improve the accuracy of energy consumption prediction or to infer control strategies.

[0004] However, the intelligent analysis of large models still relies on the limited coverage of fixed sensor networks. Fixed sensor networks have a large number of blind spots, and the sensors themselves may be offset, accumulate dust, or fail, resulting in "incomplete data sources" for energy-saving decisions. At the same time, a lot of waste in building energy consumption comes from local, dynamic, and sporadic events, such as the continuous operation of air conditioning in unattended conference rooms, the failure to detect the open windows leading to cold leakage, and the failure to detect abnormal equipment heating in time. Fixed sensors cannot fully cover such events. Furthermore, for multiple buildings in the same park or under the management of the same owner, there is information asymmetry regarding the adjustable energy consumption potential of each building at the current time. There is a lack of a mechanism to enable each building to achieve distributed game optimization under the premise of meeting the total energy consumption constraints, rather than a simple "one-size-fits-all" power rationing.

[0005] Therefore, it is necessary to provide a building cluster collaboration method and system based on embodied intelligent agents and multimodal large models to solve the above-mentioned technical problems. Summary of the Invention

[0006] This invention provides a building group collaboration method and system for embodied intelligent agents and multimodal large models, which solves the problem of the lack of a mechanism for each building to achieve distributed game optimization under the premise of meeting total energy consumption constraints, rather than a simple "one-size-fits-all" power rationing.

[0007] To address the aforementioned technical problems, this invention provides a building cluster collaboration method using embodied intelligent agents and multimodal large models, comprising the following steps:

[0008] S1: Multimodal perception access: Collect numerical time-series data from fixed sensors, mobile inspection data from embodied intelligent agents, and unstructured text data, and input them into a multimodal large model for joint encoding;

[0009] S2: Physical-Behavioral Dual-Domain Knowledge Enhanced Reasoning and Candidate Strategy Generation: Construct a domain knowledge system and inject knowledge into the reasoning context window of the multimodal large model in real time through retrieval enhancement generation technology to generate multiple candidate global energy-saving control strategies. At the same time, the multimodal large model judges energy-saving opportunities that can be dealt with in real time based on the local energy consumption anomalies discovered in the inspection of the embodied intelligent agent and generates embodied micro-adjustment instructions.

[0010] S3: Distributed game simulation and equilibrium selection of building clusters: Establish a multi-agent game model, the benefit function of each building agent, and put the candidate global control strategy framework into the simulation environment in a multi-round iterative manner. The large model acts as the simulation analyst to conduct multi-dimensional analysis of the game process and equilibrium results, and selects the global optimal solution by Pareto optimality criterion.

[0011] S4: Embodied Execution and Safety Verification Closed Loop: The optimal solution is broken down into control commands at the building equipment level and sent to the building automation system for execution. At the same time, embodied micro-adjustment commands are sent to the corresponding embodied intelligent agents for execution. Energy-saving response is achieved through a dual-speed closed loop. The embodied intelligent agents continuously verify the execution effect during the inspection process. All execution results, inspection data, and large model inference records are continuously fed back and updated to various knowledge bases, forming a continuously evolving energy-saving intelligent agent system.

[0012] Preferably, the fixed sensor data in step S1 includes temperature, humidity, CO2 concentration, illuminance, electricity meter data for each floor, and energy consumption data for each item.

[0013] Preferably, the mobile inspection data in step S1 includes thermal imaging images, RGB scene images, and local environmental parameter data.

[0014] Preferably, the domain knowledge system mentioned in step S2 includes a physical domain knowledge base and a behavioral domain knowledge base. The physical domain knowledge base consists of the thermal parameters of the building envelope and the cold energy distribution characteristics of the HVAC system, while the behavioral domain knowledge base consists of the historical patterns of personnel movement and energy consumption behavior patterns.

[0015] Preferably, the physical domain knowledge base includes building envelope thermal parameters, HVAC system cold energy distribution characteristics, lighting circuit illuminance-power relationship, elevator operating energy efficiency curve, heat capacity of each area, equipment safe temperature range, minimum equipment start-up and shutdown interval, and energy storage system charging and discharging efficiency. The behavioral domain knowledge base includes personnel density patterns in each time period and area, temperature preferences in different areas, lighting usage habits, meeting room reservation and usage patterns, and equipment operation permission rules.

[0016] Preferably, the benefit function in step S2 includes comfort maintenance benefits, energy consumption costs, and global constraints.

[0017] Preferably, the dual-speed closed loop in step S4 includes slow-loop global control and fast-loop local fine-tuning to achieve energy-saving response.

[0018] Preferably, in step S1, the embodied intelligent agent inspection involves real-time 3D scanning of the building, real-time construction of a 3D model of the inspection area, and annotation of fault points which are then transmitted to the mobile terminal.

[0019] Preferably, the embodied intelligent agent transmits the 3D model, fault marking points, real-scene images, and thermal imaging data to a handheld mobile terminal in real time via wireless data transmission.

[0020] A building cluster collaborative system of embodied intelligent agents and multimodal large models includes the building cluster collaborative method of the embodied intelligent agents and multimodal large models, comprising: a multimodal perception access module, a physical-behavioral dual-domain knowledge enhancement reasoning module, a building cluster game deduction and equilibrium selection module, and an embodied execution and security verification closed-loop control module.

[0021] Compared with related technologies, the building cluster collaboration method and system provided by this invention, which combines embodied intelligent agents and multimodal large models, has the following beneficial effects:

[0022] This invention provides a building cluster collaboration method and system using embodied intelligent agents and multimodal large models. By using an embodied intelligent robot dog as a mobile sensing node, it completely breaks through the spatial coverage limitations of fixed sensor networks. At the same time, the robot dog performs cross-verification of fixed sensors through its own high-precision sensors, promptly detecting sensor drift or failure, ensuring that the data foundation for energy-saving decisions remains reliable. Furthermore, the embodied intelligent agent is not only a mobile sensor but also a near-field actuator, achieving short-loop feedback of "perception-reasoning-execution" at the second to minute level for localized, occasional energy waste. This avoids the long link delay of traditional BAS from sensors, host computer, controller, and actuator. Moreover, through digital twin game theory, it achieves distributed and optimized allocation of energy quotas among multiple buildings, avoiding the unfair effects of traditional "one-size-fits-all" power rationing on different buildings at the game level. Attached Figure Description

[0023] Figure 1 A schematic diagram of the structure of the first embodiment of the building cluster collaboration method and system of embodied intelligent agent and multimodal large model provided by the present invention;

[0024] Figure 2 Flowchart for energy-saving inspection and local fine-tuning of an embodied intelligent agent;

[0025] Figure 3A flowchart for distributed game theory and equilibrium selection in building clusters;

[0026] Figure 4 This is a schematic diagram of the structure of a second embodiment of a building cluster collaboration method and system for embodied intelligent agents and multimodal large models provided by the present invention;

[0027] Figure 5 It is a building cluster collaborative system that integrates embodied intelligent agents and multimodal large models. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] First Embodiment

[0030] Please refer to the following: Figure 1 , Figure 2 and Figure 3 ,in, Figure 1 A schematic diagram of the structure of the first embodiment of the building cluster collaboration method and system of embodied intelligent agent and multimodal large model provided by the present invention; Figure 2 Flowchart for energy-saving inspection and local fine-tuning of an embodied intelligent agent; Figure 3 A flowchart for distributed game theory deduction and equilibrium selection in building clusters. A collaborative method for building clusters using embodied agents and multimodal large models includes the following steps:

[0031] S1: Multimodal perception access: Collects numerical time-series data from fixed sensors, thermal images, RGB scene images and local environmental parameter data collected by the embodied intelligent agent (a quadruped robot dog equipped with multimodal sensors) during mobile inspections, as well as unstructured text data, and inputs them into a multimodal large model for joint encoding to form a cross-modal "human-machine-environment" joint semantic representation;

[0032] S2: Physical-Behavioral Dual-Domain Knowledge Enhanced Reasoning and Candidate Strategy Generation: Construct a domain knowledge system containing a physical domain knowledge base (thermal parameters of building envelope and cold energy distribution characteristics of HVAC system) and a behavioral domain knowledge base (historical patterns of personnel flow and energy consumption behavior patterns). Through retrieval-enhanced generation technology, the two types of knowledge are injected into the reasoning context window of the multimodal large model in real time to generate multiple candidate global energy-saving control strategies. Each strategy is accompanied by a natural language reasoning chain. At the same time, the multimodal large model judges energy-saving opportunities that can be dealt with in real time based on the local energy consumption anomalies discovered in real time during the inspection of the embodied intelligent agent, and generates embodied micro-adjustment instructions.

[0033] S3: Distributed Game Theory and Equilibrium Selection for Building Clusters: In the digital twin simulation environment of building clusters, each building is abstracted as an independent intelligent agent, and a multi-agent game model is established. The benefit function of each building agent includes the comfort maintenance benefit, energy consumption cost expenditure and contribution to global constraints. Candidate global control strategy frameworks are deployed to the simulation environment in a multi-round iterative manner. The large model acts as the simulation analyst to conduct multi-dimensional analysis of the game process and equilibrium results, and the global optimal solution is selected by Pareto optimality criterion.

[0034] S4: The optimal solution is broken down into control commands at the building equipment level and sent to the building automation system for execution. At the same time, the individual micro-adjustment commands are sent to the corresponding individual intelligent agents for execution. Energy-saving response is achieved through a dual-speed closed loop of slow-loop global control and fast-loop local micro-adjustment. During the inspection process, the individual intelligent agents continuously verify the execution effect. All execution results, inspection data and large model inference records are continuously fed back to update each knowledge base, forming a continuously evolving energy-saving intelligent agent system.

[0035] The embodied intelligent agent is a quadrupedal robot dog equipped with a multimodal sensor module and a near-field execution module;

[0036] Multimodal sensor modules include, but are not limited to, thermal imaging cameras, RGB-D cameras, LiDAR, temperature and humidity sensors, and CO2 sensors;

[0037] The near-field execution module is used to execute local micro-adjustment commands, including an infrared remote control module and / or a short-range wireless communication module; the embodied intelligent agent actively performs cross-verification of fixed sensors during the inspection process, and identifies drifting or malfunctioning fixed sensors by comparing its own sensor readings with the fixed sensor readings, and generates sensor health alarms.

[0038] The conditions for generating embodied fine-tuning instructions include, but are not limited to, the following: the robot dog's thermal imaging detects that an area is unoccupied but the air conditioning is still running, or the corridor lighting is fully on during non-use periods at night, or the thermal imaging window shows an abnormal temperature gradient indicating that doors and windows are not closed, resulting in the leakage of cold / heat. After the large model verifies the functional usage status of the area at the current time through the behavioral domain knowledge base, it determines whether it is allowed to execute the fine-tuning operation immediately. All embodied fine-tuning operations generate detailed operation logs, including thermal imaging comparison images before and after execution and environmental parameter change information, for global consistency verification and post-event auditing.

[0039] In the distributed game simulation and equilibrium selection process for building clusters, the large model acts as the simulation analyst, outputting a game process summary, Nash equilibrium identification results, Pareto boundary analysis, and strategy fragility evaluation report after each round of simulation. Finally, it uses the Pareto optimality criterion to select the equilibrium scheme with energy saving and comfort maintenance indicators from the candidate strategy set. The multi-dimensional analysis includes: load synergy effect between buildings, comfort level of each building, global response target achievement degree, and game equilibrium stability.

[0040] The multimodal large model outputs a natural language causal reasoning chain for each specific micro-adjustment instruction and global control strategy decision, covering three dimensions: "why save energy", "how to save energy", and "energy saving effect prediction". The reasoning chain and decision record are archived together with the scheduling plan to form a traceable, auditable, and interpretable energy saving decision baseline record, which supports post-event review and query and decision logic tracing through natural language interaction.

[0041] By constructing a four-layer closed loop encompassing "human-machine-environment three-dimensional perception, physical-behavioral dual-domain reasoning, building cluster game simulation, and embodied execution and security verification," a multimodal large model and an embodied intelligent agent form a "brain-body synergy"—the large model acts as the "brain," responsible for global reasoning and strategy generation; the robot dog network acts as the "body," responsible for motion perception and near-field execution; and the building digital twin acts as the "sandbox," carrying out game simulation.

[0042] Perception blind spot elimination and sensor health self-verification: The embodied intelligent robot dog, as a mobile sensing node, completely breaks through the spatial coverage limitations of fixed sensor networks. At the same time, the robot dog performs cross-verification of fixed sensors through its own high-precision sensors, promptly detecting sensor drift or failure, and ensuring that the data foundation for energy-saving decisions is always reliable.

[0043] Energy-saving response with dual-speed closed-loop: The robot dog is not only a motion sensor but also a near-field actuator. For localized, occasional energy waste (such as lighting in uninhabited areas, air conditioning running idle, and doors and windows left open), it achieves short-loop feedback of "perception-reasoning-execution" at the second to minute level, avoiding the long link delay of traditional BAS from sensor → host computer → controller → actuator.

[0044] Interpretable Strategy Generation: The multimodal large model outputs a natural language reasoning chain for each scheduling decision, covering three dimensions: "why save energy," "how to save energy," and "estimated energy-saving effect," thus overcoming the interpretability dilemma of existing "black-box optimization." During post-event review, operations personnel can use natural language to query the decision basis, and the large model provides an understandable explanation based on the complete decision baseline record.

[0045] Building Cluster Distributed Game Theory Optimization: Through digital twin game theory simulation, the distributed optimization allocation of energy consumption quotas among multiple buildings is achieved, avoiding the unfair effects of traditional "one-size-fits-all" power rationing on different buildings at the game theory level.

[0046] Dual safeguards for physical safety and behavioral rationality: Physical-behavioral dual-domain knowledge enhancement uses building physical constraints and human behavioral knowledge as "safety railings" for reasoning, ensuring both the physical feasibility of strategies and human comfort from the source of generation.

[0047] Seamless access and ease of implementation: This system can serve as a “smart energy-saving decision-making middleware layer” to connect with the building’s existing BAS, EMS, BIM and property management systems. The robot dog network can be deployed incrementally without the need for large-scale replacement of existing sensors and equipment.

[0048] The fixed sensor data mentioned in step S1 includes temperature, humidity, CO2 concentration, illuminance, electricity meter data for each floor, and energy consumption data for each item.

[0049] The mobile inspection data mentioned in step S1 includes thermal imaging images, RGB scene images, and local environmental parameter data.

[0050] The data of the embodied intelligent agent includes thermal images, RGB scene images, local temperature and humidity data, and voiceprint data that it actively collects during building inspections.

[0051] The domain knowledge system mentioned in step S2 includes a physical domain knowledge base and a behavioral domain knowledge base. The physical domain knowledge base consists of the thermal parameters of the building envelope and the cold energy distribution characteristics of the HVAC system. The behavioral domain knowledge base consists of the historical patterns of personnel movement and energy consumption behavior patterns.

[0052] The physical domain knowledge base includes, but is not limited to, the following parameters: building envelope thermal parameters, HVAC system cold energy distribution characteristics, lighting circuit illuminance-power relationship, elevator operating energy efficiency curve, heat capacity of each area, equipment safe temperature range, minimum equipment start-up and shutdown interval, and energy storage system charging and discharging efficiency. The behavioral domain knowledge base includes, but is not limited to, the following parameters: personnel density patterns in each area at different times, temperature preferences in different areas, lighting usage habits, meeting room reservation and usage patterns, and equipment operation permission rules.

[0053] The benefit function described in step S2 includes comfort maintenance benefits, energy consumption costs, and global constraints.

[0054] The dual-speed closed-loop described in step S4 includes slow-loop global control and fast-loop local fine-tuning to achieve energy-saving response.

[0055] The working principle of the building cluster collaboration method and system based on embodied intelligent agents and multimodal large models provided by this invention is as follows:

[0056] When using it, the first step is: "Human-Machine-Ring" multimodal sensing access and joint coding.

[0057] Fixed sensor data access: The system accesses real-time data from three buildings, A, B, and C, via the BAS interface, including temperature, humidity, CO2 concentration, power of lighting circuits on each floor, operating status and set parameters of each air conditioning terminal device, and power of the total electricity meter of each building. At the same time, it also accesses weather station data (outdoor temperature 35°C, humidity 70%).

[0058] Embodied intelligent agent mobile sensing data access:

[0059] During the inspection of S1 and A buildings, the robot dog discovered through thermal imaging that the ventilation fan in the 14th floor power distribution room had reduced heat dissipation efficiency due to dust accumulation, resulting in localized high temperatures and a potential risk of abnormally high energy consumption. The infrared remote control module scanned the split air conditioners and lighting panels on each floor and recorded the actual operating status and set parameters of the equipment.

[0060] During the inspection of S2 and C buildings, the robot dog found that the actual number of people in the employee activity center area was only about 25% of the full capacity. The lighting in a large area was fully turned on and the air conditioning was set to 23°C, resulting in significant energy waste. At the same time, it was found that a ventilation window was not closed, causing cold air to leak out. Thermal imaging showed that there was obvious cold air loss in the window area.

[0061] Unstructured text data access: The system reads equipment ledgers for each building (total installed capacity of air conditioning system, total power of lighting on each floor, elevator energy efficiency rating, etc.), historical personnel flow data, current time-of-use electricity price information, and the latest demand response policy documents.

[0062] Joint encoding: The multimodal large model performs unified encoding on the above multi-source heterogeneous data to form a joint semantic representation of "human-machine-environment" for the three buildings.

[0063] Key findings: 1. The data center in Building B has a rigid heat dissipation load with limited room for significant reduction; 2. Building C has a large area of ​​low-density usage, with the greatest potential for adjustment; 3. Building A is at a medium level, and the heat dissipation problem of the power distribution room needs to be taken into consideration.

[0064] Step 2: Enhanced Reasoning and Candidate Strategy Generation Based on Physical-Behavioral Dual-Domain Knowledge

[0065] Embodying Micro-Adjustment (Short-Loop Instant Execution): Regarding the localized waste issue detected by the robot dog in Building C—full lighting, air conditioning at 23°C, and open ventilation windows in the unoccupied area of ​​the employee activity center—the large model, based on behavioral domain knowledge (this area's functional hours are 14:00-17:00, usage patterns show the lowest personnel density on weekday afternoons, and the current summer internship season further reduces the number of people indoors), determined that the lighting and air conditioning operation in this area is severely mismatched with current actual usage needs. The large model issued an instant micro-adjustment instruction:

[0066] S1: Send a command to the lighting panel via the near-field infrared remote control module of the robot dog in Building C to turn off 70% of the lighting circuits in unoccupied areas, leaving only the basic directional lighting on;

[0067] S2: Adjust the temperature setting of the split air conditioner in this area to 26°C (up from 23°C) via the infrared remote control module.

[0068] S3: Notify the property management system to close the ventilation windows to eliminate cold air leakage. Simultaneously record a complete operation log, including thermal imaging comparison images before and after the adjustment collected by the robot; this is a short-loop response.

[0069] Global candidate strategy generation (long-loop reasoning): The RAG module retrieves relevant constraints from the physical domain knowledge base (thermal parameters of building envelopes, air conditioning cooling capacity distribution characteristics, data center heat dissipation requirements, etc.) and the behavioral domain knowledge base (historical personnel flow patterns and energy consumption patterns of buildings, functional usage time characteristics of each area, etc.) and injects them into the reasoning context of the large model.

[0070] Based on the demand response target (a 15% reduction in total electricity load) and the adjustability assessment of the three buildings, and taking into full account the partial energy-saving effect already achieved by the short-loop fine-tuning of Building C (accounting for approximately 15% of the total response target), the large model generates the following candidate global control strategies:

[0071] Strategy A (Peak Shaving): The load on each of the three buildings will be reduced by 15% proportionally to the current load ratio. Due to the rigid load of the data center in Building B, this plan is expected to lead to insufficient cooling supply from the air conditioning in Building B and an increased risk of abnormal temperature in the computer room.

[0072] The large model generates the inference chain: "The load of the data center in Building B cannot be reduced on a large scale. Proportional distribution will essentially include the rigid load of Building B in the reduction scope, which may not only trigger the overheating alarm of the computer room, but also make it impossible to achieve the expected actual load reduction target of 15%. This solution is inefficient and risky in scenarios with significant heterogeneity in load composition, and is not recommended."

[0073] Strategy B (Adjustment-Transfer Priority): Keep Building B basically unchanged, Buildings A and C bear 20% and 25% of the pressure drop respectively, and in Building C, the air conditioning temperature setting in non-use areas (floors with low current occupancy rate) will be uniformly increased to 26°C, and lighting will be reduced by 50%; Building A will adjust its pre-cooling strategy, making full use of the building's heat capacity to moderately enhance pre-cooling before the response, and increase the air conditioning set temperature and reduce the fresh air volume during the response period.

[0074] The large model generates the inference chain: "The rigid load constraint of Building B has been confirmed by the physical domain knowledge base, and it is not suitable to participate in large-scale adjustments; the low occupancy rate of Building C (provided by the behavioral domain knowledge base) indicates that it has the largest adjustable flexibility, and this scheme is relatively balanced between physical feasibility and personnel comfort."

[0075] Strategy C (Energy Storage Substitution): In addition to the measures in Strategy B, the ice storage system deployed in Building A and some energy storage equipment in Building C will be used to release the stored cold energy during demand response periods to replace the electricity used for air conditioning, thereby further reducing the electricity load.

[0076] The large model generates the inference chain: "The ice storage system in Building A has the characteristics of storing energy in advance and releasing energy during the response period. The energy storage in Building C is currently in good charge state. Activating the energy storage equipment to replace the refrigeration can further tap the load reduction depth of the two buildings outside Building B, providing an additional 10%-15% adjustment margin for the whole system."

[0077] Step 3: Distributed Game Theory Deduction and Equilibrium Selection for Building Clusters

[0078] The system launches the digital twin simulation environment for the building cluster, modeling Building A, Building B, and Building C as independent intelligent agents.

[0079] The goal of the agent in Building A is to minimize its own energy consumption costs while ensuring that the comfort level does not fall below the threshold. The agent in Building B takes the safety of the computer room temperature as a rigid constraint, and the agent in Building C takes flexible load adjustment as the main means.

[0080] The global response objective (a 15% reduction in total load) and three candidate strategy frameworks are deployed to the twin environment. Each agent adjusts its internal action strategy under its own physical and behavioral constraints and engages in multiple rounds of adversarial competition.

[0081] S1: Under Strategy A: The agent in Building B is forced to withdraw from the game due to the temperature warning in the computer room caused by the forced reduction of the agent's work, so the solution is not feasible.

[0082] S2: Under Strategy B: The agents in Buildings A and C reach a Nash equilibrium at the edge of the comfort threshold, and the overall load decreases by about 14%, which is close to the target but still slightly insufficient;

[0083] S3: Under Strategy C: The combination of ice storage and energy storage release provides additional strategic space for agents in Building A and Building C, while agent in Building B can also participate safely with minimal adjustments. All three achieve Pareto optimality in terms of comfort, energy costs, and global constraint contributions.

[0084] The large-scale model's analysis of the generative game theory is summarized as follows: "The low population density of Building C in the current period gives it a load elasticity far exceeding that of Buildings A and B. In Strategy B, Building C is already close to its own adjustable limit, and the remaining gap needs to be found in Building A. Strategy C utilizes the time-shifting characteristics of energy storage equipment to open up additional adjustment space in Building A, enabling the overall goal to be achieved at a higher level of comfort. Under the Nash equilibrium state corresponding to Strategy C, any unilateral deviation will harm the interests of the deviating party (a decrease in the comfort level of Building A or the triggering of a temperature alarm in Building B), therefore, this equilibrium is stable."

[0085] The large model generates a complete inference record, which is saved as an audit baseline and supports natural language review of the game process when there is a dispute.

[0086] Step 4: Strategy Optimization, Security Verification, and Cross-Building Command Issuance

[0087] The system uses the Pareto optimality criterion to select strategy C as the final coordinated control scheme for the building group. The safety constraint verification module verifies the control plan for each building separately:

[0088] Building A: Verify whether the cooling capacity during the pre-cooling-cooling-release switching process meets the lower limit of comfort, and ensure that the temperature of each floor does not exceed the set range; at the same time, take the heat dissipation problem of the power distribution room discovered by the robot dog in step one into the scheduling considerations - when adjusting the pre-cooling strategy, appropriately reduce the heat load of this area to avoid electrical faults caused by poor heat dissipation and insufficient cooling.

[0089] Building B: Verify the cooling continuity of the data center server room; N+1 redundancy meets all requirements.

[0090] Building C: Verify the impact of large-area temperature increases on the comfort of a small number of people in the room – Based on the behavioral domain knowledge base, it was confirmed that the current location of people in the room is concentrated in area A, while the temperature increase mainly adjusts areas B / C, and the impact on comfort is controllable.

[0091] After verification, the system breaks down the global solution into control instructions for each building's equipment level (air conditioning temperature setting curves for each floor, lighting zone control plans, ice storage / energy release scheduling plans, etc.), and sends them to the BAS system of each building for execution via an encrypted channel.

[0092] During the process, the robot dogs in each building continuously patrol and verify the control effect in real time through thermal imaging and temperature and humidity sensors. If any deviation is found, it is immediately fed back to the large model for analysis and correction.

[0093] Step 5: Human Interaction and Explainable Debriefing Application

[0094] The following day, the property management staff used natural language interaction to query the system: "How much did the energy consumption of Building C decrease yesterday afternoon? What measures were mainly taken? Did it affect the experience of people inside? Did our overall response meet the standards?"

[0095] The large model responded based on the complete decision-making baseline: "Yesterday afternoon from 14:30 to 16:30, our park successfully responded to the power company's demand response signal, and the total load decreased by 16.8%, exceeding the target of 15%."

[0096] Building C contributed approximately 40% of the load reduction, mainly through the following measures: 1. Instantaneous short-loop fine-tuning by the robot dog – 70% of the lighting in the uninhabited area was turned off, the set temperature of the split air conditioner was increased from 23°C to 26°C, and ventilation windows were closed to eliminate cold leakage.

[0097] 2. BAS Global Control – Unified temperature control and zonal lighting control for air conditioning in large, low-density areas. Regarding the impact on comfort: Follow-up questionnaires of occupants showed no complaints of discomfort caused by short-term temperature increases in the afternoon. This is mainly because the system accurately identified the actual location distribution of occupants, limiting temperature adjustments to vacant areas. Today, the knowledge base has automatically updated the parameter records of thermal response characteristics for different areas of Building C based on actual execution data.

[0098] Compared with related technologies, the building cluster collaboration method and system provided by this invention, which combines embodied intelligent agents and multimodal large models, has the following beneficial effects:

[0099] This invention provides a building cluster collaboration method and system using embodied intelligent agents and multimodal large models. By using an embodied intelligent robot dog as a mobile sensing node, it completely breaks through the spatial coverage limitations of fixed sensor networks. At the same time, the robot dog performs cross-verification of fixed sensors through its own high-precision sensors, promptly detecting sensor drift or failure, ensuring that the data foundation for energy-saving decisions remains reliable. Furthermore, the embodied intelligent agent is not only a mobile sensor but also a near-field actuator, achieving short-loop feedback of "perception-reasoning-execution" at the second to minute level for localized, occasional energy waste. This avoids the long link delay of traditional BAS from sensors, host computer, controller, and actuator. Moreover, through digital twin game theory, it achieves distributed and optimized allocation of energy quotas among multiple buildings, avoiding the unfair effects of traditional "one-size-fits-all" power rationing on different buildings at the game level.

[0100] Second Embodiment

[0101] Please refer to the following: Figure 4 Based on the first embodiment of this application, which provides a method and system for building group collaboration using embodied intelligent agents and multimodal large models, the second embodiment of this application proposes another method and system for building group collaboration using embodied intelligent agents and multimodal large models. The second embodiment is merely a preferred embodiment of the first embodiment, and the implementation of the second embodiment will not affect the separate implementation of the first embodiment.

[0102] Specifically, the difference between the building group collaboration method and system of embodied intelligent agent and multimodal large model provided in the second embodiment of this application is that it also includes, in step S1, the embodied intelligent agent inspection performs real-time three-dimensional scanning of the building, constructs a three-dimensional model of the inspection area in real time, and marks the fault points and transmits them to the mobile terminal.

[0103] The embodied intelligent agent transmits 3D models, fault marking points, real-world images, and thermal imaging data to a handheld mobile device in real time via wireless data transmission.

[0104] In the multimodal perception access stage, the embodied intelligent agent is equipped with LiDAR, depth camera and high-definition vision module during building inspection to perform real-time three-dimensional spatial scanning, dynamic point cloud acquisition and three-dimensional model reconstruction of the inspection area, and complete the accurate labeling of fault points and synchronization with mobile terminal.

[0105] Real-time 3D scanning and dynamic modeling of inspection areas

[0106] During autonomous inspections, the embodied intelligent agent uses multi-line LiDAR to continuously and densely scan the indoor spaces, corridors, computer rooms, equipment rooms, ceiling mezzanines, pipe shafts, and other areas of the building. Simultaneously, it integrates texture and spatial depth information acquired by an RGB-D depth camera to build a high-precision, lightweight 3D model of the inspection area in real time.

[0107] The modeling process employs an incremental real-time reconstruction algorithm, which dynamically splices, completes, and optimizes the spatial model along the inspection path, automatically eliminating duplicate areas and filling in scanning blind spots. The model accuracy can reach the centimeter level, fully restoring key spatial information such as building spatial structure, wall outline, equipment location, pipeline routing, and door and window status.

[0108] The constructed 3D model not only contains spatial geometric information, but also synchronously links multimodal sensing data such as real-time temperature and humidity, thermal imaging temperature field, energy consumption data, and equipment operating status, forming an integrated "space-data" 3D semantic model, providing a precise spatial foundation for subsequent fault location, energy consumption analysis, and strategy deduction.

[0109] Automatic fault point identification, accurate labeling, and real-time transmission to mobile devices:

[0110] While performing 3D scanning and modeling, the embodied intelligent agent relies on multimodal large models for real-time reasoning, integrating thermal imaging, visible light images, sound patterns, vibration, and energy consumption data to automatically identify various anomalies and fault points within the building, including but not limited to: equipment overheating, pipe insulation detachment, doors and windows not closed, lights always on, air conditioners running idle, fan noise, loose wiring, sensor drift, cold air leakage, and airflow short circuits.

[0111] After identifying the fault point, the system automatically generates high-precision spatial coordinate annotations in the real-time 3D model, and overlays fault type labels, risk level, energy loss quantification value, on-site real scene map, and thermal imaging abnormal area to form structured fault annotation information.

[0112] After annotation, the embodied intelligent agent transmits the 3D model, fault annotation points, real-world images, and thermal imaging data to the handheld mobile terminal (phone / tablet) of maintenance personnel in real time via a high-speed, low-latency wireless data (5G / Wi-Fi) communication link. The mobile terminal can rotate and zoom the 3D model, jump to the fault point location, view annotation information, compare the real-world and thermal images, and backtrack historical faults. This allows maintenance personnel to intuitively grasp the fault location, severity, and on-site situation without going to the site, providing visual support for rapid repair and accurate handling.

[0113] A building cluster collaborative system of embodied intelligent agents and multimodal large models includes the building cluster collaborative method of the embodied intelligent agents and multimodal large models, comprising: a multimodal perception access module, a physical-behavioral dual-domain knowledge enhancement reasoning module, a building cluster game deduction and equilibrium selection module, and an embodied execution and security verification closed-loop control module.

[0114] Compared with related technologies, the building cluster collaboration method and system provided by this invention, which combines embodied intelligent agents and multimodal large models, has the following beneficial effects:

[0115] This invention provides a building cluster collaboration method and system using embodied intelligent agents and multimodal large models. The embodied intelligent agents perform real-time 3D scanning of buildings during inspections and construct 3D models of the inspection area in real time. Fault points are marked and transmitted to mobile devices, thereby enabling maintenance personnel to quickly locate fault points for repairs via mobile devices and increasing maintenance efficiency.

[0116] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for collaborative building clusters using embodied intelligent agents and multimodal large models, characterized in that, Including: the following step: S1: Multimodal perception access: Collect numerical time-series data from fixed sensors, mobile inspection data from embodied intelligent agents, and unstructured text data, and input them into a multimodal large model for joint encoding; S2: Physical-Behavioral Dual-Domain Knowledge Enhanced Reasoning and Candidate Strategy Generation: Construct a domain knowledge system and inject knowledge into the reasoning context window of the multimodal large model in real time through retrieval enhancement generation technology to generate multiple candidate global energy-saving control strategies. At the same time, the multimodal large model judges energy-saving opportunities that can be dealt with in real time based on the local energy consumption anomalies discovered in the inspection of the embodied intelligent agent and generates embodied micro-adjustment instructions. S3: Distributed game simulation and equilibrium selection of building clusters: Establish a multi-agent game model, the benefit function of each building agent, and put the candidate global control strategy framework into the simulation environment in a multi-round iterative manner. The large model acts as the simulation analyst to conduct multi-dimensional analysis of the game process and equilibrium results, and selects the global optimal solution by Pareto optimality criterion. S4: Embodied Execution and Safety Verification Closed Loop: The optimal solution is broken down into control commands at the building equipment level and sent to the building automation system for execution. At the same time, embodied micro-adjustment commands are sent to the corresponding embodied intelligent agents for execution. Energy-saving response is achieved through a dual-speed closed loop. The embodied intelligent agents continuously verify the execution effect during the inspection process. All execution results, inspection data, and large model inference records are continuously fed back and updated to various knowledge bases, forming a continuously evolving energy-saving intelligent agent system.

2. The building cluster collaboration method based on an embodied intelligent agent and a multimodal large model according to claim 1, characterized in that, The fixed sensor data mentioned in step S1 includes temperature, humidity, CO2 concentration, illuminance, electricity meter data for each floor, and energy consumption data for each item.

3. The building cluster collaboration method based on an embodied intelligent agent and a multimodal large model according to claim 2, characterized in that, The mobile inspection data mentioned in step S1 includes thermal imaging images, RGB scene images, and local environmental parameter data.

4. The building cluster collaboration method based on an embodied intelligent agent and a multimodal large model according to claim 1, characterized in that, The domain knowledge system mentioned in step S2 includes a physical domain knowledge base and a behavioral domain knowledge base. The physical domain knowledge base consists of the thermal parameters of the building envelope and the cold energy distribution characteristics of the HVAC system. The behavioral domain knowledge base consists of the historical patterns of personnel movement and energy consumption behavior patterns.

5. The building cluster collaboration method based on an embodied intelligent agent and a multimodal large model according to claim 4, characterized in that, The physical domain knowledge base includes building envelope thermal parameters, HVAC system cold energy distribution characteristics, lighting circuit illuminance-power relationship, elevator operation energy efficiency curve, heat capacity of each area, equipment safe temperature range, minimum equipment start-up and shutdown interval, and energy storage system charging and discharging efficiency. The behavioral domain knowledge base includes personnel density patterns in each time period and area, temperature preferences in different areas, lighting usage habits, meeting room reservation and usage patterns, and equipment operation permission rules.

6. The building cluster collaboration method of embodied intelligent agents and multimodal large models according to claim 1, characterized in that, The benefit function described in step S2 includes comfort maintenance benefits, energy consumption costs, and global constraints.

7. The building cluster collaboration method based on an embodied intelligent agent and a multimodal large model according to claim 1, characterized in that, The dual-speed closed-loop described in step S4 includes slow-loop global control and fast-loop local fine-tuning to achieve energy-saving response.

8. The building cluster collaboration method of embodied intelligent agents and multimodal large models according to claim 1, characterized in that, In step S1, the embodied intelligent agent inspection involves real-time 3D scanning of the building, real-time construction of a 3D model of the inspection area, and annotation of fault points, which are then transmitted to the mobile terminal.

9. The building cluster collaboration method of embodied intelligent agents and multimodal large models according to claim 8, characterized in that, The embodied intelligent agent transmits 3D models, fault marking points, real-world images, and thermal imaging data to a handheld mobile device in real time via wireless data transmission.

10. A building cluster collaborative system of embodied intelligent agents and multimodal large models, comprising the building cluster collaborative method of embodied intelligent agents and multimodal large models as described in any one of claims 1-9, characterized in that, include: The system includes a multimodal perception access module, a physical-behavioral dual-domain knowledge enhancement reasoning module, a building cluster game deduction and equilibrium selection module, and an embodied execution and security verification closed-loop control module.

Images