Smart park energy-saving cooperative regulation and control system and method based on multi-modal large model

Abstract

The invention discloses a smart park energy-saving cooperative regulation and control system and method based on a multi-mode large model, and belongs to the technical field of intelligent management of the Internet of Things. The method comprises the following steps: acquiring vision, internet of things, text and voice multi-mode perception data; performing feature extraction on the data to obtain multi-modal features; taking the text features and the audio features as result-oriented features, vectorizing the visual features and the Internet of Things features, and inputting the vectorized visual features and Internet of Things features into a fusion function of a multi-modal large model; generating a unified state representation vector through the fusion function, and inputting the unified state representation vector into a regulation and control decision module to generate a coordinated regulation and control instruction; and controlling the park equipment according to the instruction. Through multi-modal feature fusion and unified representation, cross-subsystem joint optimization and dynamic priority regulation and control are realized, and the park energy utilization efficiency and the intelligent operation and maintenance level are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of IoT intelligent management, and in particular to a method, system, device and storage medium for energy-saving collaborative control of smart parks based on a multimodal large model. Background Technology

[0002] With the rapid development of IoT technology, traditional IoT systems can now achieve interconnection and data collection and transmission of physical devices through various sensors, communication modules, and network protocols. Existing technologies typically employ centralized or distributed data platforms to aggregate the collected data and process, analyze, and issue early warnings using preset rules or traditional machine learning models (such as regression models and simple neural networks). For example, in the field of energy management in smart parks, the system can monitor the operating parameters and environmental information of subsystems such as HVAC, lighting, and microgrids in real time, and notify relevant personnel via SMS, email, or platform alarms when data anomalies occur. Some systems can also perform simple automatic adjustments based on preset rules.

[0003] However, such systems still have significant limitations in practical applications. First, the system's decision-making loop heavily relies on human intervention. Furthermore, each subsystem (HVAC, lighting, microgrid) optimizes independently, lacking global coordination and forming a "siloed" optimization pattern. When cross-system strategy conflicts arise, personnel struggle to coordinate quickly, impacting the overall energy-saving effect and operational reliability. Second, fault handling and decision-making processes heavily depend on the operator's personal experience. Due to the lack of systematic summarization and effective accumulation of historical handling cases and expert experience, it is difficult to form a standardized decision support knowledge base. When faced with complex or long-term dynamic situations such as equipment performance degradation and changes in usage habits, personnel often have to rely solely on their own experience, resulting in low efficiency and a high risk of decision-making bias due to insufficient experience.

[0004] Furthermore, the system's sensing capabilities are relatively limited, making it difficult to integrate multimodal data for comprehensive analysis. Existing technologies typically only process structured numerical sensor data, lacking semantic-level fusion capabilities for multi-source heterogeneous data such as video streams, natural language work orders, equipment drawings, and weather reports.

[0005] Therefore, while achieving data interconnection and interoperability, the current IoT technology framework urgently needs to break through the bottleneck of relying on manual decision-making. By introducing more intelligent data analysis, decision support, and automated execution mechanisms, it is necessary to achieve global collaborative optimization across systems, fusion perception of multimodal data, and interpretability of the decision-making process. This will improve the system's autonomous response capability, processing efficiency, and reliability, and truly realize cost reduction, efficiency improvement, and intelligent operation and maintenance of smart parks. Summary of the Invention

[0006] The present invention aims to overcome at least one of the defects of the prior art and provide a smart park energy-saving collaborative control system and method based on a multimodal large model, which solves the technical problems of the existing system having a single perception capability and relying on manual decision-making.

[0007] This invention provides a smart park energy-saving collaborative control method based on a multimodal large model, comprising:

[0008] S1: Acquire multimodal perception data, which includes visual perception data, IoT perception data, text perception data, and voice perception data; S2: Perform feature extraction on the visual perception data, IoT perception data, text perception data and voice perception data respectively to obtain visual feature V_t, IoT feature I_t, text feature T_t and audio feature A_t, where t represents the time point; S3: The text feature T_t and the audio feature A_t are used as result-oriented features. The visual feature V_t and the IoT feature I_t are converted into vector form through vectorization and then input into the fusion function F in the multimodal large model. S4: Generate a unified state representation vector S_t through the fusion function F, and input S_t into the control decision module in the multimodal large model to generate collaborative control instructions for park equipment.

[0009] S5: Control the park equipment according to the aforementioned coordinated control instructions to achieve energy-saving control.

[0010] By constructing a complete closed loop from multimodal data acquisition, fusion representation, multi-step prediction and anomaly detection, to collaborative optimization decision-making, instruction issuance and execution, real-time monitoring and evaluation, and continuous model learning, the system achieves a self-optimizing intelligent closed loop from data acquisition to continuous learning.

[0011] In step S1, multimodal perception data is acquired, including visual perception data, IoT perception data, text perception data, and voice perception data.

[0012] The visual perception data includes at least personnel distribution data and equipment status data obtained through video stream analysis; The IoT sensing data includes at least time-series data collected by sensors; The text-aware data includes at least text parsing data such as operation and maintenance logs, weather reports, and shift schedules. The voice interaction data includes at least the natural language commands input by the administrator.

[0013] In step S2, feature extraction is performed on the visual perception data, IoT perception data, text perception data, and voice perception data respectively to obtain visual feature V_t, IoT feature I_t, text feature T_t, and audio feature A_t, where t represents a time point.

[0014] In step S3, the text feature T_t and the audio feature A_t are used as result-oriented features. After the visual feature V_t and the IoT feature I_t are converted into vector form through vectorization, they are input into the fusion function F in the multimodal large model to generate a unified state representation vector S_t.

[0015] The visual feature V_t is constructed by fusing multi-dimensional visual information, and its expression is: V_t = Concat(V_occ, V_therm, V_act); Where V_occ represents the population density characteristic, V_therm represents the heat distribution characteristic, and V_act represents the population activity intensity characteristic.

[0016] The personnel density feature V_occ is used to characterize the spatial density of personnel distribution in the current scene.

[0017] First, an object detection network is used to detect pedestrians in visible light images.

[0018] Preferably, a YOLOv8 model is used, and pre-trained weights are used to fine-tune the pedestrian categories to obtain the bounding boxes of all people in the current frame Img_t, and the output is represented as follows. .

[0019] Preferably, based on this, the personnel density is calculated. When the camera calibration parameters are known, the center coordinates of each detection box are mapped to the real-world coordinate system, the actual area A of the scene (in square meters) is calculated, and the ratio of the number of detection boxes N (the number of people after occlusion processing) to the area A is used as the density d = N / A. When the camera parameters are unknown, pixel density can be used as an approximation.

[0020] Preferably, to further convert the density scalar into a feature vector, a mapping network is used for processing. Specifically, the density value is discretized into multiple intervals, represented by one-hot encoding or embedded vectors; or a multilayer perceptron (MLP) is used to map the scalar density into a D_occ-dimensional feature vector, the expression of which is: ; The thermal distribution feature V_therm is used to extract features that reflect temperature distribution patterns from thermal imaging images.

[0021] First, preprocessing is performed to convert the thermal image (Thermal_t) into a format suitable for ResNet50 input.

[0022] Preferably, the original temperature values ​​are normalized to the [0,1] interval and converted into a pseudo-color image (such as COLORMAP_JET using OpenCV) to utilize a ResNet50 model pre-trained on ImageNet.

[0023] In the feature extraction stage, a pre-trained ResNet50 is used as the backbone network. Its final classification layer is removed, and the preprocessed heatmap is input to obtain feature maps. Global average pooling yields a 2048-dimensional feature vector. To reduce dimensionality and facilitate subsequent fusion, a fully connected layer is added after ResNet50, reducing the 2048-dimensional features to D_therm dimensions, expressed as: ; The activity intensity feature V_act is used to analyze the activity status of people in a scene based on human posture sequences, including intensity levels such as sitting, walking, and gathering.

[0024] First, pose estimation is performed, using pose estimation models such as OpenPose, MediaPipe, or HRNet to extract the coordinates of human keypoints in each frame of the visible light image sequence. After obtaining the keypoint sequence within a fixed-length time window (e.g., the past 10 frames), the poses of representative individuals in the scene (e.g., individuals in the central area) are selected, or average pooling is applied to the poses of all individuals.

[0025] The pose of each frame is represented as a feature vector, which is then input into an LSTM network for sequence modeling. The LSTM network takes the pose feature vector at time step t as input and outputs the hidden state at the last time step. The hidden layer dimension is set to H, and the final output is an H-dimensional vector.

[0026] To generate features related to activity intensity, a fully connected layer is connected to the LSTM hidden state, outputting a D_act dimensional feature vector, expressed as:

[0027] Preferably, during model training, an auxiliary classification loss can be introduced to explicitly classify activity intensity (such as sitting, walking, and gathering) to improve feature representation ability. The final features used for splicing are the feature vectors output by the fully connected layer.

[0028] The IoT feature I_t is constructed by decomposing sensor time-series data, and its expression is: I_t = [I_trend, I_season, I_resid] = STL_Decompose(Sensor_t); Simultaneously, it further derives the building-level aggregation feature I_building and the system-level decomposition feature I_system, where: I_building = Σ(w_i × I_zone_i); I_system = [I_HVAC, I_light, I_plug]; The above decomposition process breaks down the total load into trend, seasonal, and residual components, and models the influencing factors of each component separately. Specifically, the trend component is mainly affected by scheduling, the seasonal component by weather changes, and the residual component by unforeseen events.

[0029] Preferably, the sensor time series Sensor_t is preprocessed before performing time series decomposition.

[0030] Preprocessing includes: imputing missing values ​​using linear interpolation, forward filling, or the mean of neighboring sensors; removing outliers using statistical methods or iterative detection based on residuals; resampling multi-source sensor data to a uniform time frequency; and Z-score normalization of sensor data with different dimensions.

[0031] Preferably, the present invention provides two decomposition methods.

[0032] Scheme 1 is a sensor-by-sensor decomposition, which involves performing STL decomposition on each sensor within each region to obtain its respective trend, seasonal, and residual components, and then using a weighted average to obtain the comprehensive characteristics of the region. This scheme preserves the detailed information of each sensor.

[0033] Option 2 is to first aggregate and then decompose, that is, first merge all sensor data in the same area into a representative value according to domain knowledge (such as summing energy consumption and averaging temperature), and then perform STL decomposition on the representative sequence of the area.

[0034] For each sensor, the decomposition yields three components: trend, seasonality, and residual, each of which is a time series. The component value at the current time t is extracted as the feature value for that time, and feature construction is performed.

[0035] For region i, if the region contains m sensors, then the feature matrix is ​​obtained:

[0036] To reduce feature dimensionality, the sensor components within the same region are fused, and the trend components are weighted and averaged to obtain the regional trend feature T_i(t). Similarly, the seasonal feature S_i(t) and residual feature R_i(t) are obtained, giving each region a three-dimensional feature vector.

[0037] Preferably, based on this, a building-level aggregated feature I_building is constructed, which is achieved by weighted aggregation of features of each region. The weight w_i is determined according to the region area, energy consumption ratio or importance.

[0038] Preferably, a system-level decomposition feature I_system is constructed, which divides the sensor data into HVAC systems, lighting systems, and plug-in load systems according to equipment type, and extracts the energy consumption characteristics of each system to form a system-level feature vector.

[0039] The text feature T_t is constructed by fusing multi-source text information, and its expression is: T_t = [T_event, T_schedule, T_rule]; Where T_event represents the event semantic embedding, T_schedule represents the structured schedule feature, and T_rule represents the energy-saving rule encoding.

[0040] The event semantic embedding T_event is used to convert unstructured event description text into semantic feature vectors. The input event description text includes natural language text such as meeting notices, alarm information, maintenance work orders, and weather reports.

[0041] Preferably, a pre-trained BERT model is used as the semantic encoder. The event description text Event_desc is input into the BERT model, and the output vector corresponding to the [CLS] position is extracted as the semantic representation of the text. A fully connected layer is added after the BERT output for dimensionality reduction to obtain the event semantic embedding vector.

[0042] The structured schedule feature T_schedule is used to represent scheduled activity information within the park, including structured data such as meetings, events, and schedules. Input features include meeting type (e.g., regular meetings, important meetings, video conferences, etc.) and number of participants.

[0043] Preferably, the meeting type is represented by one-hot encoding or embedded vector, and the number of participants is normalized and mapped to the [0,1] interval. The type encoding and the normalized number of participants are concatenated to form the schedule feature vector.

[0044] Preferably, when there are many types of meetings, an embedding layer can be used instead of one-hot encoding to reduce feature dimensionality and capture semantic relationships between types.

[0045] The energy-saving rule code T_rule is used to characterize the currently effective energy-saving strategy and constraints. The currently active rule is encoded according to a preset rule base.

[0046] Preferably, the encoding method includes: using multi-hot encoding to represent the effective status of each rule in the rule base to obtain a binary feature vector; extracting key parameters of the rule (such as temperature setpoint, lighting illuminance threshold, start and stop time period, etc.) and concatenating and fusing them with the rule type encoding to form a rule feature vector.

[0047] The feature vectors T_event, T_schedule, and T_rule are concatenated to obtain the final text feature T_t.

[0048] The audio feature A_t is constructed by fusing semantic understanding and urgency recognition, and its expression is: A_t = [A_semantic, A_urgency]; Here, A_semantic represents semantic understanding features, and A_urgency represents urgency features.

[0049] The semantic understanding feature A_semantic is used to convert the speech content in the audio stream into a semantic vector representation.

[0050] Preferably, the acquired audio stream Audio_t is input into the speech recognition model Whisper, transcribed into text, and then mapped into a fixed-dimensional semantic embedding vector using a pre-trained language model (such as BERT).

[0051] The urgency feature A_urgency is used to identify the urgency level of the tone in the audio.

[0052] Preferably, the acoustic features of the audio, such as pitch and volume, are analyzed in real time, and a classifier outputs a continuous value between 0 and 1, representing the urgency of the current audio event, where 0 represents normal and 1 represents urgent.

[0053] The audio feature A_t is obtained by concatenating the above feature vectors A_semantic and A_urgency.

[0054] By constructing visual features V_t that fuse personnel density, heat distribution features, and activity intensity features, and constructing IoT features I_t that perform STL decomposition on sensor time-series data and extract trend, seasonal, and residual terms, and constructing text features T_t that fuse event semantic embedding, structured schedule features, and energy-saving rule encoding, and constructing audio features A_t that fuse semantic understanding features and urgency features, a unified representation and comprehensive analysis of multimodal data such as video streams, sensor data, text work orders, and voice commands is achieved. This enables multimodal perception and reasoning capabilities that can simultaneously process multi-source heterogeneous data and discover implicit correlations.

[0055] In step S4, a unified state representation vector S_t is generated through the fusion function F, and S_t is input into the control decision module in the multimodal large model to generate collaborative control instructions for park equipment.

[0056] Preferably, the unified state representation vector S_t is input into the load prediction module in the multimodal large model.

[0057] Preferably, the load forecast result is input into the control decision module to construct the multi-objective optimization function, and the formula for constructing the multi-objective optimization function is as follows: min[α·E_total + β·C_carbon + γ·D_comfort]; Where E_total represents the total energy consumption index, C_carbon represents the carbon emission index, D_comfort represents the comfort deviation index, and α, β, and γ are the corresponding weighting coefficients. The weighting coefficients α range from 0.3 to 0.6, β range from 0.2 to 0.5, and γ range from 0.1 to 0.4.

[0058] Preferably, the total energy consumption index E_total is calculated in the following way: E_total = Σ(Equipment power × Operating time) + Fixed base energy consumption; The carbon emission index C_carbon is calculated in the following way: C_carbon = Σ(energy consumption × corresponding carbon emission factor); The comfort deviation index D_comfort includes at least the temperature deviation, which is calculated in the following way: Temperature deviation = .

[0059] Preferably, the fusion function F is implemented by methods such as splicing and mapping through a fully connected layer, weighted fusion based on an attention mechanism, or multimodal alignment fusion, so as to unify multi-source heterogeneous features into the same semantic space and form a comprehensive understanding of the current state of the park.

[0060] The expression for the load forecast output of the control decision module, including the sub-item loads and forecast confidence intervals, is as follows: L_pred(t+τ) = Decoder(F_fusion, H_t) + Uncertainty(t); Where L_pred(t+τ) = [L_HVAC, L_light, L_plug, L_total] represents the sub-item load prediction results, corresponding to the load of the HVAC system, the load of the lighting system, the load of the socket, and the total load, respectively; Decoder is a decoder network used to map the fusion feature F_fusion and the historical state H_t to the load prediction value; Uncertainty(t) = Quantile(F_fusion) represents the prediction confidence interval, which is calculated from the fusion feature by the quantile regression method.

[0061] Preferably, the decision-making layer uses a dynamic priority matrix to adjust the optimization strategy in real time based on the multimodal perception results, expressed by the following formula: P(t) = Softmax(W_p · [S_t, L_pred, U(t), C_user]); Where P(t) represents the dynamic priority distribution of each control strategy at the current moment; S_t is the current state vector, obtained by fusing the aforementioned four modal features; L_pred is the predicted load, which includes the predicted results of the sub-items; U(t) is the user urgency level, derived from the urgency level recognition in the audio features; C_user is the user preference configuration, including personalized settings such as comfort requirements, energy-saving preferences, and device usage permissions; W_p is a learnable weight matrix, and the Softmax function maps the input to a normalized priority distribution.

[0062] When the system detects that the user's urgency level U(t) is high, the priority distribution automatically tilts towards the corresponding response strategy; when the predicted load L_pred is close to the power consumption limit, the priority of energy-saving strategies is automatically increased; when the user preference configuration C_user explicitly requires ensuring the comfort level of a certain area, the control priority of the relevant equipment in that area is increased accordingly.

[0063] Preferably, the contribution of the above factors to carbon emissions is quantified by Transformer attention weights to generate a visualized carbon flow map and a natural language interpretation report, which includes at least one optimization strategy.

[0064] The four feature vectors are concatenated by the fusion function F and then mapped through a fully connected layer. The resulting vectors are weighted fusion based on the attention mechanism or multimodal alignment fusion to generate a unified state representation vector S_t for the park. S_t is then input into the control decision module of the multimodal large model to achieve refined output of load forecasting. Furthermore, the decision layer uses a dynamic priority matrix to adjust the priority distribution of each control strategy in real time, achieving dynamic optimization control that is "scenario-adaptive, user-configurable, and strategy-interpretable".

[0065] In step S5, the park equipment is controlled according to the coordinated control command to achieve energy-saving control.

[0066] Preferably, a real-time control module is provided to adjust the flexible load within seconds. The flexible load refers to electrical equipment whose power consumption time and power can be flexibly adjusted within a certain range without seriously affecting user experience or production needs.

[0067] Preferably, this embodiment includes a separate flexible load module to adapt to the multi-timescale characteristics of microgrid dispatch. Specifically, day-ahead / intraday optimization is used to handle predictable, slowly changing resources (such as energy storage plans and charging pile dispatch), with a time granularity of 15 minutes to 1 hour; real-time control is used to handle unpredictable, rapidly fluctuating disturbances (such as photovoltaic cloud shading, sudden equipment shutdown, and sudden load surges), with a time granularity of seconds to minutes.

[0068] Preferably, the generated collaborative control instructions and optimization strategies are distributed through a large model decision engine, and the collaborative control instructions are sent to the corresponding park equipment or its subsystem.

[0069] This invention also provides a smart park energy-saving collaborative control system based on a multimodal large model, characterized in that it includes: A multimodal perception module is used to acquire multimodal perception data, which includes visual perception data, IoT perception data, text perception data, and voice perception data. The feature extraction module is used to extract features from the multimodal sensing data to obtain visual feature vector V_t, IoT feature vector I_t, text feature vector T_t, and audio feature vector A_t, where t represents a time point; The fusion module is used to take the text feature T_t and the audio feature A_t as result-oriented features, and convert the visual feature V_t and the IoT feature I_t into vector form through vectorization processing, and then input them into the fusion function F in the multimodal large model to generate a unified state representation vector S_t; The load forecasting module is used to input the unified state representation vector S_t into the load forecasting module in the multimodal large model to generate hourly cooling load forecasting results for future periods. The control decision module is used to input the load forecast results into the control decision module in the multimodal large model. The control decision module constructs a multi-objective optimization function based on the attention mechanism and generates at least one coordinated control instruction for park equipment. The control execution module is used to control at least one park device according to the coordinated control instructions.

[0070] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the above-mentioned smart park energy-saving collaborative control method of multimodal large model.

[0071] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the above-mentioned multimodal large-scale smart park energy-saving collaborative control method.

[0072] This invention provides a multimodal large-scale model-based smart park energy-saving collaborative control system and method. Using a multimodal large-scale model as the core perception and decision-making foundation, it constructs an intelligent dynamic energy optimization system for the park. By integrating visual, IoT, text, and audio features, a unified state representation vector for the park is generated using a fusion function, achieving unified semantic understanding of multi-source heterogeneous data. Users only need to issue energy-saving commands or provide comfort feedback through natural language interaction; the system can automatically interpret the user's intent and dynamically generate collaborative control commands based on real-time perceived park status, predicted load, and user preference configurations.

[0073] The beneficial effects of this invention include: (a) Enhanced global optimization capabilities: This invention achieves unified modeling and collaborative optimization of the entire energy system chain in a park by constructing a multimodal large-scale model fusion perception and decision-making architecture. Specifically, this invention can achieve joint optimization across more than seven subsystems, including HVAC systems, lighting systems, microgrid systems, energy storage systems, charging pile systems, water supply and drainage systems, and plug-in load systems, effectively solving the problem of strategy conflicts between subsystems caused by traditional "chimney-style" optimization, reducing strategy conflicts between subsystems by more than 50%. At the same time, through multimodal perception and load forecasting, this invention can accurately match renewable energy generation with load demand, increasing the renewable energy absorption rate by 15% to 25%, and significantly enhancing the energy utilization efficiency of the park.

[0074] (II) Innovation in Operation and Maintenance Model: This invention transforms park energy management from "human experience-driven" to "data intelligence-driven," achieving a fundamental change in the operation and maintenance model. Through multimodal data fusion and continuous learning mechanisms, the system can autonomously mine optimization potential from historical operational data, eliminating the need for manual rule configuration and parameter tuning. Regarding fault early warning, this invention, through time-series decomposition and anomaly detection, can detect equipment performance degradation and potential faults in advance, increasing the fault warning lead time from hours in traditional solutions to days, allowing maintenance personnel ample time to handle the situation. Simultaneously, the energy-saving strategies generated by this invention are verifiable, traceable, and auditable. Maintenance personnel can clearly understand the basis for the control logic's generation, trace the specific causes of abnormal energy consumption, and significantly improve the system's explainability and reliability.

[0075] (III) Social Benefits This invention provides solid technical support for parks to achieve their energy-saving goals. Through refined energy-saving regulation and efficient utilization of new energy sources, it effectively reduces the carbon emission intensity of parks. This invention forms a replicable smart energy-saving solution with good universality and promotional value, and can be widely applied to various scenarios such as commercial parks, industrial parks, and science parks. This invention promotes the industrial application of large-scale artificial intelligence models in vertical fields, providing a typical example of AI models empowering the real economy. Simultaneously, by constructing an intelligent energy management platform, this invention significantly improves the digital governance level of parks, providing strong support for smart city construction. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0077] Figure 1 This is a schematic diagram illustrating the steps of the smart park energy-saving collaborative control method based on the multimodal large model.

[0078] Figure 2 This is an example diagram of the dynamic optimization process of the central air conditioning system in Example 1.

[0079] Figure 3 This is an example diagram of the output generated by the optimization strategy in Example 1.

[0080] Figure 4 This is an example diagram of the continuous learning and optimization process of the system model in Example 2. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings: This invention provides an embodiment of a smart park energy-saving collaborative control method and system based on a multimodal large model. It should be noted that although the logical order is shown in the flowchart, under certain data conditions, the steps shown or described may be completed in a different order than that shown here.

[0082] Example 1: Refer to Figures 1-2 This invention provides a smart park energy-saving collaborative control method based on a multimodal large model, characterized in that the method includes the following steps: S1: Acquire multimodal perception data, which includes visual perception data, IoT perception data, text perception data, and voice perception data; S2: Perform feature extraction on the visual perception data, IoT perception data, text perception data and voice perception data respectively to obtain visual feature V_t, IoT feature I_t, text feature T_t and audio feature A_t, where t represents the time point; S3: The text feature T_t and the audio feature A_t are used as result-oriented features. The visual feature V_t and the IoT feature I_t are converted into vector form through vectorization and then input into the fusion function F in the multimodal large model. S4: Generate a unified state representation vector S_t through the fusion function F, and input S_t into the control decision module in the multimodal large model to generate collaborative control instructions for park equipment.

[0083] S5: Control the park equipment according to the aforementioned coordinated control instructions to achieve energy-saving control.

[0084] In step S1, the large model engine receives data input from the multi-source heterogeneous sensing module. Sensors, cameras, and the scheduling system asynchronously send real-time temperature and humidity data, personnel distribution heatmaps, meeting room reservation information, and other data to the large model engine. The sensors are used to collect real-time temperature and humidity data for each area, the cameras are used to obtain personnel distribution heatmaps through video analysis, and the scheduling system is used to provide structured scheduling data such as meeting room reservation information.

[0085] In step S2, Visual perception module: By capturing video streams through cameras in the conference room on the 3rd floor of Building A and extracting visual features, it detects that 20 people will enter the conference room in 15 minutes. This prediction is based on a combination of reservation system information and personnel trajectory prediction algorithms.

[0086] IoT sensing module: Collects time-series data from sensors, and obtains the current outdoor temperature as 32℃, the indoor temperature on the 3rd floor of Building A as 26.5℃, and the temperature in the unoccupied area on the 2nd floor of Building A as 28℃.

[0087] Text perception module: Analyzes operation and maintenance logs and schedule data to obtain information that an important meeting will be held on the 3rd floor of Building A from 14:00 to 16:00, requiring priority to be given to comfort; at the same time, it reads text information from the carbon quota management system to learn that carbon quotas are tight today.

[0088] Voice interaction module: Receives the administrator's natural language command "Electricity price is high in the afternoon, try to save electricity", and after audio feature extraction and intent recognition, analyzes the user's urgency and energy-saving preferences.

[0089] In step S3, the large model engine fuses the above multimodal data to generate a unified state representation vector for the park.

[0090] In step S4, based on the current state, the engine predicts the hourly cooling load of each area over the next two hours. The prediction results show that from 14:00 to 14:30, the cooling load on the 3rd floor of Building A surges by about 30% due to the start of a meeting; from 14:30 to 15:00, the load in other areas remains stable as people return to their workstations; at the same time, photovoltaic output decreases due to cloud cover, leading to an increase in overall electricity costs.

[0091] Based on load forecasting results and multimodal sensing information, the control decision module generates optimization strategies. These optimization strategies refer to a set of specific, executable control instructions formulated by the system to achieve energy conservation and carbon reduction goals. These strategies directly correspond to the control parameters of each subsystem of the central air conditioning system, including the supply air temperature setpoint of each air conditioning unit, fan speed, water valve opening or chilled water flow rate, equipment start-up and shutdown times and operating modes, and the opening of the zone air valves at variable air volume terminals.

[0092] In this example, the generated optimization strategy is as follows: Figure 3 As shown.

[0093] In step S5, the large model decision engine sends the generated strategy instructions (such as "Set the air conditioner temperature on the 3rd floor of Building A to 24℃") to the building automation system via the API interface, and the control system drives the air conditioning equipment to perform the corresponding adjustments.

[0094] After the environmental status is updated in real time, the sensors re-monitor the temperature and humidity data of each area and feed the updated data back to the large model engine. The engine evaluates the control effect, determines whether the expected optimization goal has been achieved, and incorporates the data from this control process into the model for continuous learning and updating, forming a complete closed loop of "perception-decision-control-evaluation-optimization". Through the above iterative mechanism, the system can continuously optimize the control strategy and achieve dynamic energy-saving control of the central air conditioning system.

[0095] Example 2: Refer to Figure 4 This embodiment uses the model's continuous learning and self-evolution mechanism as an example to provide a detailed description of the technical solution of the present invention.

[0096] In step S5, after the system completes initial deployment, it enters the online inference state. During online inference, the system cyclically executes the following tasks: receiving real-time multimodal data, generating control decisions, executing control commands, collecting feedback data, and evaluating the performance of the control effect.

[0097] After performance evaluation, the system performs a multi-dimensional comprehensive judgment on the accumulated feedback data to determine whether to trigger model retraining. Unlike traditional continuous learning systems that trigger retraining solely based on time periods or data volume, this invention employs a multi-index weighted scoring system for comprehensive judgment. Specific judgment criteria include: Data volume dimension: The absolute number of new samples Count(new_samples) is greater than the preset basic threshold N_min, where N_min is preferably set to 1000 to ensure that enough new data is accumulated for effective training.

[0098] Data quality dimension: The sample diversity index Shannon_entropy(new_data) is greater than the preset diversity threshold H_min, ensuring that new data covers new application scenarios and avoiding the waste of computing resources caused by duplicate data.

[0099] Data timeliness dimension: The concept drift detection method is adopted. The difference in distribution between old and new data is calculated by KS test. When KS_test(old_dist, new_dist) is less than the preset p_value, it is determined that the data distribution has changed significantly and retraining is triggered.

[0100] Model performance dimension: Monitor online inference error. When the current error MSE_current is greater than 1.2 times the baseline error MSE_baseline, that is, the model performance drops by more than 20%, the system will urgently trigger retraining to ensure system reliability.

[0101] Business cycle dimension: Set up a fixed time trigger mechanism, preferably a guaranteed trigger at 3:00 AM every Sunday to ensure that the model is updated regularly.

[0102] When any of the above conditions are met or the overall score reaches a preset threshold, the system triggers the retraining process.

[0103] After retraining is triggered, the system integrates historical data with newly accumulated feedback data to construct a training dataset. This dataset undergoes diversity filtering to remove duplicate data and low-quality samples, ensuring the representativeness and effectiveness of the training data.

[0104] The system employs a parameter-efficient fine-tuning method to update the model. Unlike full retraining, parameter-efficient fine-tuning only updates a portion of the model's parameters (such as the adapter layer or the low-rank adaptation matrix), maintaining the original capabilities of the large model while rapidly adapting to new data distributions, significantly reducing training computation costs and time overhead.

[0105] After parameter fine-tuning, the system deploys the updated model to the simulation environment for verification. The simulation environment simulates the actual operation of the park and multimodal data input, comprehensively evaluating the model's control effect, stability, and safety. Only after successful verification can the new model enter the hot update process.

[0106] Once verified, the system seamlessly replaces the current online inference model using a hot update method. During the hot update process, the system maintains uninterrupted online service, and the new model gradually takes over the control decision-making tasks, ensuring the continuity and reliability of the control system.

[0107] After the model is hot-updated, the system returns to the online inference state and continues to perform real-time data reception and control decision-making tasks. At the same time, it begins a new round of feedback data accumulation and performance evaluation, forming a complete closed loop of continuous learning and optimization.

[0108] Through the aforementioned continuous learning mechanism, this invention can adapt to long-term factors such as device performance degradation, changes in usage habits, and seasonal changes, thereby continuously improving model performance and significantly enhancing the robustness and adaptability of the system.

[0109] Example 3: This invention also provides a smart park energy-saving collaborative control system based on a multimodal large model, characterized in that it includes: Multimodal perception module: used to acquire multimodal perception data, including visual perception data, IoT perception data, text perception data and voice perception data; Feature extraction module: used to extract features from the multimodal sensing data to obtain visual feature vector V_t, IoT feature vector I_t, text feature vector T_t, and audio feature vector A_t, where t represents a time point; The fusion module uses the text features T_t and the audio features A_t as result-oriented features, and converts the visual features V_t and the IoT features I_t into vector form through vectorization before inputting them into the fusion function F in the multimodal large model. The fusion function F is implemented by concatenating features and mapping them through a fully connected layer, weighted fusion based on an attention mechanism, or multimodal alignment fusion, unifying multi-source heterogeneous features into the same semantic space to form a comprehensive understanding of the current state of the park. Load forecasting module: used to generate a unified state representation vector S_t from the fusion function F, input S_t into the control decision module in the multimodal large model, and generate coordinated control instructions for park equipment; Control and decision module: used to input the load forecast results into the control and decision module in the multimodal large model. The control and decision module constructs a multi-objective optimization function based on the attention mechanism and generates at least one coordinated control instruction for park equipment. Control execution module: Used to control at least one piece of equipment in the park according to the coordinated control instructions. The control execution module includes a flexible load real-time control submodule, used to adjust the flexible load at the second level to adapt to the multi-timescale characteristics of microgrid dispatch and achieve timely response to rapid fluctuations and disturbances.

[0110] This invention provides a smart park energy-saving collaborative control system based on a multimodal large model. By constructing a complete technical chain of multimodal perception, feature extraction, fusion representation, load forecasting, control decision-making, and control execution, and a fusion perception and dynamic priority ranking mechanism based on the multimodal large model, the system can automatically parse user natural language commands and multi-source heterogeneous data, reducing policy conflicts between subsystems by more than 50% and increasing the renewable energy consumption rate by 15% to 25%. Simultaneously, this invention shifts the operation and maintenance mode from "human experience-driven" to "data intelligence-driven," increasing the lead time for fault warnings from hours to days. Energy-saving strategies are verifiable, traceable, and auditable, providing solid technical support for parks to achieve energy conservation and carbon reduction goals, promoting the industrial application of artificial intelligence large models in vertical fields, and significantly improving the level of digital governance and intelligent operation and maintenance efficiency of parks.

Claims

1. A smart park energy-saving collaborative control method based on a multimodal large model, characterized in that, The method includes the following steps: S1: Acquire multimodal perception data, which includes visual perception data, IoT perception data, text perception data, and voice perception data; S2: Perform feature extraction on the visual perception data, IoT perception data, text perception data and voice perception data respectively to obtain visual feature V_t, IoT feature I_t, text feature T_t and audio feature A_t, where t represents the time point; S3: The text feature T_t and the audio feature A_t are used as result-oriented features. The visual feature V_t and the IoT feature I_t are converted into vector form through vectorization and then input into the fusion function F in the multimodal large model. S4: Generate a unified state representation vector S_t through the fusion function F, and input S_t into the control decision module in the multimodal large model to generate collaborative control instructions for park equipment; S5: Control the park equipment according to the aforementioned coordinated control instructions to achieve energy-saving control.

2. The smart park energy-saving collaborative control method based on a multimodal large model according to claim 1, characterized in that, Also includes: The unified state representation vector S_t is input into the load prediction module in the multimodal large model to generate hourly cold load prediction results for future periods.

3. The smart park energy-saving collaborative control method based on a multimodal large model according to claim 2, characterized in that, Also includes: The load forecast results are input into the control decision module to construct the multi-objective optimization function.

4. The smart park energy-saving collaborative control method based on a multimodal large model according to claim 3, characterized in that, The formula for constructing the multi-objective optimization function is as follows: min[α·E_total + β·C_carbon + γ·D_comfort]; Where E_total represents the total energy consumption index, C_carbon represents the carbon emission index, D_comfort represents the comfort deviation index, and α, β, and γ are the corresponding weighting coefficients. The weighting coefficients α range from 0.3 to 0.6, β range from 0.2 to 0.5, and γ range from 0.1 to 0.

4.

5. A smart park energy-saving collaborative control method based on a multimodal large model according to claim 4, characterized in that, The total energy consumption index E_total is calculated in the following way: E_total = Σ(Equipment power × Operating time) + Fixed base energy consumption; The carbon emission index C_carbon is calculated in the following way: C_carbon = Σ(energy consumption × corresponding carbon emission factor); The comfort deviation index D_comfort includes at least the temperature deviation, which is calculated in the following way: Temperature deviation = .

6. The method according to claim 1, characterized in that, S4 also includes: The contribution of the above factors to carbon emissions is quantified by Transformer attention weights, generating a visualized carbon flow map and a natural language interpretation report, which includes at least one optimization strategy.

7. The smart park energy-saving collaborative control method based on a multimodal large model according to claim 6, characterized in that, Also includes: The large model decision engine distributes the generated coordinated control instructions and optimization strategies, and sends the coordinated control instructions to the corresponding park equipment or its subsystems.

8. A smart park energy-saving collaborative control system based on a multimodal large model, characterized in that, include: A multimodal perception module is used to acquire multimodal perception data, which includes visual perception data, IoT perception data, text perception data, and voice perception data. The feature extraction module is used to extract features from the multimodal sensing data to obtain visual feature vector V_t, IoT feature vector I_t, text feature vector T_t, and audio feature vector A_t, where t represents a time point; The fusion module is used to take the text feature T_t and the audio feature A_t as result-oriented features, and convert the visual feature V_t and the IoT feature I_t into vector form through vectorization processing, and then input them into the fusion function F in the multimodal large model; The load forecasting module is used to generate a unified state representation vector S_t from the fusion function F, and input S_t into the control decision module in the multimodal large model to generate coordinated control instructions for park equipment. The control decision module is used to input the load forecast results into the control decision module in the multimodal large model. The control decision module constructs a multi-objective optimization function based on the attention mechanism and generates at least one coordinated control instruction for park equipment. The control execution module is used to control at least one park device according to the coordinated control instructions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the smart park energy-saving collaborative control method of the multimodal large model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the smart park energy-saving collaborative control method of the multimodal large model as described in any one of claims 1 to 7.

Images