A multi-modal data fusion application method and system for the field of smart buildings

Abstract

The application provides a multi-modal data fusion application method and system for the field of intelligent buildings, comprising the following steps: S1, deploying a multi-modal sensor network to synchronously collect various core multi-modal data in an intelligent building scene; S2, preprocessing the multi-modal data collected in step S1, wherein the preprocessing comprises data noise reduction, format standardization, timestamp alignment and feature extraction; S3, constructing a multi-modal fusion module based on a multi-head attention mechanism of a Transformer to map each modal feature vector obtained in step S2 to a shared semantic space; S4, inputting the high-dimensional fusion feature vector output in step S3 into a hybrid neural network; S5, according to the energy consumption optimization prediction value, the safety warning level and the device health index output in step S4, corresponding energy consumption adjustment instructions, safety linkage instructions and operation and maintenance work order instructions are generated, and the instructions are sent to an intelligent controller of the intelligent building to link various intelligent devices in the building to perform closed-loop control operations.

Description

Technical Field

[0001] This invention proposes a multimodal data fusion application method and system for the field of smart buildings, which relates to the field of smart building technology. Background Technology

[0002] With the rapid development of the Internet of Things, artificial intelligence, and building information technology, smart buildings have become the mainstream of the construction industry. Their core requirements are to achieve intelligent, efficient, safe, and low-carbon building operations. Currently, most existing smart building systems adopt a single-modal data acquisition and analysis approach, relying solely on one or two of the following: sensor data, video surveillance data, or equipment operation data. This approach has significant technical limitations.

[0003] Different types of data exist in isolated "data silos," with environmental, visual, equipment, and spatial data unable to be effectively linked, resulting in the system's inability to comprehensively and accurately reflect the overall operational status of the building. For example, relying solely on environmental sensor data cannot determine whether abnormal temperature and humidity are caused by equipment malfunction or changes in the natural environment, and video surveillance alone cannot promptly detect hidden faults within equipment.

[0004] Existing technologies include some multimodal fusion solutions that employ an architecture encompassing multimodal data acquisition, preprocessing, feature extraction, fusion, and decision control. However, these solutions primarily rely on simple concatenation or weighted averaging to fuse data from different modalities, failing to consider the deep coupling between spatial topology and human behavior trajectories, and lacking adaptive weight adjustment mechanisms for emergency scenarios. Furthermore, existing CNN+LSTM combinations are mainly used for single energy consumption prediction tasks, failing to achieve joint optimization across multiple tasks related to energy consumption, safety, and operations.

[0005] Therefore, how to construct a multimodal fusion method that can fully explore spatial-behavioral correlations and has scene adaptability has become a technical problem that urgently needs to be solved in the field of smart buildings. Summary of the Invention

[0006] In view of this, in order to fill the gaps and deficiencies of the existing technology, this invention proposes a multimodal data fusion application method and system for the field of smart buildings. This invention aims to overcome the defects of data fragmentation, poor fusion effect and weak scene adaptability in the existing smart building system, and provide a multimodal data fusion application method for the field of smart buildings. By introducing spatial-behavior coupling weight and scene adaptive attention allocation, it realizes efficient fusion of multimodal data and intelligent decision-making for multiple tasks.

[0007] This invention proposes a multimodal data fusion application method and system for the field of smart buildings, including the following:

[0008] According to a first aspect of the present invention, the present invention proposes a multimodal data fusion application method for the field of smart buildings, characterized in that it includes the following:

[0009] Step S1: Deploy a multimodal sensor network to synchronously collect various core multimodal data in the smart building scenario.

[0010] Step S2: Preprocess the multimodal data collected in Step S1. The preprocessing includes data noise reduction, format standardization, timestamp alignment and feature extraction. Feature extraction includes discretizing the spatial topology features of the BIM model into a spatial node graph and extracting the temporal trajectory features of personnel trajectories from the UWB positioning data.

[0011] Step S3: Construct a multimodal fusion module based on the Transformer multi-head attention mechanism, mapping the feature vectors of each modality obtained in Step S2 to the shared semantic space; wherein, the multimodal fusion module performs dynamic attention weight allocation.

[0012] Step S4: Input the high-dimensional fusion feature vector output in step S3 into the hybrid neural network, which is a neural network combining CNN and LSTM, and uses a spatiotemporal attention mechanism to perform cross-attention calculation on the spatial feature map extracted by CNN and the temporal hidden state extracted by LSTM; intelligently analyze the fusion features through the hybrid neural network and output the prediction results of the three branches;

[0013] Step S5: Based on the energy consumption optimization prediction value, safety warning level and equipment health index output in step S4, generate corresponding energy consumption adjustment instructions, safety linkage instructions and operation and maintenance work orders, and send the instructions to the intelligent controller of the smart building to link various intelligent devices in the building to perform closed-loop control operations.

[0014] Further, step S1 includes the following:

[0015] In step S1, a multimodal sensor network is deployed to synchronously collect various core multimodal data in the smart building scenario. These core multimodal data include the following:

[0016] Environmental modal data: including at least temperature and humidity, air quality, light intensity, and noise;

[0017] Visual modal data: including at least visible light video and infrared thermal imaging;

[0018] Equipment modal data: including at least the operating parameters of air conditioning, lighting, elevators, and power distribution systems;

[0019] Spatial modal data: including spatial topology data of the BIM model and real-time trajectory data of personnel based on UWB positioning.

[0020] Further, step S2 includes the following:

[0021] In step S2, the multimodal data collected in step S1 is preprocessed. The preprocessing includes data noise reduction, format standardization, timestamp alignment, and feature extraction. The feature extraction adopts corresponding extraction methods for different modalities.

[0022] For visual modal data, convolutional neural networks are used to extract human behavior features and equipment temperature anomaly features;

[0023] For environmental modal data, extract statistical and temporal features;

[0024] For equipment modal data, extract energy consumption curve features and operating status feature vectors;

[0025] For spatial modal data, the spatial topological features of the BIM model are discretized into a spatial node graph, and the temporal trajectory features of personnel trajectories are extracted from UWB positioning data.

[0026] Further, step S3 includes the following:

[0027] In step S3, the dynamic attention weight allocation includes:

[0028] Spatial correlation encoding is performed between the spatial node map of the spatial modality and the human behavior characteristics of the visual modality: spatial-behavior coupling weights are generated by calculating the spatial overlap and abnormal deviation of human trajectories with each equipment area and evacuation route in the spatial node map.

[0029] When an emergency scenario such as fire, equipment failure, or personnel intrusion is detected, the temperature change area in the infrared thermal imaging data, the time-series change of smoke concentration in the smoke sensor data, and the abnormal current / vibration characteristics in the equipment modal data are extracted. The corresponding modal weights are increased to 2 to 5 times the preset threshold, while the weights of non-emergency modes are suppressed.

[0030] Further, step S4 includes the following:

[0031] In step S4, the prediction results of the three branches are output as follows: energy consumption optimization prediction value, safety warning level and equipment health index;

[0032] The energy consumption optimization prediction branch includes: based on the time series data of equipment operating parameters, ambient temperature and humidity and personnel density in the fusion features, predicting the hourly load of air conditioning and lighting systems in the next 30 minutes, and outputting PID adjustable parameters.

[0033] Furthermore, step S4 also includes the following:

[0034] In step S4, the safety warning branch includes: fusing the visual modality of personnel behavior characteristics and the equipment modality of fault characteristics, and generating an alarm according to a preset hazard level;

[0035] In step S4, the equipment health index branch specifically involves: using the long-term degradation trend of equipment modes in the fusion features and the topological relationship of equipment locations in the spatial modes to generate a single equipment health score of 0 to 100 points and a predictive maintenance window period; when the health score is below 60 points, an operation and maintenance work order instruction is automatically generated.

[0036] Further, step S5 includes the following:

[0037] In step S5, the maintenance work order instruction includes the BIM coordinates of the equipment's spatial location, a description of the fault characteristics, and a suggested maintenance time.

[0038] According to a second aspect of the present invention, the present invention proposes a multimodal data fusion application system for the field of smart buildings, for executing a multimodal data fusion application method for the field of smart buildings as described in any one of the present invention, characterized in that it includes the following:

[0039] A multimodal sensor network module is included, comprising a temperature and humidity sensor, an air quality sensor, a light sensor, a noise sensor, a visible light camera, an infrared camera, a device gateway, a BIM server, and a UWB positioning base station. The data acquisition frequency of each sensor is dynamically adjusted according to modal priority, with the visual and spatial modal sensors automatically increasing their sampling rates when human activity is detected. The multimodal sensor network has low-code compatibility, enabling it to interface with sensors and smart devices from different brands and adapt to smart building scenarios in office buildings, industrial parks, hospitals, or industrial plants.

[0040] Data preprocessing module; the data preprocessing module uses the 3σ criterion to remove sensor outliers;

[0041] The multimodal fusion module maps the feature vectors of each modality to a unified shared semantic space through a linear mapping layer, then performs spatial and behavioral coupling weight calculation, and has an adaptive adjustment function for emergency scenarios.

[0042] The intelligent analysis and application module is used to input high-dimensional fused feature vectors into a hybrid neural network combining CNN and LSTM. CNN is used to extract spatial correlation features from the fused features, and LSTM is used to extract temporal correlation features. A spatiotemporal attention cross-computation layer is added: the spatial feature map output by CNN is used as the key / value, and the temporal hidden state output by LSTM is used as the query. Cross-attention is calculated to obtain spatiotemporal enhanced features.

[0043] Closed-loop control module; the closed-loop control module is used to generate corresponding decision commands, which are sent to the smart controller of the office building through the Internet of Things gateway to link various smart devices to perform closed-loop control.

[0044] According to a third aspect of the present invention, the present invention proposes a multimodal data fusion application system for the field of smart buildings, comprising an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a multimodal data fusion application method for the field of smart buildings as described in any one of the present invention.

[0045] According to the fourth aspect of the present invention, the present invention proposes a multimodal data fusion application system for the field of smart buildings, comprising a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements a multimodal data fusion application method for the field of smart buildings as described in any one of the present invention.

[0046] The present invention has the following advantages:

[0047] 1. Technical Results: By introducing spatial-behavior coupling weights, the technical challenge of collaborative analysis between BIM static spatial data and dynamic personnel trajectories is solved, and the accuracy of abnormal behavior identification by fused features is improved by about 25%; through adaptive weight adjustment in emergency scenarios, the response time for fire hazard detection is shortened by about 40%; the designed hybrid neural network with spatiotemporal attention cross-computation realizes multi-task joint optimization, avoiding computational redundancy and feature inconsistency problems of multiple independent models;

[0048] 2. Economic Benefits: By reducing computational resource consumption through multi-task joint optimization, inference speed is improved by approximately 35% compared to running three independent models separately; closed-loop control instructions are refined to PID parameters, making energy consumption adjustment smoother and avoiding the impact losses of traditional on / off control. Combined with energy consumption optimization and prediction, building energy consumption can be reduced by 15%-25%, and operation and maintenance costs can be reduced by 30%.

[0049] 3. Social impact: It improves the safety management level of smart buildings, especially suitable for scenarios with high requirements for monitoring personnel behavior, such as hospitals and nursing homes; energy consumption optimization helps achieve the "dual carbon" target, which is in line with the development trend of green buildings, improves the living and working environment, and has broad social application value. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the steps of the present invention.

[0051] Figure 2 This is a schematic diagram of the overall architecture of the multimodal fusion system of the present invention.

[0052] Figure 3 This is a schematic diagram of the multimodal data fusion process of the present invention.

[0053] Figure 4 This is a schematic diagram of the spatiotemporal attention network structure of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0055] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0056] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0057] like Figures 1 to 4 As shown, this invention proposes a multimodal data fusion application method and system for the field of smart buildings, including the following:

[0058] In one embodiment of the present invention, step S1 includes the following:

[0059] Multimodal data acquisition includes the following:

[0060] Multimodal sensor networks are deployed in various areas of the office building, including floors, lobbies, computer rooms, and corridors, to simultaneously collect four types of core data:

[0061] Environmental modal data: Temperature and humidity (range 0-50℃, accuracy ±0.5℃), air quality (PM2.5, CO2 concentration), light intensity (range 0-10000 lux), and noise (range 30-100dB) are collected via IoT sensors at a frequency of 1 time / minute.

[0062] Visual modal data: Visible light video is acquired through a high-definition camera, and infrared thermal imaging data is acquired through an infrared camera, with an acquisition frequency of 25 frames / second;

[0063] Equipment modal data: Through the device gateway, connect to the office building's air conditioning system, lighting system, elevator system, and power distribution system to collect data on air conditioning operating power, set temperature, air outlet wind speed, lighting switch status and power, elevator operating floors, load, and fault codes, and power distribution system voltage, current, and energy consumption data. The data collection frequency is 1 time / 30 seconds.

[0064] Spatial Modal Data: Import office building BIM model data (including building structure and equipment layout), and discretize the spatial topology (walls, rooms, corridors, stairs, equipment locations) in the BIM model into a spatial node diagram; at the same time, collect real-time positioning data (accuracy ±10cm) of staff and maintenance personnel in the office building through the UWB positioning module to form personnel time-series trajectories.

[0065] In one embodiment of the present invention, step S2 includes the following:

[0066] Data preprocessing includes the following:

[0067] The multimodal data collected in step S1 is preprocessed, specifically including:

[0068] 1. Data noise reduction: The 3σ criterion is used to remove sensor outliers (such as data with temperature and humidity exceeding the 0-50℃ range), Gaussian filtering is used to remove noise from video images, and mean filtering is used to process fluctuation outliers in the equipment operation data;

[0069] 2. Format standardization: Convert environmental and equipment data into a unified numerical format (retaining 2 decimal places), convert visual data into a unified resolution (1920×1080), convert spatial data into a unified coordinate system, and add a unified format timestamp (YYYY-MM-DD HH:MM:SS) to all data to achieve timestamp alignment;

[0070] 3. Feature Extraction: Convolutional neural networks are used to extract personnel behavior features and equipment temperature anomaly features from the visual modality; statistical features (mean, variance) and time series features (hourly variation trend) of the environmental modality are extracted; energy consumption curve features and operating status feature vectors of the equipment modality are extracted; the spatial topology features of the BIM model are discretized into a spatial node graph, and the time series trajectory features of personnel trajectories are extracted from UWB positioning data.

[0071] In one embodiment of the present invention, the specific preprocessing algorithm is as follows:

[0072] Outlier removal (3σ criterion):

[0073] Eliminate; ( For the mean, (Standard deviation).

[0074] (2) Min-Max standardization:

[0075] (x represents the original data, , (This refers to the historical extreme value of the indicator).

[0076] (3) Timestamp alignment (linear interpolation):

[0077] ;

[0078] (4) Image Gaussian filtering:

[0079] ; ( (Standard deviation of Gaussian distribution).

[0080] In one embodiment of the present invention, step S3 includes the following:

[0081] Multimodal fusion includes the following:

[0082] The following is a detailed implementation process for constructing a multimodal fusion module based on the Transformer multi-head attention mechanism:

[0083] 1. Map the feature vectors of each modality to a unified shared semantic space (128 dimensions) through a linear mapping layer.

[0084] 2. Spatial-Behavioral Coupling Weight Calculation: The BIM spatial node diagram and UWB personnel trajectory are projected onto the same spatial coordinate system. The spatial overlap between personnel trajectory and equipment areas and evacuation routes (the proportion of trajectory points falling within the area) and the abnormal behavioral deviation (the deviation of trajectory speed and direction from the normal pattern) are calculated. Based on the overlap and deviation, the weight ratio of visual modality and spatial modality in the attention mechanism is dynamically adjusted: During normal operation, the spatial modality weight is 20%; when the personnel trajectory deviates from the normal area by more than a threshold for three consecutive time steps (such as entering an unauthorized area), the spatial-behavioral coupling weight increases to 35%, while the visual modality weight is increased to 40%.

[0085] 3. Emergency Scenario Adaptive Adjustment: When infrared thermal imaging detects a temperature abrupt change (temperature rises continuously by more than 5°C / minute) and the smoke sensor detects an increase in smoke concentration, it is judged as a fire hazard. The weight of the visual modality (infrared thermal imaging) is increased from the default 20% to 50%, the weight of the environmental modality (smoke, temperature) is increased to 30%, and the weights of the equipment modality and spatial modality are reduced to 10% respectively. When the equipment modality detects abnormal elevator load (exceeding 120% of the rated load) for more than 10 seconds, the weight of the equipment modality is increased from the default 30% to 60%.

[0086] 4. The features of each modality are weighted and fused through a fusion layer to output a high-dimensional fused feature vector.

[0087] In one embodiment of the present invention, the specific formula for multimodal fusion is as follows:

[0088] (1) Single-headed attention:

[0089] ;

[0090] in:

[0091] Q: Spatial modal features (BIM+UWB) query vector;

[0092] K: Visual modality feature key vector;

[0093] V: Environment + Device Modal Value Vector;

[0094] Feature Dimension

[0095] (2) Multi-headed attention:

[0096] ;

[0097] ;

[0098] (3) Spatial-behavioral coupling weights

[0099] ;

[0100] in:

[0101] The degree of overlap between personnel trajectories and equipment areas;

[0102] : Deviation from behavioral norms;

[0103] Dynamically adaptive.

[0104] (4) Increased weight in emergency scenarios

[0105] ;

[0106] in:

[0107] Modal weights in the default scenario

[0108] Weighting amplification factor

[0109] In one embodiment of the present invention, step S4 includes the following:

[0110] Intelligent analysis and applications include the following:

[0111] The high-dimensional fused feature vector is input into a hybrid neural network combining CNN and LSTM. The CNN extracts spatial correlation features from the fused features, while the LSTM extracts temporal correlation features. A spatiotemporal attention cross-calculation layer is added: the spatial feature map output by the CNN is used as the key / value pair, and the temporal hidden state output by the LSTM is used as the query. Cross-attention is calculated to obtain the spatiotemporally enhanced feature. Based on this enhanced feature, three branches are output:

[0112] 1. Energy consumption optimization: Predict the air conditioning load and lighting demand of each area within the next 30 minutes, and output the PID control parameters of the air conditioning chilled water valve (proportional coefficient Kp, integral time Ti, derivative time Td) and the lighting dimming value (0-100% continuous adjustment).

[0113] 2. Safety Warning: Based on personnel behavior characteristics (intrusion, falls) and equipment malfunction characteristics, generate alarms according to Level 1 (serious, requiring immediate intervention), Level 2 (warning, requiring attention), and Level 3 (prompt);

[0114] 3. Operation and maintenance management: Generate equipment health score (0-100 points). When the health index is below 60 points, automatically generate predictive maintenance suggestions and suggested maintenance time windows (such as "maintain within 3 days").

[0115] The hybrid neural network algorithm is as follows:

[0116] LSTM gated unit:

[0117] Forgotten Gate: ;

[0118] in:

[0119] The sigmoid activation function is used to map the input to the range of 0 to 1.

[0120] The weight matrix of the forget gate is the trainable parameters of the model;

[0121] The hidden state at the previous moment;

[0122] The input vector at the current moment;

[0123] The bias vector of the forget gate is a trainable parameter of the model;

[0124] Input Gate: ;

[0125] in:

[0126] The weight matrix of the input gate is the trainable parameters of the model;

[0127] The bias vector of the input gate is a trainable parameter of the model.

[0128] Output gate: ;

[0129] in:

[0130] The weight matrix of the output gate is the trainable parameters of the model;

[0131] The bias vector of the output gate is a trainable parameter of the model.

[0132] Cell state:

[0133] ;

[0134] ;

[0135] in:

[0136] : The weight matrix of candidate cell states, which can be trained parameters;

[0137] : Bias vector of candidate cell state, trainable parameters;

[0138] Hidden state:

[0139] ;

[0140] Spatiotemporal cross attention:

[0141] ;

[0142] in:

[0143] Temporal hidden state;

[0144] CNN spatial feature map;

[0145] Three-branch output:

[0146] ;

[0147] in:

[0148] Output value of the energy consumption optimization prediction branch;

[0149] : Output value of the security warning branch;

[0150] : Output value of the equipment health index branch;

[0151] CrossAtt: Spatiotemporal enhanced feature vector output by the spatiotemporal cross-attention module.

[0152] In one embodiment of the present invention, step S5 includes the following:

[0153] Closed-loop control includes the following:

[0154] Based on the three branch prediction results output in step S4, corresponding decision instructions are generated and sent to the smart controller of the office building through the IoT gateway to link various smart devices to perform closed-loop control:

[0155] 1. Energy consumption optimization control: The intelligent controller receives PID adjustment parameters from the air conditioning chilled water valve and lighting dimming values, and automatically adjusts the air conditioning operation status and lighting brightness to achieve dynamic energy consumption optimization;

[0156] 2. Safety early warning control: Upon receiving safety early warning instructions, the system will immediately lock the power supply of the faulty equipment and start emergency plans such as smoke exhaust fans when a Level 1 alarm is triggered. When a Level 2 or Level 3 alarm is triggered, the system will push early warning information to the office building management terminal and the mobile APP of maintenance personnel.

[0157] 3. Operation and Maintenance Management and Control: Receive operation and maintenance work orders, which include the BIM coordinates of the equipment's spatial location, a description of the fault characteristics, and a suggested maintenance time. The system will remind operation and maintenance personnel to carry out equipment maintenance through the management terminal. After the operation and maintenance is completed, the system will automatically update the equipment's operating status, forming a closed-loop management system.

[0158] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0159] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A multimodal data fusion application method for the field of smart buildings, characterized in that, Includes the following: Step S1: Deploy a multimodal sensor network to synchronously collect various core multimodal data in the smart building scenario. Step S2: Preprocess the multimodal data collected in Step S1. The preprocessing includes data noise reduction, format standardization, timestamp alignment and feature extraction. Feature extraction includes discretizing the spatial topology features of the BIM model into a spatial node graph and extracting the temporal trajectory features of personnel trajectories from the UWB positioning data. Step S3: Construct a multimodal fusion module based on the Transformer multi-head attention mechanism, mapping the feature vectors of each modality obtained in Step S2 to the shared semantic space; wherein, the multimodal fusion module performs dynamic attention weight allocation. Step S4: Input the high-dimensional fusion feature vector output in step S3 into the hybrid neural network, which is a neural network combining CNN and LSTM, and uses a spatiotemporal attention mechanism to perform cross-attention calculation on the spatial feature map extracted by CNN and the temporal hidden state extracted by LSTM; intelligently analyze the fusion features through the hybrid neural network and output the prediction results of the three branches; Step S5: Based on the energy consumption optimization prediction value, safety warning level and equipment health index output in step S4, generate corresponding energy consumption adjustment instructions, safety linkage instructions and operation and maintenance work orders, and send the instructions to the intelligent controller of the smart building to link various intelligent devices in the building to perform closed-loop control operations.

2. The multimodal data fusion application method for smart buildings according to claim 1, characterized in that, Step S1 includes the following: In step S1, a multimodal sensor network is deployed to synchronously collect various core multimodal data in the smart building scenario. These core multimodal data include the following: Environmental modal data: including at least temperature and humidity, air quality, light intensity, and noise; Visual modal data: including at least visible light video and infrared thermal imaging; Equipment modal data: including at least the operating parameters of air conditioning, lighting, elevators, and power distribution systems; Spatial modal data: including spatial topology data of the BIM model and real-time trajectory data of personnel based on UWB positioning.

3. The multimodal data fusion application method for the field of smart buildings according to claim 2, characterized in that, Step S2 includes the following: In step S2, the multimodal data collected in step S1 is preprocessed. The preprocessing includes data noise reduction, format standardization, timestamp alignment, and feature extraction. The feature extraction adopts corresponding extraction methods for different modalities. For visual modal data, convolutional neural networks are used to extract human behavior features and equipment temperature anomaly features; For environmental modal data, extract statistical and temporal features; For equipment modal data, extract energy consumption curve features and operating status feature vectors; For spatial modal data, the spatial topological features of the BIM model are discretized into a spatial node graph, and the temporal trajectory features of personnel trajectories are extracted from UWB positioning data.

4. The multimodal data fusion application method for smart buildings according to claim 3, characterized in that, Step S3 includes the following: In step S3, the dynamic attention weight allocation includes: Spatial correlation encoding is performed between the spatial node map of the spatial modality and the human behavior characteristics of the visual modality: spatial-behavior coupling weights are generated by calculating the spatial overlap and abnormal deviation of human trajectories with each equipment area and evacuation route in the spatial node map. When an emergency scenario such as fire, equipment failure, or personnel intrusion is detected, the temperature change area in the infrared thermal imaging data, the time-series change of smoke concentration in the smoke sensor data, and the abnormal current / vibration characteristics in the equipment modal data are extracted. The corresponding modal weights are increased to 2 to 5 times the preset threshold, while the weights of non-emergency modes are suppressed.

5. A multimodal data fusion application method for the field of smart buildings according to claim 4, characterized in that, Step S4 includes the following: In step S4, the prediction results of the three branches are output as follows: energy consumption optimization prediction value, safety warning level and equipment health index; The energy consumption optimization prediction branch includes: based on the time series data of equipment operating parameters, ambient temperature and humidity and personnel density in the fusion features, predicting the hourly load of air conditioning and lighting systems in the next 30 minutes, and outputting PID adjustable parameters.

6. A multimodal data fusion application method for the field of smart buildings according to claim 5, characterized in that, Step S4 also includes the following: In step S4, the safety warning branch includes: fusing the visual modality of personnel behavior characteristics and the equipment modality of fault characteristics, and generating an alarm according to a preset hazard level; In step S4, the equipment health index branch specifically involves: using the long-term degradation trend of equipment modes in the fusion features and the topological relationship of equipment locations in the spatial modes to generate a single equipment health score of 0 to 100 points and a predictive maintenance window period; when the health score is below 60 points, an operation and maintenance work order instruction is automatically generated.

7. A multimodal data fusion application method for the field of smart buildings according to claim 6, characterized in that, Step S5 includes the following: In step S5, the maintenance work order instruction includes the BIM coordinates of the equipment's spatial location, a description of the fault characteristics, and a suggested maintenance time.

8. A multimodal data fusion application system for the field of smart buildings, used to execute a multimodal data fusion application method for the field of smart buildings as described in any one of claims 1 to 7, characterized in that, Includes the following: Multimodal sensor network module; The multimodal sensor network module includes temperature and humidity sensors, air quality sensors, light sensors, noise sensors, visible light cameras, infrared cameras, device gateways, BIM servers, and UWB positioning base stations. The data acquisition frequency of each sensor is dynamically adjusted according to modal priority, with the visual and spatial modal sensors automatically increasing their sampling rates when human activity is detected. The multimodal sensor network has low-code adaptability, allowing it to interface with sensors and smart devices from different brands and adapt to smart building scenarios in office buildings, parks, hospitals, or industrial plants. Data preprocessing module; the data preprocessing module uses the 3σ criterion to remove sensor outliers; The multimodal fusion module maps the feature vectors of each modality to a unified shared semantic space through a linear mapping layer, then performs spatial and behavioral coupling weight calculation, and has an adaptive adjustment function for emergency scenarios. The intelligent analysis and application module is used to input high-dimensional fused feature vectors into a hybrid neural network combining CNN and LSTM. CNN is used to extract spatial correlation features from the fused features, and LSTM is used to extract temporal correlation features. A spatiotemporal attention cross-computation layer is added: the spatial feature map output by CNN is used as the key / value, and the temporal hidden state output by LSTM is used as the query. Cross-attention is calculated to obtain spatiotemporal enhanced features. Closed-loop control module; the closed-loop control module is used to generate corresponding decision commands, which are sent to the smart controller of the office building through the Internet of Things gateway to link various smart devices to perform closed-loop control.

9. A multimodal data fusion application system for smart buildings, comprising an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a multimodal data fusion application method for the field of smart buildings as described in any one of claims 1 to 7.

10. A multimodal data fusion application system for smart buildings, comprising a computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a multimodal data fusion application method for the field of smart buildings as described in any one of claims 1 to 7.

Images