Conference room space occupation detection method based on multi-mode adaptive fusion

Abstract

The invention discloses a conference room space occupation detection method based on multi-modal adaptive fusion. The method comprises the following steps: carrying out unsupervised abnormal value processing on a conference room multi-modal data set based on a variational auto-encoder; dividing a data set according to modals, and respectively training occupation detection sub-models based on time sequence deep learning in combination with a sliding window; redundant features are removed through a Spearman correlation coefficient, and an optimal feature subset of each modal is screened based on the minimum feature number of GBDT; and constructing a fusion mechanism based on the BLS, splicing the sub-model output probability and the screened feature vectors, inputting the spliced sub-model output probability and the screened feature vectors into the BLS, realizing multi-modal deep fusion by using feature nodes and enhanced nodes, and outputting a final occupancy judgment result. Experiments show that the F1 score of the fusion model reaches 0.9842 and is obviously superior to that of a single-mode model, the detection precision and robustness are effectively improved, and reliable support can be provided for energy saving control and space dynamic allocation of building space.

Description

Technical Field

[0001] This invention belongs to the field of building space occupancy detection, specifically involving a method for detecting conference room space occupancy based on multimodal adaptive fusion. Background Technology

[0002] In modern office environments, meeting rooms are core spaces supporting corporate collaboration, decision-making, and business operations. They are used frequently and consume significant resources, directly impacting organizational efficiency. However, due to a lack of accurate awareness of the actual occupancy status of meeting rooms, companies face a dual dilemma in space management and energy usage.

[0003] On the one hand, in terms of space resource allocation, enterprises generally face a structural contradiction: "rooms are hard to find during peak hours, while many are vacant during off-peak hours." Blindly expanding conference rooms will lead to severe resource idleness during off-peak hours, driving up construction and maintenance costs. Relying on existing simple reservation systems makes it difficult to determine whether booked meetings have ended early, failing to promptly free up space for others, resulting in resource misallocation and low utilization efficiency. On the other hand, in terms of energy management, conference room usage is significantly intermittent and uncertain. After a meeting ends or people temporarily leave, lighting, air conditioning, ventilation, and multimedia equipment often continue to run idly because the vacancy status cannot be detected in time, causing a large amount of unnecessary energy waste, which contradicts the current goals of advocating energy conservation, high efficiency, and intelligent operation.

[0004] To address the aforementioned issues, achieving accurate, real-time, and non-invasive detection of meeting room occupancy has become a key breakthrough. Traditional occupancy detection methods largely rely on invasive sensing techniques such as cameras and infrared thermal imaging. While these methods can directly obtain information about the presence of people, in the highly sensitive business setting of a meeting room, such data collection methods involving visual or biometric features are prone to privacy leaks, facing challenges in user acceptance and practical implementation. In recent years, occupancy detection based on non-invasive sensor data has gradually progressed. However, most studies generally adopt empirical strategies, selecting only a few environmental features considered "strongly correlated" (such as CO2 concentration and temperature) for modeling. This ignores the fact that sensor data in actual deployments contains a large amount of redundant or noisy features unrelated to occupancy detection, and also ignores other modal data outside the environment (such as equipment operating status), failing to effectively identify and utilize the full range of heterogeneous data. Even the few studies that attempt to introduce multimodal data often employ simple splicing or static weighted fusion strategies, failing to model multimodal data at a deeper level, unable to effectively integrate and complement each other, easily leading to information redundancy or modal information conflicts, which weakens model performance. Summary of the Invention

[0005] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies and provide a meeting room space occupancy detection method based on multimodal adaptive fusion. First, multimodal data consisting of meeting room environmental sensors and equipment operation records is collected. The raw data is then cleaned using an unsupervised anomaly detection mechanism based on a variational autoencoder (VAE) to improve data reliability. Next, time-series inputs are constructed based on sliding windows for both environmental and equipment modalities. Occupancy detection models are trained using two mainstream time-series deep learning models: TCN and LSTM, with the optimal model selected based on the F1 score. Furthermore, an XGBoost model is introduced, combined with SHAP values, to evaluate the importance of high-dimensional time-series features. Redundant features are removed based on SHAP values, and a forward iterative filtering strategy is used to determine the optimal feature set for each modality. Finally, the output probabilities of each modality's occupancy detection model and the filtered features are used as inputs to a multilayer perceptron (MLP). End-to-end training achieves deep fusion of multimodal information, outputting a highly robust final occupancy status determination result.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A meeting room space occupancy detection method based on multimodal adaptive fusion is characterized by the following steps:

[0008] S1. For all modal data that can be obtained in the conference room, an unsupervised outlier detection method based on variational autoencoder is used to identify and remove outliers from the sensor data.

[0009] S2. Based on the differences in the modalities of the data sources, the preprocessed data is divided into datasets according to different modalities, and the space occupancy detection capability is independently trained on each modal dataset based on a deep learning model to obtain the occupancy detection model of the relevant modality.

[0010] S3. Conduct importance assessment and screening for each modal feature:

[0011] A minimum number of features selection strategy based on gradient boosting decision trees is adopted. The contribution of each feature to the occupancy detection task is quantified by the SHAP value, and the temporal features are aggregated by a sliding window to calculate the feature importance ranking. On this basis, Spearman correlation coefficient is used to find features carrying similar information and redundant features with SHAP values ​​below a preset threshold are removed. Finally, feature subsets are constructed in ascending order of feature contribution, and the occupancy detection model is trained iteratively. The iteration is terminated when the model performance declines, thereby determining the optimal feature set.

[0012] S4. Construct an adaptive modality fusion mechanism based on a width learning system:

[0013] The output probabilities of each modal occupancy detection model trained in step S2, together with their respective sliding window temporal features after feature filtering, are concatenated into a fusion input vector. The width learning system model adaptively learns the joint contribution of each component in the multimodal input to the final judgment, and achieves deep fusion of multimodal information through the collaborative representation ability of its feature nodes and enhancement nodes. Finally, the meeting room occupancy status detection result is generated.

[0014] As a preferred technical solution of the present invention, step S1 includes the following steps:

[0015] S11. The multimodal data collected in the current conference room can be divided into environmental modal datasets. With device modal dataset and environmental modality dataset With device modal dataset An unsupervised outlier detection method based on variational autoencoder is used to identify and clean outliers in the data;

[0016] In the outlier identification task, a deep generative model is used to model the potential distribution of normal data, and outlier samples that deviate from the normal pattern are identified by reconstructing errors. The specific steps are as follows:

[0017] S111, respectively, for the environmental modality dataset With device modal dataset Construct an independent variational autoencoder network, in which,

[0018] The encoder will input data vector Mapped to the mean in a low-dimensional latent space With variance And sample the latent variables from them. ;

[0019] The decoder is based on Reconstruct the original input The model is optimized with the joint objective of maximizing the reconstruction likelihood and minimizing the KL divergence. The objective function is defined by the following formula:

[0020] (1);

[0021] in, This is the input data vector for a certain mode; This represents sensor data in a given conference room. Under the condition of latent variables The approximate posterior distribution; Representing approximate posterior with prior The distance between them is used to avoid large deviations in the distribution of latent variables in the encoder output; It is the prior distribution of the latent variable z, usually set as a standard Gaussian distribution. ; Indicates that given a latent variable Under these conditions, generate observation data The probability of; Used to measure the current decoder from Reconstructed With reality The degree of fit, i.e., the reconstruction error, and the expected value. The larger the value, the more accurate the reconstruction.

[0022] S112. Use unlabeled data collected in the conference room to train the variational autoencoder model so that it learns the data distribution characteristics of the environment and equipment modes when it is in a "normal state". After training, the model can reconstruct data that conforms to the normal mode, while generating reconstruction bias for abnormal data caused by the sensing and recording process.

[0023] S113. For the current multimodal data sample to be detected Input the variational autoencoder model of the corresponding mode and calculate its reconstruction error. To account for differences in dimensions, the reconstruction error was normalized to obtain a standardized reconstruction score. ;

[0024] S114. Set dynamic anomaly threshold If the standardized reconstruction score of a certain sample If a value is found to be outlier, it is considered an outlier; samples deemed outliers are removed directly.

[0025] As a preferred technical solution of the present invention, step S2 includes the following steps:

[0026] S21. After completing the outlier cleaning of the multimodal data, based on the environmental modality dataset... With device modal dataset A time-aware initial occupancy state detection model is constructed, and a sliding window strategy is used to construct the input sequence from the preprocessed data, as follows:

[0027] Set window length to Each window generates one input sample. or ,in and These are the environmental feature dimension and the equipment feature dimension, respectively, and the corresponding target labels. This indicates whether the meeting room is occupied at the current time step, where 1 indicates occupied and 0 indicates idle;

[0028] S22. Two temporal deep learning architectures are selected: a temporal convolutional network and a long short-term memory network. End-to-end training is performed on the sliding window data constructed above. All models are defined with the objective of binary classification of meeting room occupancy status. The output layer uses the sigmoid activation function to generate occupancy probabilities. ;

[0029] S23. The model training uses binary cross-entropy as the loss function, and its mathematical expression is:

[0030] (2);

[0031] in, The total number of training samples, For real labels, The occupancy probability predicted by the model;

[0032] S24. On the test set, the F1 score is used as the evaluation metric, and the accuracy (ACC) is used as a secondary evaluation metric, defined by the following formula:

[0033] (3);

[0034] (4);

[0035] (5);

[0036] (6);

[0037] Wherein, TP refers to a sample that is actually a positive class and is correctly predicted as a positive class, TN refers to a sample that is actually a negative class and is correctly predicted as a negative class, FP refers to a sample that is actually a negative class but is incorrectly predicted as a positive class, and FN refers to a sample that is actually a positive class but is incorrectly predicted as a negative class.

[0038] The trained models are evaluated on a test set to obtain the F1 score and accuracy (ACC) score of each model trained on environmental and device data. These scores serve as performance benchmarks for subsequent feature selection steps to verify the effectiveness of the method.

[0039] As a preferred technical solution of the present invention, step S3 includes the following steps:

[0040] S31. Minimum number feature selection based on gradient boosting tree is carried out for environmental modal features and equipment modal features respectively, and XGBoost is used as the basic model;

[0041] S32. Based on the input samples constructed using the sliding window, the original time-series features within each time window are aggregated, as follows:

[0042] For a window length of The sequence is used to calculate its mean along the feature dimension as a static representation of the window, thereby representing the time series input. Convert to aggregated feature vector Similarly, the aggregation vector of device modes is obtained. ;

[0043] S33, respectively in the environmental modality dataset With device modal dataset The XGBoost classifier is trained on the model, and the SHAP value is calculated based on the aggregated vector of each feature in each modality to measure the average contribution of each feature to the model's detection.

[0044] For environmental modalities, the original environmental dimension is: The feature set is denoted as ,in Indicates the first Each environmental characteristic; calculate each characteristic. The mean absolute value of SHAP over all samples is denoted as:

[0045] (7);

[0046] For device modalities, the original device dimension is: The feature set is denoted as ,in Indicates the first Each device feature; calculate each feature The mean absolute value of SHAP over all samples is denoted as:

[0047] (8);

[0048] S34. Mean of absolute values ​​of SHAP based on all features within each modality and The features are sorted in descending order of importance to obtain a ranking list of environmental features by importance. And device feature importance ranking list ;

[0049] S35. By calculating the Spearman correlation coefficient Find features that carry similar information, when the calculated values ​​of two features are similar. If the value exceeds the set threshold, it is considered to carry similar information, and the list is sorted according to the importance of the features. or Remove redundant features from the SHAP values ​​that are ranked below a preset threshold, and retrain XGBoost to update the feature importance ranking list. or ;

[0050] S36. Construct the optimal feature subset for each modality using a stepwise forward selection strategy, as follows:

[0051] S361, Initializing the feature subset is defined as follows: , ;

[0052] S362. For environmental modes, in the... Wheel The Middle Features Add to the current subset to obtain the updated environmental feature subset. Similarly, we can obtain ;

[0053] S363. After each feature is added, the environment-aware occupancy detection model and the device behavior occupancy detection model are retrained using the updated feature subset, and their performance is evaluated by calculating the F1 score on an independent test set. This score is denoted as... and ;

[0054] S364, If in a certain round The addition of new environmental features caused the F1 score to drop compared to the previous round. If the iterative selection process of environmental modal features is terminated, the feature set from the previous round is taken as the optimal environmental feature set, that is... Similarly, if Then let ;

[0055] This feature selection strategy ensures that the selected feature set maintains optimal performance while utilizing the minimum number of features, based on... and The retrained model is defined as the optimal environment-aware occupancy detection model. and optimal device behavior occupancy detection model .

[0056] As a preferred technical solution of the present invention, step S4 includes the following steps:

[0057] S41. Design an adaptive fusion mechanism based on a width learning system. This mechanism dynamically integrates the prediction confidence and original temporal features of each modality occupancy detection model by constructing a joint representation space of feature node layer and enhancement node layer, and outputs the final meeting room occupancy status discrimination result.

[0058] S42. Let the current time step be... The length of the sliding window is After feature filtering in step S3, only the dimensions corresponding to the optimal feature subset are retained, and the original input of the environmental modality is... The original input of the device modality is Flatten the above two-dimensional time series tensor into a one-dimensional vector:

[0059] (9);

[0060] Environmental perception occupancy detection model The output occupancy probability is Device behavior occupancy detection model The output occupancy probability is ;

[0061] The above vectors are concatenated to form the complete input feature vector for the fusion module of the width learning system:

[0062] (10);

[0063] Among them, the input feature dimension ;

[0064] S43, Input matrix As the actual input sample for the width learning system, its structure includes a feature node layer. With enhanced node layer :

[0065] Feature node layer: for input Perform a linear transformation to map to the feature plane and generate a feature node matrix:

[0066] (11);

[0067] in, This represents the optimal input weight matrix obtained using the sparse autoencoder method. For bias terms, The number of feature nodes. It is a non-linear activation function;

[0068] Enhanced Node Layer: Introducing a random weight matrix on top of the feature nodes. and bias Generate enhanced nodes:

[0069] (12);

[0070] in, It is a non-linear activation function. To increase the number of nodes;

[0071] Merging the input layer: Feature nodes and augmentation nodes are concatenated to form the actual input signal of the width learning system.

[0072] (13);

[0073] S44. The output layer of the width learning system directly solves for the connection weight matrix through the ridge regression generalized inverse. In the occupation detection task It is calculated using the following ridge regression approximation formula:

[0074] (14);

[0075] in, The input matrix is ​​the combined input matrix of all samples in the training set, where N is the total number of samples in the training set. For the actual label vector, For regularization parameters,

[0076] At this point, the output prediction probability is:

[0077] (15);

[0078] like If the condition is met, output 1 if the condition is "occupied"; otherwise, output 0 if the condition is "unoccupied".

[0079] S45. The model is trained end-to-end through supervised learning to accurately occupy labels. As a supervisory signal, the optimization objective is to minimize the binary cross-entropy loss:

[0080] (16);

[0081] And monitor the F1 score on the validation set to implement an early stop strategy;

[0082] S46. After training, the width learning system model serves as the final multimodal meeting room occupancy discriminator. It integrates the confidence of each modal occupancy detection model with the original temporal features within the complete sliding window to output the occupancy detection results.

[0083] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0084] While ensuring user privacy, the system fully leverages the complementarity of non-intrusive environmental and device data to effectively address challenges such as complex occupancy patterns, heterogeneous sensors, and feature redundancy in conference room scenarios. This significantly improves the accuracy and generalization ability of occupancy detection, providing reliable technical support for energy-saving control and dynamic scheduling of space resources in smart buildings. Attached Figure Description

[0085] Figure 1 This is a flowchart of the meeting room space occupancy detection method based on multimodal adaptive fusion in this invention.

[0086] Figure 2 This is the Spearman correlation coefficient matrix diagram of environmental modal features in this invention.

[0087] Figure 3 This is the Spearman correlation coefficient matrix diagram of the device modal characteristics in this invention.

[0088] Figure 4 This is a bar chart ranking the importance of environmental modal features in this invention.

[0089] Figure 5 This is a bar chart ranking the importance of device modal features in this invention. Detailed Implementation

[0090] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0091] The flowchart of the meeting room space occupancy detection method based on multimodal adaptive fusion proposed in this invention is as follows. Figure 1 As shown, the method includes the following steps:

[0092] Step S1: For all modal data obtainable in the conference room of a real building, an unsupervised outlier detection method based on variational autoencoder is used to identify and remove outliers in the sensor data to improve the reliability of the input data of the subsequent occupancy detection model. The data used in this invention is the operational data collected from the Yangtze River Urban Smart Headquarters office building to verify the effectiveness of the proposed method. The collected data is divided into two modalities: environmental modality and equipment modality. The information contained is shown in Table 1. Table 1 is the definition and description of variables in the conference room multimodal dataset.

[0093]

[0094] Table 1

[0095] The data collection interval is 5 minutes, and the data collection time for each modality is kept constant. A continuous time period of data is extracted based on the average changes in each quarter. The total number of samples is 8208.

[0096] Step S11: Divide the multimodal data collected in the current meeting room into environmental modal datasets. With device modal dataset To address potential issues such as sensor noise, device malfunctions, or communication packet loss in the data, an unsupervised outlier detection method based on Variational Autoencoder (VAE) is employed for data cleaning. The core idea of ​​VAE is to model the data generation process as a probabilistic model and learn the data representation by maximizing the data's probability. In the outlier identification task, this method utilizes a deep generative model to model the potential distribution of normal data and identifies anomalous samples deviating from normal patterns by reconstructing errors. The specific steps are as follows:

[0097] Step S111: Process the environmental modality datasets respectively. With device modal dataset Construct an independent variational autoencoder network; where the encoder takes the input data vector as input. Mapped to the mean in a low-dimensional latent space With variance And sample the latent variables from them. The decoder is based on Reconstruct the original input The model is optimized with the joint objective of maximizing the reconstruction likelihood and minimizing the KL divergence. The objective function is defined by the following formula:

[0098] (1);

[0099] in, This is the input data vector for a certain mode; This represents sensor data in a given conference room. Under the condition of latent variables The approximate posterior distribution, Representing approximate posterior with prior The distance between them is used to avoid large deviations in the distribution of latent variables in the encoder output; It is the prior distribution of the latent variable z, usually set as a standard Gaussian distribution. ; Indicates that given a latent variable Under these conditions, generate observation data The probability of; Used to measure the current decoder from Reconstructed With reality The degree of fit, i.e., the reconstruction error, and the expected value. The larger the value, the more accurate the reconstruction.

[0100] Step S112: Use unlabeled data collected in the conference room to train the VAE model so that it learns the data distribution characteristics of the environment and equipment modes in the "normal state". After training, the model can reconstruct data that conforms to the normal mode with low error, while producing significant reconstruction deviation for abnormal data caused by the sensing and recording process.

[0101] Step S113: For the current multimodal data sample to be detected Input the VAE model corresponding to the mode and calculate its reconstruction error. To account for differences in dimensions, the reconstruction error was normalized to obtain a standardized reconstruction score. ;

[0102] Step S114: If the standardized reconstruction score of a certain sample... If a value is found to be an anomaly, it is considered an outlier. Samples deemed anomalies are directly removed to ensure the reliability and consistency of the input data for subsequent anomaly detection models. In practice, a dynamic anomaly threshold is set. At the 99th percentile, a total of 82 original data samples were treated as outliers, and the outliers were processed by averaging the adjacent timestamps.

[0103] Step S2: Based on the differences in data source modalities, the preprocessed data is divided into datasets according to environment and device modalities. Space occupancy detection capabilities are then trained separately on the environment and device modal datasets using deep learning models, resulting in initial occupancy detection models for the relevant modalities—the environment-aware occupancy detection model and the device behavior occupancy detection model. The specific steps are as follows:

[0104] Step S21: After completing the outlier cleaning of the multimodal data, based on the environmental modality dataset... With device modal dataset A time-aware occupancy detection model is constructed. Considering the significant time dependency between the meeting room occupancy status and the historical behavior of sensors, a sliding window strategy is used to construct the input sequence from the preprocessed data: the window length is set to... Each window generates one input sample. or ,in and These are the environmental feature dimension and the equipment feature dimension, respectively, and the corresponding target labels. This indicates whether the meeting room is occupied at the current time step (1 for occupied, 0 for idle); in practice, the window length is set to... The window spans 15 minutes.

[0105] Step S22: Two mainstream temporal deep learning architectures were selected—Temporal Convolutional Network (TCN) and Long Short-Term Memory Network (LSTM)—and trained end-to-end on the sliding window data constructed above. All models were defined to target the binary classification task of meeting room occupancy status, and the output layer used the Sigmoid activation function to generate occupancy probabilities. ;

[0106] Step S23: The model training uses binary cross-entropy (BCE) as the loss function, and its mathematical expression is:

[0107] (2);

[0108] in, The total number of training samples, For real labels, The occupancy probability predicted by the model;

[0109] Step S24: On the test set, use the F1 score as the evaluation metric and accuracy (ACC) as a secondary evaluation metric. The ACC is defined by the following formula:

[0110] (3);

[0111] (4);

[0112] (5);

[0113] (6);

[0114] Wherein, TP refers to a sample that is actually a positive class and is correctly predicted as a positive class, TN refers to a sample that is actually a negative class and is correctly predicted as a negative class, FP refers to a sample that is actually a negative class but is incorrectly predicted as a positive class, and FN refers to a sample that is actually a positive class but is incorrectly predicted as a negative class. The trained model is evaluated on the test set to obtain the F1 score and ACC score of each model under environmental data and device data training, which serve as the performance benchmark for subsequent feature selection steps to verify the effectiveness of the method.

[0115] This invention sets an early stopping strategy for the training process. The occupancy detection capability is trained on environmental data and device data based on LSTM model and TCN model respectively. The performance of the initial occupancy detection model is shown in Table 2. Table 2 is a performance list of environmental perception occupancy detection model and device behavior occupancy detection model.

[0116]

[0117] Table 2

[0118] Step S3: Evaluation and selection of the importance of environmental and device modal features: A minimum-quantity feature selection strategy based on Gradient Boosting Decision Tree (GBDT) is adopted. The SHAP value is used to quantify the contribution of each feature to the occupancy detection task, and a sliding window is used to aggregate temporal features and calculate the feature importance ranking. Based on this, Spearman correlation coefficients are used to find features carrying similar information, and redundant features with low SHAP are removed. Finally, feature subsets are constructed in ascending order of feature contribution, and the occupancy detection model is iteratively trained. The iteration is terminated when the model performance declines, thus determining the optimal feature set. The specific steps are as follows:

[0119] Step S31: Perform minimum feature selection based on Gradient Boosting Decision Tree (GBDT) for both environmental modal features and equipment modal features. Specifically, XGBoost is used as the base model because of its excellent performance in classification tasks and built-in regularization mechanism, making it suitable for robust modeling of high-dimensional temporal features. However, XGBoost always selects the direction with the largest gain as the splitting feature. In building sensor data, there are often multiple highly correlated data points (such as fan speed and energy consumption). The model may only select one to obtain a large gain, and other related features may be ignored for deduplication in the tree, resulting in a lower score in the final calculation.

[0120] Step S32: Based on the input samples constructed using the sliding window, aggregate the original temporal features within each time window: for windows with a length of... The sequence is used to calculate its mean along the feature dimension as a static representation of the window, thereby representing the time series input. Convert to aggregated feature vector Similarly, the aggregation vector of device modes is obtained. This aggregation operation reasonably preserves the overall trend information within the window while significantly reducing input complexity.

[0121] Step S33: Train the XGBoost classifier on the environment modality dataset and the device modality dataset respectively, and use XGBoost to calculate the SHAP (SHapley Additive ex Planations) value based on the aggregate vector of each feature in each modality to measure the average contribution of each feature to the model's occupancy detection.

[0122] For environmental modalities, the original environmental dimension is: The feature set is denoted as ,in Indicates the first Each environmental characteristic; calculate each characteristic. The mean absolute value of SHAP over all samples, denoted as

[0123] (7);

[0124] For device modalities, the original device dimension is: The feature set is denoted as ,in Indicates the first Each device feature; calculate each feature The mean absolute value of SHAP over all samples, denoted as

[0125] (8);

[0126] Step S34: Calculate the mean absolute value of SHAP based on all features within each modality. and The features are sorted in descending order of importance to obtain a ranking list of environmental features by importance. And device feature importance ranking list ;

[0127] Step S35: Calculate the Spearman correlation coefficient. Find features that carry similar information, when the calculated values ​​of two features are similar. If the value exceeds the set threshold, it is considered to carry similar information, and the list is sorted according to the importance of the features. or Remove redundant features from the SHAP values ​​that are ranked below a preset threshold. The preset threshold can be set according to the actual situation. Then, retrain XGBoost to update the feature importance ranking list. or In specific implementation, the following settings are made: Assuming a high correlation between features, calculate the Spearman correlation coefficient between environmental and device modes, such as... Figure 2 , Figure 3 The results shown yield a list of highly correlated features in multimodal data. As shown in Table 3;

[0128]

[0129] Table 3

[0130] In the initial SHAP values ​​generated by XGBoost, the environmental modal features are ranked as follows: out_co2 (SHAP: 0.2096) > out_temp (SHAP: 0.1833) > out_rh (SHAP: 0.1050). Therefore, out_co2 is retained, and redundant environmental features out_temp and out_rh are removed. The device modal features are ranked as follows: fcu_speed (SHAP: 0.5247) > fcu_fan_energy (SHAP: 0.2875). Therefore, fcu_speed is retained, and redundant environmental feature fcu_fan_energy is removed. After removing redundant features, XGBoost is retrained, and the final feature importance ranking list is generated. and ,like Figure 4 and Figure 5 The results shown : spl > in_co2 > in_temp > illu > in_rh > pm25 > out_co2 > wind_dir > baro_press > wind_spd > voc, :wifi_dev > fan_energy > light_energy > plug_energy > chill_energy > fcu_fan_energy > temp_set > sup_temp > sup_press.

[0131] Step S36: Construct the optimal feature subset for each modality using a stepwise forward selection strategy:

[0132] Step S361, Initialize the feature subsets as follows: , ;

[0133] Step S362: For the environmental mode, in the first... Wheel The Middle Features Add to the current subset to obtain the updated environmental feature subset. Similarly, we can obtain ;

[0134] Step S363: After each feature is added, retrain the environment-aware occupancy detection model and the device behavior occupancy detection model using the updated feature subset, and calculate the F1 score on the independent test set to evaluate their performance, denoted as... and ;

[0135] Step S364: If in a certain round The addition of new environmental features caused the F1 score to drop compared to the previous round. If the iterative selection process of environmental modal features is terminated, the feature set from the previous round is taken as the optimal environmental feature set, that is... Similarly, if Then let ;

[0136] This feature selection strategy ensures that the selected feature set maintains optimal performance while utilizing as few features as possible, avoiding model overfitting or performance degradation caused by the introduction of noisy features. Based on and The retrained model is defined as the optimal environment-aware occupancy detection model. and optimal device behavior occupancy detection model In practice, the LSTM and TCN models are retrained each time features are added. The performance of the resulting occupancy detection models is shown in Tables 4 and 5. Table 4 compares the performance of the LSTM model during iterative training, and Table 5 compares the performance of the TCN model during iterative training.

[0137]

[0138] Table 4

[0139]

[0140] Table 5

[0141] Both LSTM and TCN models experience a drop in F1-score when an 8th feature is added to the environment feature subset, at which point iterative training stops. Therefore, the optimal number of environment features and the optimal environment feature subset are both 7. The optimal environmental perception occupancy detection model is obtained by training the subset of environmental features {spl, in_co2, in_temp, illu,in_rh, pm25, out_co2}. Its base model is LSTM, with F1-score and ACC of 0.9704 and 0.9689, respectively;

[0142] Both LSTM and TCN models experience a drop in F1-score when a fifth feature is added to the device feature subset, halting iterative training. Therefore, the optimal number of device features and the optimal device feature subset are both 4. The optimal device behavior occupancy detection model is trained using the subset of device features {light_energy, fan_energy, fcu_fan_energy, wifi_dev}. Its base model is LSTM, with F1-score and ACC of 0.9632 and 0.9628, respectively;

[0143] Step S4: Construct an adaptive modality fusion mechanism based on a Broad Learning System (BLS): The output probabilities of each modality occupancy detection model trained in Step S2, together with their respective sliding window temporal features after feature filtering, are concatenated into a fusion input vector; the BLS model adaptively learns the joint contribution of each component in the multimodal input to the final judgment, and achieves deep fusion of multimodal information through the collaborative representation ability of its feature nodes and enhancement nodes, ultimately generating a highly robust meeting room occupancy status detection result.

[0144] Step S41: To achieve collaborative decision-making between environmental perception and equipment behavior modalities in determining occupancy status, an adaptive fusion mechanism based on a Broad Learning System (BLS) is designed. This mechanism constructs a joint representation space of feature node layer and enhancement node layer, dynamically integrates the prediction confidence and original temporal features of each modal occupancy detection model, and outputs the final meeting room occupancy status determination result.

[0145] Step S42: Set the current time step to be... The length of the sliding window is After feature filtering in step S3, only the dimensions corresponding to the optimal feature subset are retained, and the original input of the environmental modality is... The original input of the device modality is Flatten the above two-dimensional time series tensor into a one-dimensional vector:

[0146] (9);

[0147] Environmental perception occupancy detection model The output occupancy probability is Device behavior occupancy detection model The output occupancy probability is The above vectors are concatenated to form the complete input feature vector for the BLS fusion module:

[0148] (10);

[0149] Among them, the input feature dimension ;

[0150] Step S43: Input matrix As the actual input sample of the BLS system, its structure includes a feature node layer. With enhanced node layer :

[0151] Feature node layer: for input Perform a linear transformation to map to the feature plane and generate a feature node matrix:

[0152] (11);

[0153] in, This represents the optimal input weight matrix obtained using the sparse autoencoder method. For bias terms, The number of feature nodes. For non-linear activation functions (such as ReLU or Sigmoid)

[0154] Enhanced Node Layer: Introducing a random weight matrix on top of the feature nodes. and bias Generate enhanced nodes:

[0155] (12);

[0156] in, It is a non-linear activation function (such as Tanh). To increase the number of nodes;

[0157] Merging the input layer: Feature nodes and augmentation nodes are concatenated to form the actual input signal of BLS.

[0158] (13);

[0159] Step S44: The output layer of the BLS system directly solves for the connection weight matrix using the ridge regression generalized inverse. In the occupancy detection task of this invention (Binary classification task), calculated using the following ridge regression approximation formula:

[0160] (14);

[0161] in, The input matrix is ​​the combined input matrix of all samples in the training set, where N is the total number of samples in the training set. For the actual label vector, This is the regularization parameter.

[0162] At this point, the output prediction probability is:

[0163] (15);

[0164] like If the condition is met, it is determined as "occupied" (output 1); otherwise, it is determined as "unoccupied" (output 0).

[0165] Step S45: The model undergoes end-to-end training through supervised learning to accurately occupy labels. As a supervisory signal, the optimization objective is to minimize the binary cross-entropy loss:

[0166] (16);

[0167] The F1 score is monitored on the validation set to implement an early stopping strategy to prevent overfitting.

[0168] Step S46: After training, the BLS model serves as the final multimodal meeting room occupancy discriminator. It integrates the confidence of each modal occupancy detection model with the original temporal features within the complete sliding window to output robust and high-precision occupancy detection results. This structure avoids the backpropagation iterative optimization of traditional deep neural networks and has advantages such as fast training speed, strong generalization ability, and resistance to local optima. It is particularly suitable for edge scenarios in the building environment that require rapid deployment and stable operation.

[0169] In the specific implementation, the optimal environment perception occupancy detection model (base model: LSTM) and device behavior occupancy detection model (base model: LSTM) obtained by step S3 training are selected. The model output probability and feature vector are concatenated to form the input vector of BLS. The performance of the multimodal fusion occupancy detection model obtained by BLS training is shown in Table 6. Table 6 is a comparison of the performance of multimodal fusion and single-modal occupancy detection models.

[0170] The environmental perception model has an F1 score of 0.9704 and an accuracy of 0.9689; the device behavior model has an F1 score of 0.9632 and an accuracy of 0.9628; while the final model obtained by the multimodal fusion mechanism described in this invention has an F1 score of 0.9842 and an accuracy of 0.9801.

[0171]

[0172] Table 6

[0173] The results fully demonstrate that the BLS-based multimodal fusion model successfully achieves effective coordination between environmental information and device behavior, significantly outperforming any single-modal model. It effectively overcomes the perception limitations of a single modality in complex conference room scenarios, significantly improves the accuracy and robustness of occupancy status determination, and fully verifies the technical advancement and practical value of the "multimodal adaptive fusion mechanism" proposed in this invention.

[0174] Based on highly reliable occupancy detection technology, building space energy management and space resource optimization can achieve a fundamental shift from "passive response" to "proactive intelligence." Specifically, in meeting room scenarios, high-precision occupancy status perception can drive the on-demand start / stop and dynamic parameter adjustment of air conditioning, lighting, fresh air, and multimedia equipment in real time, effectively avoiding energy waste caused by "power remaining on even when people leave," and significantly reducing unnecessary energy consumption. Simultaneously, the system can compare real occupancy data with reservation information, automatically identifying idle periods of "reserved but not used" or "meetings ending early," and immediately releasing resources and pushing availability status to the shared scheduling platform. This improves meeting room turnover efficiency and alleviates the structural contradiction of resource shortages during peak hours and large-scale idleness during off-peak hours. Furthermore, the long-term accumulated high-confidence occupancy data can also be used to analyze space usage patterns, optimize meeting room configuration strategies, and assist in facility planning decisions, providing a solid data foundation and closed-loop control capabilities for building a green, efficient, and human-centered smart office environment.

[0175] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. The multimodal fusion described in the present invention can be extended to all modalities that can be obtained in any scenario, and is not limited to the environment and device modalities in the examples of the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A conference room space occupancy detection method based on multi-modal adaptive fusion, characterized in that, Comprise the following steps: S1, for all modal data available in the conference room, through the unsupervised anomaly detection method based on variational autoencoder, identify and eliminate the abnormal values in the sensor data; S2, according to the difference of data source mode, the data set is divided according to different modes after preprocessing, and the spatial occupancy detection ability is independently trained based on deep learning model on each modal data set, and the occupancy detection model of related mode is obtained; S3, importance evaluation and screening are carried out for each modal feature: The minimum number of feature screening strategy based on gradient boosting decision tree is adopted, the SHAP value is used to quantify the contribution of each feature to the occupancy detection task, and the sliding window is used to aggregate the time series features, and the feature importance ranking is calculated; On this basis, the features carrying similar information are found by the Spearman correlation coefficient, and the redundant features with SHAP value below the preset threshold are removed; Finally, the feature subset is constructed in order of feature contribution, and the occupancy detection model is iteratively trained, and the iteration is terminated when the model performance decreases, so as to determine the optimal feature set; S4, an adaptive modal fusion mechanism based on width learning system is constructed. The output probability of each modal occupancy detection model trained in step S2 is spliced into a fusion input vector together with the sliding window time series features after feature screening; The width learning system model adaptively learns the joint contribution of each component in the multi-modal input to the final discrimination, realizes the deep fusion of multi-modal information through the cooperative representation ability of its feature nodes and enhanced nodes, and finally generates the conference room occupancy state detection result.

2. The multi-modal adaptive fusion based conference room space occupancy detection method of claim 1, wherein, Step S1 comprises the following steps: S11, the multi-modal data collected in the current conference room can be divided into an environment modal data set and a device modal data set , the environment modal data set and the device modal data set An unsupervised outlier detection method based on variational autoencoder is used to identify and clean outliers in the data. In the abnormal value identification task, the latent distribution of normal data is modeled by using a deep generative model, and the abnormal samples deviating from the normal mode are identified by reconstruction error, and the specific steps are as follows: S111, constructing separate variational autoencoder networks for the environment modal dataset and the device modal dataset wherein, An encoder maps an input data vector into a mean and variance in a low-dimensional latent space, and samples from it to obtain a latent variable ; The decoder then optimizes the reconstruction likelihood reconstructs the original input The model is optimized with a joint objective of maximizing the reconstruction likelihood and minimizing the KL divergence, defined by the following equation: （1）； in, This is the input data vector for a certain mode; This represents sensor data in a given conference room. Under the condition of latent variables The approximate posterior distribution; Representing approximate posterior with prior The distance between them is used to avoid large deviations in the distribution of latent variables in the encoder output; It is the prior distribution of the latent variable z, usually set as a standard Gaussian distribution. ; Indicates that given a latent variable Under these conditions, generate observation data The probability of; Used to measure the current decoder from Reconstructed With reality The degree of fit, i.e., the reconstruction error, and the expected value. The larger the value, the more accurate the reconstruction. S112, the unlabeled data collected in the conference room is used to train the variational autoencoder model, so that the data distribution characteristics of the environment and device modal in the "normal state" are learned; After training, the model can reconstruct the data conforming to the normal mode, and the abnormal data caused by the sensor recording process will produce reconstruction deviation; S113、to the current to be detected multi-modal data sample , input the variational autoencoder model corresponding to the mode, calculate its reconstruction error ; In order to take into account the dimensional difference, the reconstruction error is normalized to obtain the standardized reconstruction score ; S114, set dynamic anomaly threshold If the normalized reconstruction score of a sample is determined to be an anomaly value; the sample determined to be an anomaly is directly eliminated.

3. The multi-modal adaptive fusion based conference room space occupancy detection method of claim 1, wherein, Step S2 comprises the following steps: S21, after completing the multi-modal data outlier cleaning, respectively based on the environment modal data set and the device modal data set Construct the initial occupancy state detection model of time sequence perception, and use the sliding window strategy to construct the input sequence for the preprocessed data, as follows: Set the window length as Each window generates an input sample Or Wherein And The corresponding target label Indicates whether the conference room is occupied at the current time step, where 1 is occupied and 0 is idle. S22, select two kinds of timing deep learning architecture, respectively, time convolution network and long short-term memory network, on the above-constructed sliding window data to carry out end-to-end training; define all models to take the conference room occupancy state two classification task as the goal, the output layer adopts Sigmoid activation function, generates occupancy probability ; S23, the model training adopts binary cross entropy as the loss function, and its mathematical expression is: （2）； wherein, is the total number of training samples, is the true label, is the occupancy probability predicted by the model; S24, in the test set, F1 score is used as the evaluation index, and accuracy ACC is used as the auxiliary evaluation index, which is defined by the following formula: （3）； （4）； （5）； （6）； Where, TP indicates the sample actually belonging to positive class and being correctly predicted as positive class, TN indicates the sample actually belonging to negative class and being correctly predicted as negative class, FP indicates the sample actually belonging to negative class but being wrongly predicted as positive class, and FN indicates the sample actually belonging to positive class but being wrongly predicted as negative class; The trained model is evaluated on the test set, and the F1 score and accuracy ACC score of each model trained on environmental data and device data are obtained as the performance benchmark of the subsequent feature screening step, to verify the effectiveness of the method.

4. The multi-modal adaptive fusion based conference room space occupancy detection method of claim 1, wherein, Step S3 comprises the following steps: S31, carry out minimum number feature screening based on gradient boosting tree for environmental modal features and device modal features respectively, and adopt XGBoost as a basic model; S32, on the basis of input samples constructed by a sliding window, aggregate the original time sequence features in each time window, and the specific process is as follows: For a sequence with window length , the mean value is calculated as the static representation of the window in the feature dimension, thus converting the time series input into an aggregated feature vector , and similarly for the aggregated vector of the device modality ; S33, respectively in the environmental modal dataset with the device modal dataset The XGBoost classifier is trained on and utilized to measure the average contribution of each feature to the model occupancy detection using the method of SHAP values based on the aggregated vector of each feature within each modality; For the environmental modalities, the original environmental dimension is , the feature set is denoted as , where denotes the th environmental feature; the SHAP absolute value mean of each feature over all samples is denoted as （7）； For device modality, the original device dimension is , the feature set is denoted as , where denotes the th device feature; the SHAP absolute value mean of each feature over all samples is denoted as: （8）； S34, mean of SHAP absolute values of all features within each modality and rank the importance of features in descending order, respectively obtaining an environmental feature importance ranking list and a device feature importance ranking list ; S35, calculating Spearman correlation coefficient Finding features carrying similar information, when the calculated greater than the set threshold value, it is considered to carry similar information, and the feature importance ranking list is sorted Or Remove redundant features with SHAP values whose ranking is lower than the preset threshold value, and retrain XGBoost to update the feature importance ranking list Or ; S36, combine the step-by-step forward selection strategy to construct the optimal feature subset under each modality, and the specific process is as follows: S361, the initialization feature subsets are defined as , ; S362、For the environmental modal, in the first wheel will the middle of the characteristics Join the current subset, get the updated environmental feature subset ; similarly, we can get ; S363. After each feature is added, the environment-aware occupancy detection model and the device behavior occupancy detection model are retrained using the updated feature subset, and their performance is evaluated by calculating the F1 score on an independent test set. This score is denoted as... and ; S364、if in a certain round the newly added environmental features cause the F1 score to decrease compared to the previous round, i.e. , then terminate the iterative screening process of the environmental modal features, and take the feature set of the last round as the optimal environmental feature set, i.e. ; similarly, if , then let ; The feature selection strategy ensures that the selected feature set utilizes the least amount of feature quantity while maintaining optimal performance, based on and The retrained model is defined as an optimal environment-aware occupancy detection model and an optimal device-behavior occupancy detection model .

5. The multi-modal adaptive fusion based conference room space occupancy detection method of claim 1, wherein, Step S4 includes the following steps: S41, design an adaptive fusion mechanism based on a width learning system, which dynamically integrates the prediction confidence of each modality occupancy detection model and the original time sequence features by constructing a feature node layer and an enhanced node layer joint representation space, and outputs the final conference room occupancy state discrimination result; S42, let the current time step be , the sliding window length be , and only the dimensions corresponding to the optimal feature subset be retained after the feature screening in step S3, then the original input of the environment modality is , and the original input of the device modality is , the two-dimensional time series tensors are flattened into one-dimensional vectors: （9）； Environment-aware occupancy detection model The output occupancy probability is Device behavior occupancy detection model The output occupancy probability is ; The above vector is spliced into a complete input feature vector of the width learning system fusion module: （10）； wherein the input feature dimension ; S43, the input matrix As a practical input sample for a wide learning system, its structure contains a layer of feature nodes with a layer of augmented nodes : Feature node layer: to input Linear transformation, mapping to feature plane, generating feature node matrix: （11）； wherein, is the optimal input weight matrix obtained by sparse auto-encoding, is the bias term, is the number of feature nodes, is a non-linear activation function; Enhanced node layer: on the basis of feature nodes, introduce random weight matrix and bias , generate enhanced nodes: （12）； wherein, is a non-linear activation function, is the number of enhancement nodes; Merge the input layer: splice the feature nodes and enhanced nodes to form the actual input signal of the width learning system: （13）； S44, the output layer of the width learning system directly solves the connection weight matrix by ridge regression generalized inverse , in the occupancy detection task , is calculated by the following ridge regression approximation formula: （14）； wherein, is the merged input matrix for all samples of the training set, N is the total number of samples of the training set, is the true label vector, is the regularization parameter, At this time, the output prediction probability is: （15）； If "occupied" and outputs 1, otherwise it is determined to be "unoccupied" and outputs 0. S45, the model is trained end-to-end by supervised learning with real occupancy labels For supervised signals, the optimization objective is to minimize the binary cross-entropy loss: （16）； And monitor the F1 score on the validation set to implement the early stopping strategy; S46, after training, the width learning system model is used as the final multi-modal conference room occupancy discriminator, which integrates the confidence of each modality occupancy detection model and the original time sequence features in the complete sliding window to output the occupancy detection result.

Images