Method and Device for Predicting Working Status of Office Buildings Based on Multimodal Data

Abstract

This application provides a method and apparatus for predicting the working status of office buildings based on multimodal data, belonging to the field of artificial intelligence. The method includes: collecting objective environmental data and crowd perception data to form a raw dataset; preprocessing and feature construction of the raw dataset to form a new dataset; encoding the multi-source data in the dataset to obtain multiple modality vectors of different modes; fusing the multiple modality vectors to obtain a fused vector; establishing a classifier based on a time attention mechanism; and training the time attention-based classifier using the fused vectors within several consecutive time windows to obtain a human state prediction model. This method can achieve accurate prediction of human state.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and apparatus for predicting the working status of office buildings based on multimodal data. Background Technology

[0002] In applications such as smart health, environmental assessment, and behavior recognition, simultaneously collecting environmental data (temperature, humidity, wind speed, PM, CO2, etc.), human perception data (EEG, skin conductance), and subjective feedback (questionnaires / labels) has become an important means of assessing human condition.

[0003] Existing methods often process multimodal data using a single modality or simple concatenation, which fails to effectively address issues such as temporal scale differences between modalities, noise and data loss, coupling of short-term and long-term trends, and latency constraints on real-time inference. Therefore, there is an urgent need for a deployable model and method that can both process the "current set of data" for immediate identification and integrate "historical multiple sets of data with the current set" for trend and state prediction. Summary of the Invention

[0004] This application provides a method and apparatus for predicting the working status of office buildings based on multimodal data, aiming to achieve accurate prediction of human body status.

[0005] Firstly, this application provides a method for predicting the working status of office buildings based on multimodal data, including: Collect objective environmental data and human perception data to form a raw dataset; The original dataset is preprocessed and features are constructed to form a new dataset; Encode the multi-source data in the dataset to obtain modality vectors of multiple different modalities; The modal vectors of multiple different modes are fused to obtain a fused vector; Develop a classifier based on the time attention mechanism; A human state prediction model is obtained by training a classifier based on a time attention mechanism using fusion vectors within several consecutive time windows.

[0006] Optionally, after collecting objective environmental data and crowd perception data, the objective environmental data and crowd perception data are time-synchronized and windowed to form the original dataset.

[0007] Optionally, the steps of preprocessing and feature construction of the original dataset include: Environmental modality features and perceptual modality features are extracted from the original dataset. The features extracted for environmental modality include one or more of the following: temperature-related features, humidity features, wind speed features, and air quality-related features. The features extracted for perceptual modality include one or more of the following: electroencephalogram (EEG) signal features, time-domain perceptual features, skin conductance features, and subjective feedback features. The extracted features are then normalized.

[0008] Optionally, the steps of encoding multi-source data in the dataset include: All feature variables within the same time window and under the same modality are concatenated into a one-dimensional feature vector in a predefined order; The input one-dimensional feature vector is mapped layer by layer through a multi-layer fully connected network, and a non-linear activation function is introduced to enhance the representation capability. One-dimensional feature vectors of different lengths and semantics are uniformly mapped to modal vectors of fixed dimensions, thereby obtaining modal vectors of multiple different modalities.

[0009] Optionally, the steps for fusing mode vectors of multiple different modes include: Map each modality vector to a unified attention computation space; Calculate the correlation between each modality and the current state query vector; The degree of relevance is normalized into attention weights; The weights are used to sum the modal vectors to obtain the fusion vector.

[0010] Alternatively, a time-based attention mechanism may be adopted as follows: Calculate the importance score of the fusion vector and its current state within several time windows; The temporal attention weights are derived based on the importance of the fusion vectors within several time windows. Based on temporal attention weights, the fusion vectors within several time windows are weighted and summed to obtain the temporal context representation at the current moment.

[0011] Optionally, the steps for training a classifier based on a time attention mechanism using fusion vectors from several consecutive time windows include: Construct a training sample set ; Perform forward computation on the input sample to obtain the predicted state probability; Calculate the loss function based on the prediction results and the true labels; The gradient is calculated through backpropagation; Update the parameters of each module; Repeat the above steps until training converges or the preset number of iterations is reached.

[0012] Secondly, this application provides a device for predicting the working status of office buildings based on multimodal data, comprising: The data acquisition and synchronization module is used to collect objective environmental data and human perception data to form a raw dataset. The preprocessing and feature engineering module is used to preprocess the original dataset and construct features to form a new dataset. The modality-specific encoding and vectorization module is used to encode multi-source data in a dataset to obtain modality vectors of multiple different modalities; The multimodal fusion module is used to fuse modal vectors of multiple different modalities to obtain a fused vector; The human state prediction module is used to build a classifier based on a time attention mechanism; and to train the classifier based on a time attention mechanism using fusion vectors within several consecutive time windows to obtain a human state prediction model.

[0013] Thirdly, this application provides an electronic device, including the office building working status prediction device based on multimodal data as described above.

[0014] Fourthly, this application provides a computer-readable storage medium storing at least one piece of program code, which is executed by a processor to implement the office building working status prediction method based on multimodal data as described in any of the preceding claims.

[0015] The beneficial effects of this application include: This application provides a method for predicting the working status of office buildings based on multimodal data. It involves collecting and processing multi-source data to form a fusion vector. This fusion vector, generated within several consecutive time windows, is used to train a classifier based on a time attention mechanism, thereby obtaining a human state prediction model and achieving accurate prediction of human state. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for predicting the working status of office buildings based on multimodal data, provided in an embodiment of this application; Figure 2 A flowchart for encoding multi-source data is provided as an embodiment of this application; Figure 3 A flowchart illustrating the workflow of a classifier based on temporal attention weights provided in an embodiment of this application; Figure 4 A schematic diagram of a training process provided in an embodiment of this application; Figure 5 A structural block diagram of an office building work status prediction device based on multimodal data provided in an embodiment of this application; Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0018] The attached figures are labeled as follows: 11: Data acquisition and synchronization module; 12: Preprocessing and feature engineering module; 13: Modality-specific encoding and vectorization module; 14: Multimodal fusion module; 15: Human state prediction module; 21: Processor; 22: Memory. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Figure 1 A flowchart illustrating a method for predicting the working status of office buildings based on multimodal data, provided in one embodiment of this application. See also... Figure 1 ,include: S101. Acquire environmental data and crowd perception data to form a raw dataset.

[0021] In some examples, environmental data include: temperature, humidity, wind speed, PM levels, CO2 concentration, etc.

[0022] In some examples, crowd perception data includes both objective and subjective crowd perception data. Objective crowd perception data includes EEG data, ductal electroencephalogram (EEG) data, etc., while subjective crowd perception feedback includes questionnaire results.

[0023] The core purpose of step S101 is to map multimodal data from different sources, with different sampling frequencies and different generation methods to the same time reference through the data acquisition and synchronization module, thereby providing a reliable and alignable raw data foundation for subsequent multimodal fusion, real-time judgment and time series trend modeling.

[0024] In applications such as built environment assessment and spatial crowd behavior perception assessment, there are significant differences between physical environmental indicators (such as temperature, humidity, wind speed, PM2.5, and CO2 concentration) collected by environmental sensors and crowd perception data (objective crowd perception data such as EEG and ductus skinitis data, and subjective crowd perception feedback such as questionnaire results). For example, environmental parameters such as temperature and humidity are usually sampled at a frequency of seconds or lower, while perception signals such as EEG and ductus skinitis may be sampled continuously at frequencies of tens or even hundreds of hertz. Subjective crowd perception feedback data is often generated irregularly depending on the form of the event. Without unified time synchronization, these data are prone to state judgment bias during subsequent fusion, failing to accurately reflect the true situation at the same moment or under the same state.

[0025] Therefore, in step S101, multi-source heterogeneous data is acquired in parallel, and then a unified timestamp mechanism is used to convert the multi-source data into a multimodal data stream that can correspond on the same time axis. At the same time, an additional windowed timestamp index is generated for signals with sampling frequencies significantly higher than other modalities, ensuring that downstream models perform parallel feature extraction and aligned modeling within the same time window.

[0026] It should be noted that "high sampling rate signal" in this application refers to data in which the number of sampling points per unit time is much higher than that of other modalities.

[0027] For example, if the sampling rate of EEG data is 250Hz, 250 voltage sampling points will be generated per second; while environmental data may only generate one record per second, and subjective feedback may even occur every few minutes. This inconsistency in sampling frequency leads to a severe mismatch in time scale between the multi-source data. To address this issue, this application utilizes a "windowing mechanism" to replace the method of "aligning high-sampling-rate signals point-by-point with low-frequency data," allowing high-sampling-rate signals to be aggregated within a time window and generating a unified timestamp index for that window for parallel processing by subsequent modules.

[0028] The inputs obtained in step S101 include the following three types of raw data: (1) Environmental modal raw data: including numerical data such as temperature, relative humidity, wind speed, PM, CO2, etc., which are usually collected by environmental sensors at fixed time intervals, such as generating a record every 1 second or every 5 seconds.

[0029] (2) Raw data of sensory modality: including raw EEG channel signals and continuous EEG signals. EEG and EEG are usually high sampling rate signals, that is, a large number of sampling points per unit time. For example, EEG is sampled at 128Hz, 250Hz or higher, which means that 128 to 250 data points are generated per second.

[0030] (3) Subjective feedback data: including scale scores (1, 0.5, 0, -0.5, -1 corresponding to very satisfied, fairly satisfied, average, dissatisfied, very dissatisfied), and the timestamps corresponding to the specific time of filling or recording.

[0031] Once the raw data is obtained, it needs to be time-synchronized and windowed to form the original dataset, as follows: First, select a unified time base, such as system time or server time, and then perform the following steps: Step 1: Unified timestamp recording: All modal data will be accompanied by a timestamp during collection to indicate the actual time point when the data was generated.

[0032] Step 2: Time window division: Based on the system's personalized configuration, the continuous time is divided into several time windows of fixed length (e.g., 30 seconds or 60 seconds per window).

[0033] Step 3: Generate windowed index for high sampling rate signals: For high sampling rate signals such as EEG and TENS, in order to identify all sampling points within the window, they are placed into the corresponding window according to the timestamp, without directly participating in cross-modal point-by-point alignment, and a window index (Window ID) is generated for each window.

[0034] Step 4: Window mapping of low-frequency data and event data: Environmental data and subjective feedback data serve as the environmental and subjective state inputs for this window, and will be mapped to the corresponding window according to the timestamp.

[0035] Table 1: Time Synchronization and Windowed Data Alignment Table

[0036] See Table 1 for an example of how different modal data can be mapped to a unified window (using a 30-second window as an example) through time synchronization and windowing.

[0037] In this way, each window ID corresponds to a complete set of multimodal data, which includes both the original or aggregated information of the high sampling rate sensing signal and low-frequency environmental parameters and subjective feedback.

[0038] In step S101, after processing, two types of data structures will be output: The first type is single-window multimodal raw data packets: each window corresponds to a set of environmental, perceptual, and subjective data, used for real-time state determination.

[0039] The second type is a set of windows arranged in time series: multiple consecutive windows form a historical sequence in chronological order, which is used by time series models for trend analysis and population state prediction. In step S101, time synchronization and windowing processing at the front end of the system ensure a one-to-one correspondence between different modal data in the time dimension, providing a reliable foundation for subsequent cross-modal coding, contrastive learning, and attention fusion.

[0040] Meanwhile, by leveraging the windowed indexing mechanism of high-sampling-rate signals, the system not only preserves fine-grained information of the sensed signals but also prevents computational complexity from spiraling out of control due to differences in sampling frequencies, thus ensuring the system's real-time performance and scalability from an engineering perspective. This design, through "early synchronization and window-level alignment" during data collection, fundamentally reduces the time skew problem in multimodal fusion, significantly improving the system's robustness and decision stability in real-world complex scenarios.

[0041] S102. Preprocess and feature construct the original dataset to form the original dataset.

[0042] In step S102, the preprocessing and feature engineering module uses a fixed-length time window as the basic processing unit, and performs feature calculations on environmental data and crowd perception data in each window.

[0043] In some examples, step S102 includes: S1021. Extract features from the original dataset.

[0044] In some examples, step S1021 includes: Step 1: Environmental modal feature extraction.

[0045] Environmental modal data includes continuous sensor signals such as temperature, relative humidity, wind speed, particulate matter concentration (PM), and carbon dioxide concentration (CO2). For each environmental variable, the system extracts the following statistical and trend characteristics after smoothing within a window.

[0046] (1) Temperature-related characteristics ENV_MEAN_TEMP: The arithmetic mean of the smoothed temperature samples within the window is used to represent the overall temperature level of the current window.

[0047] ENV_STD_TEMP: Calculates the standard deviation of temperature samples within the window, used to characterize temperature stability.

[0048] ENV_MIN_TEMP / ENV_MAX_TEMP: These values ​​represent the minimum and maximum temperatures within a window, respectively, and are used to capture short-term extreme changes.

[0049] ENV_RANGE_TEMP: Calculates the temperature fluctuation range by the difference between the maximum and minimum values.

[0050] ENV_SLOPE_TEMP: The temperature change trend is obtained by dividing the difference between the average temperature of the current window and the average temperature of the previous window by the window duration. This trend is used to reflect the short-term rate of rise and fall.

[0051] (2) Humidity and wind speed characteristics ENV_MEAN_HUMIDITY / ENV_STD_HUMIDITY: Calculate the mean and standard deviation of humidity within the window, respectively, to describe humidity levels and fluctuations.

[0052] ENV_MEAN_WIND: Takes the mean value of wind speed samples within the window, used to characterize ventilation intensity.

[0053] (3) Air quality related characteristics ENV_PM_MEAN / ENV_PM_MAX / ENV_PM_PCT_OVER_X: These calculate the mean, maximum, and proportion of samples exceeding a preset threshold PM concentration within the window, respectively, to quantify particulate matter exposure levels and the risk of exceeding limits.

[0054] ENV_CO2_MEAN / ENV_CO2_MAX / ENV_CO2_SLOPE: These calculate the average, maximum, and trend of CO2 concentration, respectively, to reflect ventilation conditions and gas accumulation.

[0055] (4) Comprehensive environmental indicators ENV_COMBO_TEMP_HUMIDITY_INDEX: A combined comfort index constructed based on temperature and humidity characteristics, used to comprehensively quantify environmental comfort.

[0056] Step 2: Perceptual Modality Feature Extraction The perceptual modalities include electroencephalography (EEG), electrical conductance analysis (EDA), and subjective feedback ratings. The system constructs features from the perceptual signals at the time domain, frequency domain, and statistical levels.

[0057] In some examples, step 2 includes: The first step is to extract features from the electroencephalogram (EEG) signals.

[0058] The characteristics of EEG signals include: frequency band energy characteristics, spectral entropy characteristics, dominant frequency characteristics, and Hjorth parameters.

[0059] (1) Frequency band energy characteristics PHY_BANDPOWER_DELTA~GAMMA: Performs short-time Fourier transform (STFT) or Welch power spectrum estimation on the EEG signal within the window to obtain the power spectral density. Calculate the relative energy within each predefined frequency band. :

[0060] In the formula: This represents the Delta, Theta, Alpha, Beta, or Gamma frequency bands. This vector is used to characterize the distribution structure of brainwave rhythms.

[0061] (2) Spectral entropy characteristics PHY_SPECTRAL_ENTROPY: In the normalized power spectrum Calculate spectral entropy based on :

[0062] This feature is used to describe the complexity and disorder of the EEG spectrum.

[0063] (3) Dominant frequency characteristics PHY_PEAK_FREQUENCY: Finds the frequency position corresponding to the maximum power spectrum within the preset frequency band, used to indicate the currently dominant EEG rhythm.

[0064] (4) Hjorth parameter PHY_HJORTH_ACTIVITY / MOBILITY / COMPLEXITY: Calculates Hjorth activity, mobility, and complexity parameters based on time-series signals to describe the dynamic characteristics and structural changes of EEG signals.

[0065] The second step is to extract time-domain perceptual features.

[0066] PHY_TIME_ENERGY: Calculates the signal energy or root mean square (RMS) within the window, used to reflect the overall signal strength.

[0067] PHY_AMPLITUDE_MEAN / STD / MAX / MIN: Calculate the mean, standard deviation, maximum and minimum amplitude values ​​respectively, used for basic amplitude statistical description.

[0068] PHY_ZERO_CROSSING_RATE: Counts the number of times the signal crosses zero within the window and normalizes it, used to capture high-frequency jitter characteristics.

[0069] Step 3: Extracting electrical skin characteristics (EDA).

[0070] EDA_TONIC_LEVEL: Extracts low-frequency baseline components within the window to reflect the baseline arousal level.

[0071] EDA_PHASIC_COUNT: The number of skin conductance transient response (SCR) events within the statistics window, used to characterize the frequency of stress responses.

[0072] EDA_PHASIC_MEAN_AMPL: Calculates the average amplitude of SCR events to reflect the intensity of the reaction.

[0073] The fourth step is to extract subjective feedback features.

[0074] SUBJECT_SCORE_NORM: Maps subjective scale scores (1, 0.5, 0, -0.5, -1) to continuous normalized values ​​to express an individual's subjective feelings.

[0075] Step 5: Extract window quality assessment features.

[0076] META_WINDOW_QUALITY_SCORE: Constructs a window quality score based on factors such as sampling completeness, noise level, and outlier ratio. This score is used by the subsequent model to suppress or remove low-quality windows.

[0077] S1022. Perform feature normalization on the extracted features.

[0078] To eliminate the interference of different physical dimensions and value ranges on model training, the system will uniformly perform numerical scaling on all feature dimensions after completing the calculation of various feature variables.

[0079] During the model training phase, the system will use a standardization transformation to normalize the features, and calculate the mean and standard deviation of each feature dimension using the training sample set as the statistical benchmark. Let the th feature be... The sample values ​​corresponding to each feature in the training set are: Their mean and standard deviation are respectively and Then the standardized features It can be represented as:

[0080] In this approach, all features can be mapped to a unified numerical space with zero mean and unit variance during the training phase, which can improve the model's convergence speed and parameter learning stability.

[0081] During the inference phase, the system directly reuses the mean and standard deviation parameters obtained during the training phase to perform the same standardization transformation on the real-time input features, without recalculating the statistics to ensure the consistency of feature distribution during training and inference.

[0082] This design avoids the risk of future information or data leakage introduced during the inference phase, thus ensuring that the model will not lead to decision bias due to input scale drift during actual deployment. Simultaneously, this module solidifies standardized parameters as part of the model, providing a stable and reliable numerical foundation for multimodal fusion and state determination through a training-inference consistent feature scale control mechanism.

[0083] The advantage of this module lies in its ability to stably transform multi-source heterogeneous raw data into high-quality structured feature representations by introducing signal cleaning, window-level feature construction, and unified standardization mechanisms during the preprocessing and feature engineering stages. Addressing the potential numerical instability and training bias issues that may arise when multimodal data is directly input into the model, this module effectively reduces the impact of differences in sampling rate, physical dimensions, and numerical scales among multi-source heterogeneous data, namely environmental parameters, objective indicators of human perception, and subjective feedback.

[0084] To address the issue of distorted state determination caused by input scale drift or statistical bias, this module can use the training sample set as a benchmark to solidify standardized parameters of features, thereby achieving consistent normalization mapping rules during the inference phase. This ensures consistency in feature distribution between the training and actual deployment phases, improving the system's reliability and reproducibility from an engineering perspective.

[0085] Furthermore, to ensure the system maintains robust output even when faced with increased noise, missing data, or local anomalies, and to provide a stable and comparable input basis for subsequent modality coding and fusion stages, this module proposes a feature extraction and quality assessment strategy based on windows.

[0086] This module improves the usability, robustness, and engineering feasibility of multimodal features without significantly increasing computational complexity, providing key support for the overall performance and generalization ability of this invention.

[0087] S103. Encode the multi-source data in the dataset to obtain modality vectors of multiple different modalities.

[0088] After the preprocessing and feature engineering modules are run, the original multi-source data will be transformed into a structured, normalized feature set. In addition, the modality-specific encoding and vectorization module can further map the structured features under different modalities into vector representations in a unified dimension, continuous numerical space to provide a standardized input form for subsequent state determination or fusion calculation.

[0089] Since the feature selection and numerical normalization of the data modalities in this system have been completed in the previous stage, and the data modalities included are only two types, namely environmental modalities and human perception modalities (including objective EEG, skin conductance and subjective feedback), in order to reduce system complexity and enhance engineering feasibility, multilayer perceptron (MLP) will be used as the unified vectorization coding method for this module.

[0090] This module constructs independent MLP encoders for different modalities, but their overall processing flow is consistent, specifically including the following steps: (1) Intramodal feature concatenation: All feature variables within the same time window and under the same modality are concatenated into a one-dimensional feature vector in a predefined order.

[0091] (2) Linear mapping and nonlinear transformation: The input feature vector is mapped layer by layer through a multi-layer fully connected network, and a nonlinear activation function is introduced to enhance the representation capability.

[0092] (3) Fixed-dimensional vector output: The feature sets of different lengths and semantics are uniformly mapped to fixed-dimensional modal vector representations.

[0093] See Figure 2 In some examples, step S103 includes: S1031. Vectorize the features of the environmental modalities.

[0094] (1) Input feature construction.

[0095] In the environmental modality, the system sequentially concatenates the environmental features extracted within the same time window (including the corresponding statistics and trend features of temperature, relative humidity, wind speed, PM, CO2, etc.) to form the environmental modality input vector. :

[0096] In the formula: The total dimensions representing the modal characteristics of the environment; each dimension , All of these are environmental characteristic variables that have already undergone normalization.

[0097] (2) MLP encoding process The vectorized encoding of environmental modes is implemented through an L-layer fully connected network, and its layer-by-layer computation process is as follows: ,

[0098] In the formula: , which is the input feature vector; For the first Layer fully connected weight matrix; This is the corresponding bias vector; This is a non-linear activation function (ReLU is used here) to enhance the non-linear expressive power of the model.

[0099] (3) Environmental mode vector output.

[0100] The output of the final layer serves as a vectorized representation of the environment modality:

[0101] In the formula: , This is the preset vector dimension, used to maintain consistency with other modalities.

[0102] S1032. Vectorize and encode the perceptual modality.

[0103] (1) Input feature construction.

[0104] Perceptual modalities include multi-source features such as EEG, SCADA, and subjective ratings. The system concatenates all perceptual-related features within the same time window into a unified input vector:

[0105] In the formula: The total dimension of the perceived modal features; all feature variables are numerical values ​​after feature engineering and normalization.

[0106] (2) MLP encoding process.

[0107] The perceptual modality is also vectorized using a multilayer perceptron, and its computational form is as follows:

[0108] In the formula: , , These are the weight matrix and bias vector of the perceptual modality MLP, respectively; the activation function is set to be consistent with the environmental modality to maintain numerical stability.

[0109] (3) Sensing modal vector output.

[0110] The final vector representation of the perceptual modality is: ,in: The output dimension is consistent with the environment modality vector, which facilitates subsequent unified processing.

[0111] After processing by this module, the system outputs two types of vector representations within each time window: Environment mode vector: .

[0112] Perceptual modal vectors: .

[0113] The vectors mentioned above are all fixed-dimensional, continuous numerical representations. Their numerical scale and distribution have been constrained by the normalization in the previous stage and the nonlinear mapping in this module, and can be directly used as inputs for subsequent fusion, judgment or regression modules.

[0114] The aforementioned feature engineering module has completed noise suppression, scale unification, and semantically relevant feature construction for different data modalities. Furthermore, the input features for each modality possess high information density and discriminative power, thus enabling effective representation learning without relying on complex deep models. Based on this, this application employs a multilayer perceptron (MLP) as a modality-specific vectorization encoding method. Under the premise of simple structure and controllable computational overhead, a stable mapping of different modal features to a unified representation space is achieved. This design avoids the implementation difficulties and inference overhead associated with complex temporal or generative models. Simultaneously, it fully inherits and amplifies the physical and perceptual semantic information contained in the preceding feature engineering stage, resulting in vector representations that exhibit good numerical stability and state discriminative ability.

[0115] Furthermore, this design constructs independent encoding networks for different modalities, effectively reducing mutual interference caused by differences in feature distributions between modalities. This enhances the system's robustness under complex environmental conditions and data fluctuations, demonstrating good engineering feasibility and practical application value.

[0116] S104 performs a fusion process on the modal vectors of multiple different modes to obtain a fused vector.

[0117] The attention-based multimodal fusion module performs unified fusion processing on vectorized representations from different modalities. After independently encoding and vectorizing the environmental modality, perceptual modality, and subjective feedback modality, the data from each modality has been represented as vectors with consistent structure and uniform scale. Based on this, this module generates a fused vector representation that comprehensively reflects the overall state of the current time window by weighted integration of the multimodal vectors.

[0118] This fusion vector serves as a unified input for subsequent system state determination, trend analysis, or decision-making models, enabling information from different sources to participate in modeling within the same representation space. This avoids the structural complexity caused by scattered input of multimodal data and improves the stability and consistency of the overall system modeling.

[0119] In this embodiment, the system receives the following mode vectors that have undergone intra-modal encoding and normalization within each time window: Environment mode vector: .

[0120] Perceptual modality vectors (such as a combined representation of conductance of skin and electroencephalography): .

[0121] in, To achieve the unified vector dimension, the modal vectors have already undergone scale unification through the preceding module (e.g., normalization to the interval [-1, 1] or [0, 1]).

[0122] The goal of this module is to adaptively calculate modal weights based on the relative contribution of each modality to the state determination within the current time window, and generate a fusion vector representation.

[0123] This module adopts a weighted attention fusion mechanism. Its core idea is: (1) map each modal vector to a unified attention computation space; (2) calculate the correlation between each modality and the "current state query vector"; (3) normalize the correlation into attention weights; (4) use the weights to perform weighted summation on the modal vectors to obtain the fusion representation.

[0124] In some examples, step S104 fuses the multimodal vectors using an attention-based multimodal fusion module to form a fused vector.

[0125] In some examples, the specific steps and formulas for attention calculation are as follows: Step 1: Linear mapping of modal vectors (Key / Value construction) First, a linear mapping is performed on each modality vector to obtain the key vector and value vector required for attention computation:

[0126]

[0127] in: Indicates the modal type; , It is a trainable linear mapping matrix; Indicates the first The key vector of each modality; This represents the modal value vector used for fusion.

[0128] Step 1 is used to map vectors from different modalities to a unified attention space, ensuring that subsequent similarity calculations are comparable.

[0129] Step 2: Construct the query vector (Query).

[0130] Introduce a query vector for modal importance evaluation. The query vector is a learnable parameter in the model, and its dimension is the same as the modal key vector. During model training, the query vector... The backpropagation error signal of the downstream state determination task is optimized to characterize the system's focus on multimodal information in the current application scenario.

[0131] Step 3: Calculate the modal correlation score.

[0132] The relevance score between the query vector and each modal key vector is calculated using the dot product method:

[0133] In the formula: Indicates the first The original relevance score of each modality; The dimension of the key vector; the denominator This is used to prevent the dot product value from becoming too large and to stabilize the training process. (Score) This reflects the current time window, the first The degree of matching between modal information and the target of system state determination.

[0134] Step 4: Adjust modal quality.

[0135] If the system simultaneously obtains data quality scores for each modality Then the correlation score Adjustments are needed:

[0136] In the formula: It reflects modal integrity, noise level, or sampling reliability; when the quality of a certain modality is low, its score is automatically suppressed.

[0137] Step 5: Attention weight normalization.

[0138] The system performs a normalization operation on the adjusted relevance scores to obtain attention weights:

[0139] in: The sum of all modal weights satisfies: This step maps modal importance to interpretable relative weights.

[0140] Step 6: Weighted fusion to generate a fusion vector.

[0141] The system performs a weighted summation of the modality value vectors based on attention weights to obtain the fused representation:

[0142] in: This is the multimodal fusion vector for the current time window; this vector serves as the unified input for subsequent state determination or trend analysis modules.

[0143] This application introduces an explicit fusion mechanism based on attention weights in the multimodal vector fusion stage. This enables the system to adaptively adjust the contribution ratio of each modality in the fusion process according to changes in the state of modalities such as environment, perception, and subjective feedback within different time windows. Compared with simple vector concatenation, this method effectively suppresses the interference of information redundancy or low-relevance modalities on the overall judgment result by explicitly calculating modal importance. At the same time, by introducing a modal quality factor in the weight calculation, the fusion result has better robustness to increased modal noise, signal instability, or local missing data, avoiding the degradation of the overall system performance due to anomalies in a single modality.

[0144] This fusion method mainly involves linear mapping, weighted summation, and then normalization to obtain the output result. Its calculation process is simple and clear, easy to implement, suitable for engineering, and suitable for deployment on some real-time and edge devices.

[0145] S105. Establish a classifier based on the time attention mechanism.

[0146] After completing the preprocessing, feature engineering, modality-specific encoding, and fusion of the aforementioned multimodal data, the system will obtain a fusion vector representing the current environment and individual state within each time window. However, the human body's comfort state (e.g., focus, fatigue, stress, etc.) is usually not determined by instantaneous characteristics within a single time window, but rather by the cumulative effect of state changes within multiple consecutive time windows.

[0147] Therefore, the core task of this module (human state prediction module based on temporal fusion vector) is to use fusion vectors within multiple consecutive time windows to model the evolution relationship of the state over time, and to determine the human comfort state of the subject at the current moment based on the relationship.

[0148] To address the aforementioned issues, this module designs a temporal state determination method based on a time attention mechanism. This method achieves adaptive utilization of multi-window temporal information by explicitly modeling the contribution of different historical time windows to the current state, thereby avoiding the risk of misjudgment caused by simple time averaging or relying solely on the current window.

[0149] The module's input format: Suppose that before the current decision time, a total of consecutive... Each time window has a corresponding fusion vector, which is represented as follows:

[0150] In the formula: Indicates the first The multimodal fusion vector corresponding to each time window; Indicates the dimension of the fusion vector; This indicates the time window corresponding to the current decision moment; The number of historical windows used for status determination.

[0151] Module output format: This module outputs the state category prediction result at the current moment:

[0152] The processing flow of this module can be summarized in the following steps: Temporal input construction: Constructing a time sequence from the fusion vectors obtained within multiple consecutive time windows in chronological order. Preceding input sequence; Temporal attention weight calculation: For each time window vector, calculate its importance weight for the current state determination; Temporally weighted feature aggregation: Based on attention weights, the vectors of each time window are weighted and summed to form the temporal context representation of the current moment; State classification determination: Input the weighted time series representation into the classifier and output the current human comfort state category.

[0153] See Figure 3 In some examples, the specific process of the time attention mechanism is as follows: The first step is to calculate the window correlation score.

[0154] For the Fusion vector of each time window The system first calculates the score related to the current state determination task:

[0155] In the formula: Represents a linear mapping matrix; Represents the bias vector; Represents the attention score vector; Indicates the intermediate hidden space dimension; Represents a nonlinear activation function; Indicates the first Each time window is used to score the importance of the current state determination.

[0156] This step is used to map the fused vectors from different time windows to a unified relevance scoring space.

[0157] The second step is to normalize the time attention weights. To ensure the comparability of the importance of each time window, the system normalizes the above scores to obtain the time attention weights:

[0158] In the formula: , .

[0159] This weight is used to represent the relative contribution of different historical time windows to the current state determination.

[0160] Step 3: Time-weighted feature aggregation.

[0161] Based on the calculated temporal attention weights, the system performs a weighted summation of the fused vectors of all time windows to obtain the temporal context representation of the current moment:

[0162] In the formula: This vector comprehensively considers the temporal characteristics after the evolution of historical states. It contains both short-term change information and long-term trend information.

[0163] Step 4: State classification model.

[0164] In obtaining the temporal context representation Then, the system uses a classifier to determine its state:

[0165] In the formula: Represents the classification mapping matrix; Indicates category bias; The function is used to output the probability distribution of each state category.

[0166] Compared with existing methods that rely solely on a single time window or simple time averaging, this module offers at least the following technical advantages: (1) Make full use of multi-window time sequence information: The introduction of time attention mechanism enables the system to integrate information from multiple historical time windows, thereby effectively depicting the evolution process of human comfort state.

[0167] (2) The importance of adaptive modeling of different time windows: The influence of different time windows on the current state is automatically learned by the model, avoiding information loss caused by manually setting weights or equal weighting.

[0168] (3) Improve the stability and accuracy of state determination: It has a natural ability to suppress instantaneous abnormal windows, reducing the interference of noise or occasional fluctuations on the final determination result.

[0169] S106 uses fusion vectors within several consecutive time windows to train a classifier based on a time attention mechanism, thus obtaining a human state prediction model.

[0170] This module is used to perform unified end-to-end training and parameter optimization for the aforementioned multimodal feature engineering module, modality-specific vectorized encoding module, multimodal fusion module, and temporal state determination module.

[0171] Its core objective is to enable the system to learn the mapping relationship between human comfort states (focus / fatigue / stress) from multimodal environments and sensory data through supervised learning. Furthermore, it continuously optimizes the parameters of each module through a backpropagation mechanism, thereby improving the accuracy and stability of overall state determination.

[0172] See Figure 4 In some examples, step S106 includes: Step 1: Construct training samples.

[0173] The first step is to construct the input sample. ) During the training phase, the system uses multimodal data collected within a continuous time window as a basis and completes the process sequentially according to the aforementioned modules: Preprocessing and feature engineering of raw environmental and sensory data; vectorization encoding of features of each modality; construction of multimodal fusion vectors; organization of multi-time window fusion vector sequences.

[0174] Finally, the input representation for each training sample is:

[0175] In the formula: Indicates the first The multimodal fusion vector corresponding to each time window ; This indicates the number of historical windows used for status determination.

[0176] Step 2: Construct training labels (supervisory signals) )).

[0177] Each input sample corresponds to a specific human comfort state label: The labels can be defined as follows: 0 represents a focused state, 1 represents a fatigued state, and 2 represents a stressful state. These labels can be obtained from experimental task design, participant self-labeling scales, or expert labeling results.

[0178] Step 3: Construct the loss function.

[0179] For the three-class human comfort state determination task, this module uses the multi-class cross-entropy loss function as the training objective function.

[0180] Assume the model is effective for the input samples. The predicted probability output is: .

[0181] The one-hot representation corresponding to the real label is: .

[0182] The loss function for a single sample is then defined as:

[0183] This loss function is used to measure the deviation between the model's predictions and the actual state.

[0184] Step 4: Parameter update and end-to-end training mechanism.

[0185] During training, the system employs an end-to-end backpropagation mechanism to jointly update the trainable parameters in the following modules: (1) Parameters of the multilayer perceptron in the modality-specific vectorized coding module; (2) Linear mapping and attention-related parameters in the multimodal fusion module; (3) Attention parameters and classifier parameters in the temporal state determination module.

[0186] Specifically, after completing forward propagation and calculating the loss function, the system will use the backpropagation algorithm to calculate the gradient of the loss function with respect to the parameters of each module. Then, the parameters are updated according to the gradient descent strategy, allowing the model to gradually converge to the optimal state.

[0187] The overall training process for this module includes the following steps: Step 1: Construct a training sample set ; The second step is to perform forward computation on the input sample to obtain the predicted state probability. The third step is to calculate the loss function based on the prediction results and the true labels. Step 4: Calculate the gradient through backpropagation; Step 5: Update the parameters of each module; Step 6: Repeat the above steps until training converges or the preset number of iterations is reached.

[0188] In one feasible embodiment, the system employs the following training and model parameter configuration: (1) Vectorization encoding module parameters Each modality's vectorized encoding employs a multilayer perceptron (MLP) structure; MLP layer count: 2–3 layers; Number of neurons per layer: 64 or 128; Activation function: ReLU; Output vector dimension: (d=64).

[0189] (2) Parameters of multimodal fusion module Number of input modalities: environmental modalities and sensory modalities; The dimension of the fused vector is 64. Number of fusion layers: 1 linear mapping layer + attention weighting.

[0190] (3) Parameters of the timing state determination module Input the number of time windows: (l=5 / 10); Time-attention layer: 1 layer; Number of classifier layers: 1–2 fully connected layers; Number of output categories: 3.

[0191] (4) Training strategy parameters Optimizer: Adam optimization algorithm; Initial learning rate: (1 × 10^{-3}); Batch size: 32 or 64; Training epochs: 50–100; Learning rate strategy: A fixed learning rate or a rate that decays with each training round can be used.

[0192] The above training process design achieved the following technical effects: (1) Achieve joint optimization of parameters of multiple modules: enable each functional module to learn collaboratively under a unified objective function.

[0193] (2) Make full use of temporal and multimodal information: During the training process, multiple time windows and multimodal features are considered at the same time to improve the accuracy of state determination.

[0194] (3) Stable training process: Mature optimization algorithms and standard loss functions are adopted to reduce the difficulty of engineering implementation and parameter tuning.

[0195] (4) Adapt to different experiments and scenarios: By adjusting the number of windows, vector dimensions and network layers, it can be flexibly adapted to different devices and computing power conditions.

[0196] Figure 5 A structural block diagram of an office building work status prediction device based on multimodal data, provided as an embodiment of this application. See also... Figure 5 ,include: Data acquisition and synchronization module 11 is used to collect objective environmental data and human perception data to form a raw dataset; The preprocessing and feature engineering module 12 is used to preprocess the original dataset and construct features to form a dataset; Modality-specific encoding and vectorization module 13 is used to encode multi-source data in the dataset to obtain modality vectors of multiple different modalities; The multimodal fusion module 14 is used to fuse modal vectors of multiple different modes to obtain a fused vector; The human state prediction module 15 is used to establish a classifier based on a time attention mechanism; and to train the classifier based on a time attention mechanism using fusion vectors within several consecutive time windows to obtain a human state prediction model.

[0197] Figure 6 This is a structural block diagram of an electronic device provided according to an embodiment of this application. See also... Figure 6 Electronic devices may include Figure 5 The aforementioned office building operational status prediction device is based on multimodal data. Typically, the electronic device includes a processor 21 and a memory 22. The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. The memory 22 may include one or more computer-readable storage media, which may be non-transitory. The memory 22 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in memory 22 is used to store at least one instruction, which is executed by processor 21 to implement the office building working status prediction method based on multimodal data provided by an electronic device in the method embodiments of this application.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting the working status of office buildings based on multimodal data, characterized in that, include: Collect objective environmental data and human perception data to form a raw dataset; The original dataset is preprocessed and features are constructed to form a new dataset; Encode the multi-source data in the dataset to obtain modality vectors of multiple different modalities; The modal vectors of multiple different modes are fused to obtain a fused vector; Develop a classifier based on the time attention mechanism; A human state prediction model is obtained by training a classifier based on a time attention mechanism using fusion vectors within several consecutive time windows.

2. The method for predicting the working status of office buildings based on multimodal data according to claim 1, characterized in that, After collecting objective environmental data and crowd perception data, the objective environmental data and crowd perception data are time-synchronized and windowed to form the original dataset.

3. The method for predicting the working status of office buildings based on multimodal data according to claim 1, characterized in that, The steps for preprocessing and feature construction of the original dataset include: Environmental modality features and perceptual modality features are extracted from the original dataset. The features extracted for environmental modality include one or more of the following: temperature-related features, humidity features, wind speed features, and air quality-related features. The features extracted for perceptual modality include one or more of the following: electroencephalogram (EEG) signal features, time-domain perceptual features, skin conductance features, and subjective feedback features. The extracted features are then normalized.

4. The method for predicting the working status of office buildings based on multimodal data according to claim 1, characterized in that, The steps for encoding multi-source data in a dataset include: All feature variables within the same time window and under the same modality are concatenated into a one-dimensional feature vector in a predefined order; The input one-dimensional feature vector is mapped layer by layer through a multi-layer fully connected network, and a non-linear activation function is introduced to enhance the representation capability. One-dimensional feature vectors of different lengths and semantics are uniformly mapped to modal vectors of fixed dimensions, thereby obtaining modal vectors of multiple different modalities.

5. The method for predicting the working status of office buildings based on multimodal data according to claim 1, characterized in that, The steps for fusing mode vectors from multiple different modes include: Map each modality vector to a unified attention computation space; Calculate the correlation between each modality and the current state query vector; The degree of relevance is normalized into attention weights; The weights are used to sum the modal vectors to obtain the fusion vector.

6. The method for predicting the working status of office buildings based on multimodal data according to claim 1, characterized in that, The time-based attention mechanism is as follows: Calculate the importance score of the fusion vector and its current state within several time windows; The temporal attention weights are derived based on the importance of the fusion vectors within several time windows. Based on temporal attention weights, the fusion vectors within several time windows are weighted and summed to obtain the temporal context representation at the current moment.

7. The method for predicting the working status of office buildings based on multimodal data according to claim 1, characterized in that, The steps for training a classifier based on a time attention mechanism using fused vectors from several consecutive time windows include: Construct a training sample set ; Perform forward computation on the input sample to obtain the predicted state probability; Calculate the loss function based on the prediction results and the true labels; The gradient is calculated through backpropagation; Update the parameters of each module; Repeat the above steps until training converges or the preset number of iterations is reached.

8. A device for predicting the working status of office buildings based on multimodal data, characterized in that, include: The data acquisition and synchronization module is used to collect objective environmental data and human perception data to form a raw dataset. The preprocessing and feature engineering module is used to preprocess the original dataset and construct features to form a new dataset. The modality-specific encoding and vectorization module is used to encode multi-source data in a dataset to obtain modality vectors of multiple different modalities; The multimodal fusion module is used to fuse modal vectors of multiple different modalities to obtain a fused vector; The human state prediction module is used to build a classifier based on a time attention mechanism; and to train the classifier based on a time attention mechanism using fusion vectors within several consecutive time windows to obtain a human state prediction model.

9. An electronic device, characterized in that, Includes the office building working status prediction device based on multimodal data as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is executed by a processor to implement the office building working status prediction method based on multimodal data as described in any one of claims 1 to 7.

Images