A multi-modal data fusion analysis and judgment hotel first-aid method and system

Abstract

This invention discloses a hotel emergency rescue method and system based on multimodal data fusion analysis and judgment, relating to the field of general control systems or methods. The method includes the following steps: generating voice feature vectors and heart rate feature vectors from voice and heart rate signals collected from the target room and adjacent rooms; constructing a multi-level adjacent room network based on topological relationship data to generate topological feature vectors; fusing the three feature vectors, analyzing the synchronicity and spatial distribution characteristics of signal mutations between the target room and adjacent rooms; when the correlation assessment result of the signal mutation is an isolated event, adjusting the weight coefficients of the fused feature vectors and inputting them into a pre-spatiotemporal fusion network model to determine whether the target room is in an emergency situation. This invention, based on multimodal collaborative perception, dynamically adjusts the weight coefficients of the fused feature vectors through the analysis of signal mutations and inputs them into a spatiotemporal fusion network for decision-making, significantly improving the accuracy and reliability of emergency situation identification.

Description

Technical Field

[0001] This invention relates to the field of general control systems or methods, specifically to a hotel emergency rescue method and system for multimodal data fusion analysis and judgment. Background Technology

[0002] In the field of modern hotel management and security, ensuring the safety of guests is a crucial task, especially the ability to respond quickly to sudden health crises or emergencies, which directly relates to the hotel's service quality and social responsibility. However, current emergency monitoring technologies in hotel settings generally suffer from serious limitations. Mainstream solutions rely on heart rate or voice sensors in a single room for anomaly identification, but in practice, sudden noises such as television stereos and falling objects share similar spectral characteristics with genuine emergency signals (such as falls, impacts, and cries for help), leading to persistently high false alarm rates. More critically, the dense room layout of hotels causes cross-transmission of signals between adjacent spaces—an emergency signal from a target room may be misinterpreted as an event in the next room, and the system cannot pinpoint the true source of risk when multiple people experience anomalies simultaneously. Existing technologies neither effectively utilize the topological relationships of hotel rooms to construct a spatial perception model nor possess a collaborative analysis mechanism for cross-room signal mutations, causing a sharp drop in system reliability under complex interference environments. This passive, single-point monitoring architecture has become an industry-wide bottleneck restricting the accuracy of hotel emergency response. Summary of the Invention

[0003] The purpose of this invention is to provide a hotel emergency rescue method and system for multimodal data fusion analysis and judgment, which aims to overcome the above-mentioned problems existing in the prior art.

[0004] To achieve the objective, the present invention provides the following technical solution:

[0005] A hotel emergency rescue method based on multimodal data fusion analysis and judgment includes the following steps:

[0006] Step 1: Collect voice and heart rate signals from the target room and its adjacent rooms, and simultaneously obtain topological relationship data of the target room and its adjacent rooms from the hotel management system;

[0007] Step 2: Perform preprocessing and framing operations on the speech signal, extract features using the short-time Fourier transform method, and generate speech feature vectors representing the speech content;

[0008] Step 3: Perform window segmentation and wavelet transform processing on the heart rate signal, and extract QRS complex feature points from the heart rate signal using the empirical mode decomposition method to generate a heart rate feature vector representing vital signs.

[0009] Step 4: Based on the topological relationship data, construct a multi-level neighboring room network centered on the target room, and generate a topological feature vector representing spatial correlation;

[0010] Step 5: Normalize the generated speech feature vector, heart rate feature vector, and topological feature vector, and concatenate them into a unified fusion feature vector;

[0011] Step 6: Analyze the temporal synchronization and spatial distribution characteristics of signal abrupt changes between the target room and adjacent rooms based on the speech features and heart rate features contained in the fused feature vector, and generate signal abrupt change correlation assessment results.

[0012] Step 7: If the correlation assessment result of signal mutation is lower than the preset threshold, and the signal mutation of the target room is not significantly related to other rooms in terms of spatial distribution, then adjust the weight coefficients of each feature of the fused feature vector.

[0013] Step 8: Input the fused feature vector and the weight coefficients of each feature into the pre-trained spatiotemporal fusion network model, perform comprehensive analysis and processing, and generate a judgment result on whether the target room is in an emergency situation.

[0014] Furthermore, it also includes step 9: if the generated judgment result indicates that the target room is in an emergency situation, then the shortest path from the target room to the rescue location is calculated based on the topological relationship data, triggering the hotel's internal linkage alarm mechanism and generating an alarm signal.

[0015] Furthermore, the heart rate signal in step 1 is acquired via non-contact millimeter-wave radar, specifically including:

[0016] Step 1.1: Transmit frequency-modulated continuous wave millimeter wave signals to the target room and adjacent rooms;

[0017] Step 1.2: After receiving the reflected signal, adaptive beamforming technology is used to separate the reflected signal components from different rooms;

[0018] Step 1.3: For the reflected signal components of the target room, the chest cavity motion waveform is analyzed using a micro-Doppler feature extraction algorithm to generate a heart rate signal;

[0019] Step 1.4: Dynamically suppress cross-room signal crosstalk based on the energy intensity of signal components in adjacent rooms.

[0020] Furthermore, step 1.4 specifically includes the following steps:

[0021] Step 1.4.1: Calculate the reflected signal components of the target room in real time. Reflected signal components from adjacent rooms Energy ratio:

[0022]

[0023] Where N is the total number of topologically adjacent rooms;

[0024] Step 1.4.2: Determine the number of rooms When any one At that time, an adaptive notch filter is activated for room i, and the number of rooms is updated. The adaptive notch filter is:

[0025]

[0026] in, The preset energy leakage threshold is used; the filter parameters are dynamically configured as follows:

[0027] Center frequency locking Main frequency component; attenuation factor Controlling stopband depth; bandwidth factor Control and suppress bandwidth;

[0028] Step 1.4.3: Coherently accumulate the filtered signal with the original signal from the target room:

[0029]

[0030] in, , where is the cumulative weighting coefficient; M is the coefficient that satisfies The number of rooms.

[0031] Furthermore, in step 4, when constructing a multi-level adjacent room network, the hotel building information model (BIM) database of the hotel management system is accessed simultaneously to extract the acoustic attenuation coefficient of the walls and spatial geometric structure data, and generate a weighted topology map that integrates physical propagation characteristics; then, a topology feature vector is generated based on the weighted topology map; each dimension of the topology feature vector includes: the signal propagation path loss value between room nodes, and the spatial correlation strength factor based on historical emergency event data.

[0032] Furthermore, the aforementioned weighted topology graph Among them, vertex set Room nodes (target room + adjacent rooms); edge set For room connectivity, edge weights ;in, The acoustic attenuation coefficient of the walls of the target room and adjacent rooms; The straight-line distance between the target room and adjacent rooms. Correlate the frequency of historical emergency medical events in the target room and adjacent rooms;

[0033] Input the weighted topology graph into a graph neural network (GNN) and perform the following operations to generate topology feature vectors:

[0034] Aggregate neighborhood weights through two-layer graph convolution operations:

[0035]

[0036] in, For the target room node, For adjacent room nodes, For the target room node The neighborhood group, To the target room node The total number of adjacent rooms; The edge weight represents the target room node. Nodes with neighboring rooms The strength of the correlation between them; For the first Layer feature vectors, The initial characteristics of adjacent room nodes; The aggregated output features are used as input to the next layer or as the final features; The activation function is ReLU: ;

[0037] Perform hierarchical aggregation on the target room node:

[0038] First convolutional layer output: That is, aggregating the features of adjacent rooms at the same degree;

[0039] Second convolutional output: That is, aggregating the features of second-degree adjacent rooms;

[0040] Embedding vectors of target room nodes Learnable pooling is performed to output a 32-dimensional topological feature vector. The topological feature vector After normalization, it is finally used in step 6 to calculate the path loss compensation factor in the spatial distribution characteristics.

[0041] Furthermore, in step 5, a normalization processing modal adaptive scaling algorithm is adopted, including: applying max-min normalization to the speech feature vector; applying max-min normalization to the heart rate feature vector; and applying graph embedding normalization to the topological feature vector.

[0042] Furthermore, in step 8, the spatiotemporal fusion network model includes a two-stream adversarial training architecture:

[0043] The spatial feature processing flow is a three-level graph convolutional neural network hierarchical structure. Each level performs a neighborhood node aggregation operation to process topological feature vectors.

[0044] The temporal feature processing flow is a five-level dilated causal convolutional layer stacked structure used to process signal sequences;

[0045] Adversarial discriminators are used to constrain the consistency of dual-stream feature distributions through KL divergence.

[0046] The decision-making level generates emergency status assessment results based on a fully connected network.

[0047] A hotel emergency rescue system based on multimodal data fusion analysis and judgment, used to implement any of the methods described above; the system includes the following modules:

[0048] The acquisition module, deployed in the target room and adjacent rooms, includes a voice acquisition unit and a millimeter-wave radar unit, used to acquire voice signals and heart rate signals in real time.

[0049] The hotel management system interface module is used to retrieve the topological relationship data between the target room and its adjacent rooms.

[0050] The multimodal feature extraction module includes:

[0051] The speech processing unit is used to perform framing and short-time Fourier transform on the speech signal to generate speech feature vectors;

[0052] The heart rate analysis unit is used to extract QRS complex feature points through wavelet transform and empirical mode decomposition to generate heart rate feature vectors.

[0053] The topology modeling unit is used to construct a multi-level adjacent room network based on topological relationship data and generate a topological feature vector.

[0054] The feature fusion module is used to normalize the three types of feature vectors and concatenate them into a fused feature vector;

[0055] The dynamic weight adjustment module is used to analyze the spatiotemporal synchronicity of signal mutations. When the correlation between the signal mutation in the target room and the adjacent rooms is lower than the threshold, the weight coefficients of each feature vector in the fused feature vector are automatically adjusted.

[0056] The spatiotemporal fusion decision module integrates a pre-trained spatiotemporal fusion network model, receives the fusion feature vector and the weight coefficients of each feature vector, and generates the emergency status judgment result.

[0057] Furthermore, the system also includes an emergency response module, which is used to respond to the emergency status assessment results and perform route planning, linkage alarm and emergency notification operations.

[0058] Furthermore, the system includes a gateway, a voice display screen, and a vital signs detector; the voice display screen is used to collect and process voice signals, the vital signs detector is used to collect and analyze heart rate signals, and the gateway is used to generate a topological feature vector and fuse it with the voice feature vector and heart rate feature vector. Based on the fused feature vector and its weight coefficients, an emergency status assessment result is generated.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] Firstly, this invention innovatively constructs a multimodal collaborative perception architecture by simultaneously collecting voice and heart rate signals from the target room and adjacent rooms, and fusing topological relationship data of hotel rooms. Based on the analysis of the temporal synchronization and spatial distribution isolation of signal mutations, the weight coefficients of each feature in the fused feature vector are dynamically adjusted. The normalized feature vector and dynamic weight parameters are then input into a spatiotemporal fusion network for decision-making. This solution effectively overcomes key challenges in hotel scenarios such as noise transmission between adjacent rooms and confusion caused by sudden interference, significantly reducing the false alarm rate in strong interference environments and greatly improving the accuracy and reliability of emergency status recognition.

[0061] Secondly, this invention employs millimeter-wave radar, utilizing non-contact detection of frequency modulated continuous wave (FMCW) millimeter-wave signals. It innovatively integrates beam control from radar engineering, acoustic characteristics of hotel buildings, and physiological signal processing to form a non-contact heart rate monitoring paradigm for complex scenarios involving multiple hotel rooms. This allows for the acquisition of purer heart rate signals from the target rooms, thus providing a solid foundation for accurately analyzing and judging emergency signals in the target rooms.

[0062] Thirdly, this invention transforms the inherent physical characteristics of hotel buildings into dynamically calculable topological feature vectors, breaking through the limitation of traditional emergency rescue systems where topological relationships are merely logical connections. By deeply integrating acoustic attenuation coefficients, spatial geometric data, and historical behavioral patterns from Building Information Modeling (BIM), it for the first time constructs a weighted topological graph that integrates physical propagation loss and behavioral correlation strength, accurately depicting the true attenuation law of signals in the complex wall structure of a hotel. More importantly, it innovatively employs graph neural networks to perform neighborhood aggregation calculations on the weighted topology: through two layers of graph convolution operations, the feature vector of the target room... It not only aggregates the direct signal influence of first-degree adjacent rooms, but also captures the indirect propagation effect of second-degree adjacent rooms, and its denominator term This innovative approach solves the problem of neighbor number discrepancies caused by star-shaped / chain-shaped heterogeneous hotel layouts. The final generated 32-dimensional topological feature vector... As a spatial propagation compensation factor, it is input into the subsequent analysis module to reduce the false positive rate. Attached Figure Description

[0063] Figure 1 This is a flowchart of the method in this invention.

[0064] Figure 2 The structural frame of the system in this invention Figure 1 .

[0065] Figure 3 The structural frame of the system in this invention Figure 2 . Detailed Implementation

[0066] Specific embodiments of the present invention will now be described with reference to the accompanying drawings. Many details are described below to provide a comprehensive understanding of the invention; however, those skilled in the art will be able to implement the invention without these details.

[0067] like Figure 1 As shown, a hotel emergency rescue method based on multimodal data fusion analysis and judgment includes the following steps:

[0068] Step 1: Collect voice and heart rate signals from the target room and its adjacent rooms, and simultaneously obtain the topological relationship data of the target room and its adjacent rooms from the hotel management system.

[0069] In one specific embodiment, the heart rate signal is acquired via non-contact millimeter-wave radar, specifically including:

[0070] Step 1.1: Transmit frequency-modulated continuous wave millimeter wave signals to the target room and adjacent rooms.

[0071] Step 1.2: After receiving the reflected signal, adaptive beamforming technology is used to separate the reflected signal components of different rooms.

[0072] Step 1.3: For the reflected signal component of the target room, the chest cavity motion waveform is analyzed using a micro-Doppler feature extraction algorithm to generate a heart rate signal.

[0073] Step 1.4: Dynamically suppress cross-room signal crosstalk based on the energy intensity of signal components in adjacent rooms.

[0074] Preferably, step 1.4 above specifically includes the following steps:

[0075] Step 1.4.1: Calculate the reflected signal components of the target room in real time. Reflected signal components from adjacent rooms Energy ratio:

[0076]

[0077] Where N is the total number of topologically adjacent rooms.

[0078] Step 1.4.2: Determine the number of rooms When any one At that time, an adaptive notch filter is activated for room i, and the number of rooms is updated. The adaptive notch filter is:

[0079]

[0080] in, The preset energy leakage threshold is used; the filter parameters are dynamically configured as follows:

[0081] Center frequency locking Main frequency component; attenuation factor Controlling stopband depth; bandwidth factor Control and suppress bandwidth.

[0082] Step 1.4.3: Coherently accumulate the filtered signal with the original signal from the target room:

[0083]

[0084] in, , where is the cumulative weighting coefficient; M is the coefficient that satisfies The number of rooms.

[0085] It employs millimeter-wave radar, using non-contact detection with frequency-modulated continuous wave (FMCW) millimeter-wave signals to capture millimeter-level micro-motions of chest cavity movement in hotel rooms. It innovatively integrates radar signal processing technology with the structural characteristics of the hotel building; adaptive beamforming technology precisely separates reflected signals from the target room and adjacent rooms through spatial filtering, essentially constructing a virtual signal cage with physical walls as a natural barrier. Simultaneously, a dynamic crosstalk suppression mechanism is employed, based on real-time calculations using the energy ratio formula. This allows the system to intelligently identify the differences in interference caused by concrete walls (high attenuation) and lightweight partitions (low attenuation). When energy leakage exceeds a threshold, a parameterized notch filter, based on a mathematical mapping relationship... The stopband depth is dynamically adjusted. Finally, a coherent accumulation compensation algorithm is used to reconstruct a pure heart rate signal, reducing cross-room crosstalk errors while preserving the original physiological characteristics. It is evident that the specific technical solution in step 1 above cross-integrates three independent technical domains in radar engineering—beam control, hotel building acoustics, and physiological signal processing—to form a non-contact heart rate monitoring paradigm for hotel scenarios. This allows for the acquisition of purer heart rate signals from the target room, providing a solid foundation for accurately analyzing and judging emergency signals in the target room.

[0086] Step 2: Perform preprocessing and framing operations on the speech signal, extract features using the short-time Fourier transform method, and generate speech feature vectors that represent the speech content.

[0087] In one specific embodiment, step 2 above specifically includes the following steps:

[0088] Step 2.1: Enhance the speech signal by eliminating noise and channel distortion, thus completing the preprocessing of the speech signal. The main purpose of this step is to eliminate interference from fixed noise sources in the hotel, such as television sound and air conditioning noise.

[0089] Step 2.2: Using a frame length of 30ms, a frame shift of 10ms, and a Hanning window function, perform frame segmentation on the preprocessed speech signal to obtain framed speech data, mathematically represented as:

[0090]

[0091] in, For frame number, The number of frame-shifted samples. Number of samples per frame length This is the Hanning window function. This step enables the precise capture of sudden cries for help (such as short-duration "help" pulses).

[0092] Step 2.3: Perform a short-time Fourier transform on the segmented speech data:

[0093] First, STFT calculation is performed on the framed speech data to obtain the complex spectrum of the m-th frame. :

[0094]

[0095] in, The number of FFT points covers a 30ms frame length;

[0096] Secondly, the complex spectrum of the m-th frame Perform power spectrum calculation:

[0097]

[0098] Then, the power spectrum is used to generate log-Melbourne spectrum features through a 40-dimensional Melbourne filter bank:

[0099]

[0100] in, Let be the triangular window function of the i-th Mel filter;

[0101] Next, calculate the dynamic difference spectrum components:

[0102]

[0103] The final feature vector, i.e., the speech feature vector, is obtained:

[0104] .

[0105] In this step, the STFT power spectrum converts the time-domain signal into a joint time-frequency representation, simultaneously capturing the semantics of the distress call (frequency domain) and changes in sound intensity (time domain). A Mel filter bank simulates human hearing characteristics, focusing on the sensitive frequency bands of human voices and suppressing irrelevant high-frequency noise. Dynamic differential calculation enhances the vibrato features of painful groans, improving the recognition rate of these sounds. This step converts the original speech into a task-specific low-dimensional feature vector (Mel spectrum + dynamic differential), retaining only the acoustic patterns required for distress call detection (such as changes in sound intensity and vibrato), discarding phase information and full-frequency details that could reconstruct semantics. This makes the original dialogue difficult to reverse engineer, preventing information leakage. Preferably, the entire process of speech acquisition and feature extraction is completed on a local device, ensuring that the original audio does not leave the device; only encrypted feature vectors are uploaded, completely blocking leakage paths and further preventing information leakage.

[0106] Step 3: Perform window segmentation and wavelet transform processing on the heart rate signal, and extract QRS complex feature points from the heart rate signal using the Empirical Mode Decomposition (EMD) method to generate a heart rate feature vector representing vital signs.

[0107] In one specific embodiment, the heart rate signal is windowed and processed using wavelet transform. Specifically, the heart rate signal is windowed with a window length of 5-10 seconds to ensure coverage of 3-5 complete heartbeat cycles. Then, wavelet transform processing is performed using the db6 wavelet basis to match the QRS complex morphology. The final heart rate feature vector includes, but is not limited to, the standard deviation of the RR interval. and QRS group width .

[0108] Step 4: Based on the topological relationship data, construct a multi-level neighboring room network centered on the target room, and generate a topological feature vector representing spatial correlation.

[0109] In one specific embodiment, when constructing the multi-level adjacent room network in step 4 above, the hotel building information model (BIM) database is accessed simultaneously to extract wall acoustic attenuation coefficients and spatial geometric structure data, generating a weighted topology map that integrates physical propagation characteristics; then, a topology feature vector is generated based on the weighted topology map; each dimension of the topology feature vector includes:

[0110] (a) Signal propagation path loss between room nodes;

[0111] (b) Spatial correlation strength factor based on historical emergency rescue event data.

[0112] As a preferred option, the following steps are included:

[0113] Step 4.1: When constructing a multi-level adjacent room network, simultaneously access the hotel's Building Information Modeling (BIM) database, extract wall acoustic attenuation coefficients and spatial geometric structure data, and generate a weighted topology map that incorporates physical propagation characteristics. Among them, vertex set For room nodes, there is the target room node and its neighboring room nodes; edge set For room connectivity, edge weights ;in, The acoustic attenuation coefficient of the walls of the target room and adjacent rooms; The straight-line distance between the target room and adjacent rooms. Correlate the frequency of historical emergency medical events in the target room and adjacent rooms;

[0114] Step 4.2: Input the weighted topology graph above into a graph neural network (GNN) and perform the following operations to generate topological feature vectors:

[0115] Step 4.2.1: Aggregate neighborhood weights through two-layer graph convolution operations:

[0116]

[0117] in, For the target room node, For adjacent room nodes, For the target room node The neighborhood group, To the target room node The total number of adjacent rooms; The edge weight represents the target room node. Nodes with neighboring rooms The strength of the correlation between them; For the first Layer feature vectors, The initial characteristics of adjacent room nodes; The aggregated output features are used as input to the next layer or as the final features; The activation function is ReLU: .

[0118] Step 4.2.2: Perform hierarchical aggregation on the target room node:

[0119] First convolutional layer output: That is, aggregating the features of adjacent rooms at the same degree;

[0120] Second convolutional output: That is, aggregating the features of second-degree adjacent rooms.

[0121] Step 4.2.3: Embed vectors for the target room nodes Learnable pooling is performed to output a 32-dimensional topological feature vector. The topological feature vector After normalization, it is finally used in step 6 to calculate the path loss compensation factor in the spatial distribution characteristics.

[0122] This step transforms the inherent physical characteristics of the hotel building into dynamically calculable topological feature vectors, overcoming the limitation of traditional emergency systems where topological relationships are merely logical connections. By deeply integrating acoustic attenuation coefficients, spatial geometric data, and historical behavioral patterns from Building Information Modeling (BIM), it constructs a weighted topological graph that integrates physical propagation loss and behavioral correlation strength, accurately depicting the true attenuation patterns of signals within the complex wall structure of the hotel. Innovatively, a graph neural network is used to perform neighborhood aggregation calculations on the weighted topology: through two layers of graph convolution operations, the feature vectors of the target rooms... It not only aggregates the direct signal influence of first-degree adjacent rooms, but also captures the indirect propagation effect of second-degree adjacent rooms, and its denominator term This innovative approach solves the problem of neighbor number discrepancies caused by star-shaped / chain-shaped heterogeneous hotel layouts. The final generated 32-dimensional topological feature vector... As a spatial propagation compensation factor, it is input into the subsequent analysis module to reduce the false positive rate.

[0123] Step 5: Normalize the generated speech feature vector, heart rate feature vector, and topological feature vector, and concatenate them into a unified fusion feature vector.

[0124] In one specific embodiment, step 5 employs a normalization processing modal adaptive scaling algorithm, including: applying maximum-minimum normalization to the speech feature vector; applying maximum-minimum normalization to the heart rate feature vector; and applying maximum-minimum normalization to the topological feature vector. Graph embedding normalization is used. Finally, the normalized speech features, heart rate features, and topological features are combined to form a new fused feature vector.

[0125] Step 6: Analyze the temporal synchronization and spatial distribution characteristics of signal abrupt changes between the target room and adjacent rooms based on the speech and heart rate features contained in the fused feature vector, and generate a signal abrupt change correlation assessment result.

[0126] In one specific implementation, step 6 includes the following steps:

[0127] Step 6.1: Identify and mark speech abrupt change points based on the following conditions:

[0128]

[0129] in, This refers to dynamic threshold coefficients, including but not limited to daytime (e.g., from 7 AM to 7 PM). The value is 3.5, at night (e.g., from 7 PM to 7 AM). Reduced to 2.5; This represents the baseline standard deviation of the speech dynamic difference spectrum.

[0130] Step 6.2: Identify and mark heart rate inflection points based on the following conditions:

[0131]

[0132] in, This represents the standard deviation of the RR interval (interval between adjacent heartbeats); This refers to the QRS complex width (ventricular depolarization time).

[0133] Step 6.3: The formula for measuring the synchronicity between the target room (target) and its neighboring room (j) is as follows:

[0134]

[0135] in, The time decay constant, For the maximum physiological response delay, including but not limited to s and s; The indicator function (set to zero if timeout occurs, i.e., if the condition is not met) is used. (Time indicator function set to zero);

[0136] Using the above synchronization measurement formula, calculate the synchronization measure of the speech signals between the target room (target) and the adjacent room (j). and heart rate signal synchronicity measurement .

[0137] Step 6.4: Calculate the joint synchronicity of the speech signal and the heart rate signal:

[0138]

[0139] in, For speech weights, including but not limited to .

[0140] Step 6.5: Using topological feature vectors Calculate the path loss between the target room (target) and its neighboring room (j):

[0141]

[0142] like If , it indicates a concrete wall (high attenuation); if This indicates a lightweight partition wall (low attenuation).

[0143] Computational space correction synchronization:

[0144]

[0145] in, Building attenuation factor, including but not limited to .

[0146] Based on steps 6.1-6.5, the spatial correction synchronization between the target room and each adjacent room is obtained: .

[0147] Step 6.6: Calculate the time dimension score:

[0148] ;

[0149] Calculate the spatial dimension score:

[0150]

[0151] Step 6.7: Generate the signal mutation correlation assessment results, specifically:

[0152] like If it is an isolated event, then it is judged as an isolated event; if If, then it is determined to be a related event; if If so, it is judged as a propagation event across multiple rooms.

[0153] Step 7: If the signal mutation correlation assessment result is lower than the preset threshold, and the signal mutation in the target room is not significantly related to other rooms in terms of spatial distribution (i.e., the signal mutation correlation assessment result is an isolated event), then adjust the weight coefficients of each feature in the fused feature vector. If the signal mutation correlation assessment result is a related event or a propagation event, then save the default weight coefficients.

[0154] In one specific embodiment, the basic weight allocation (i.e., the default weight coefficient) is as follows:

[0155] The weighting coefficient for the speech features is 0.4;

[0156] The weighting coefficient for heart rate features is 0.5;

[0157] The weighting coefficient for topological features is 0.1.

[0158] When the correlation assessment result of the signal mutation is an isolated event, the weighting coefficients are adjusted as follows:

[0159] Weighting coefficients of speech features for ;

[0160] Weighting coefficients of heart rate features for ;

[0161] Weight coefficients of topological features It is 0.1;

[0162] in, , is the weight decay factor for speech features. , which is the weighting enhancement factor for heart rate characteristics;

[0163] Weighting coefficients for speech features Weighting coefficients of heart rate characteristics Weight coefficients of topological features Normalization is performed:

[0164]

[0165] Step 8: Input the fused feature vector and the weight coefficients of each feature into the pre-trained spatiotemporal fusion network model, perform comprehensive analysis and processing, and generate a judgment result on whether the target room is in an emergency situation.

[0166] In one specific embodiment, step 8 specifically includes the following steps:

[0167] Step 8.1: Construct the weighted feature vector:

[0168]

[0169] Step 8.2: Input the weighted feature vector into the pre-trained spatiotemporal fusion network model for spatiotemporal fusion network processing to generate a judgment result on whether the target room is in an emergency situation. The judgment result is divided into two types: emergency situation and non-emergency situation. The spatiotemporal fusion network model includes a two-stream adversarial training architecture, as detailed below:

[0170]

[0171] The spatial feature processing flow employs a three-level graph convolutional neural network hierarchy, with each level performing a neighborhood node aggregation operation to process topological feature vectors: the first level aggregates directly adjacent rooms, the second level expands to rooms on the same floor, and the third level incorporates vertically adjacent rooms.

[0172] The temporal feature processing flow is a five-level dilated causal convolutional layer stacked structure used to process signal sequences;

[0173] Adversarial discriminators are used to constrain the consistency of dual-stream feature distributions through KL divergence.

[0174] The decision layer generates emergency status judgment results based on a fully connected network. The judgment results are divided into emergency status and non-emergency status. The decision layer includes, but is not limited to, using the Softmax function to transform the feature vector into an emergency status probability distribution. When the probability of the emergency category exceeds a preset threshold, the output result is an emergency status; otherwise, the output result is a non-emergency status.

[0175] Step 9: If the generated judgment result indicates that the target room is in an emergency situation, calculate the shortest path from the target room to the rescue location based on the topology data, trigger the hotel's internal linkage alarm mechanism, and generate an alarm signal; send an emergency notification to relevant areas through the hotel's communication system, and record the event log for subsequent optimization processing.

[0176] like Figure 1 and Figure 2 As shown, this invention also discloses a hotel emergency rescue system for multimodal data fusion analysis and judgment, which is used to implement the method described above and includes the following modules:

[0177] The acquisition module, deployed in the target room and adjacent rooms, includes a voice acquisition unit and a millimeter-wave radar unit, used to acquire voice signals and heart rate signals in real time.

[0178] The hotel management system interface module is configured to retrieve topological relationship data between the target room and its adjacent rooms;

[0179] The multimodal feature extraction module includes a speech processing unit, a heart rate analysis unit, and a topology modeling unit. The speech processing unit performs framing and short-time Fourier transform on the speech signal to generate speech feature vectors; the heart rate analysis unit extracts QRS group feature points through wavelet transform and empirical mode decomposition to generate heart rate feature vectors; and the topology modeling unit constructs a multi-level adjacent room network based on topological relationship data to generate topological feature vectors.

[0180] The feature fusion module is used to normalize the three types of feature vectors and concatenate them into a fused feature vector;

[0181] The dynamic weight adjustment module is used to analyze the spatiotemporal synchronicity of signal mutations. When the correlation between the signal mutation in the target room and the adjacent rooms is lower than the threshold, the weight coefficients of each feature vector in the fused feature vector are automatically adjusted.

[0182] The spatiotemporal fusion decision module integrates a pre-trained spatiotemporal fusion network model, receives the fused feature vector and the weight coefficients of each feature vector, and generates the emergency status judgment result.

[0183] The emergency response module is used to respond to the emergency status assessment results and perform route planning, alarm linkage, and emergency notification issuance operations.

[0184] like Figure 3 As shown, in one specific embodiment, the system includes a gateway 300, a voice display 100, and a vital signs detector 200. The voice display is used to collect and process voice signals, and the vital signs detector is used to collect and analyze heart rate signals. The gateway is responsible for edge computing, used to generate topological feature vectors, and fuses them with the voice feature vectors and heart rate features. Based on the fused feature vectors and their respective weight coefficients, an emergency status assessment result is generated.

[0185] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A hotel emergency rescue method based on multimodal data fusion analysis and judgment, characterized in that... This includes the following steps: Step 1: Collect voice signals and heart rate signals from the target room and its adjacent rooms, and simultaneously obtain the topological relationship data of the target room and its adjacent rooms from the hotel management system; the voice signals complete the entire process of voice acquisition and feature extraction on the local device to ensure that the original audio does not leave the device; Step 2: Perform preprocessing and framing operations on the speech signal, extract features using the short-time Fourier transform method, and generate a speech feature vector representing the speech content; the speech feature vector retains only the acoustic patterns required for emergency call detection; Step 3: Perform window segmentation and wavelet transform processing on the heart rate signal, and extract QRS complex feature points from the heart rate signal using the empirical mode decomposition method to generate a heart rate feature vector representing vital signs. Step 4: Based on the topological relationship data, construct a multi-level neighboring room network centered on the target room, and generate a topological feature vector representing spatial correlation; Step 5: Normalize the generated speech feature vector, heart rate feature vector, and topological feature vector, and concatenate them into a unified fusion feature vector; Step 6: Analyze the temporal synchronization and spatial distribution characteristics of signal abrupt changes between the target room and adjacent rooms based on the speech features and heart rate features contained in the fused feature vector, and generate signal abrupt change correlation assessment results. Step 7: If the correlation assessment result of signal mutation is lower than the preset threshold, and the signal mutation of the target room is not significantly related to other rooms in terms of spatial distribution, then adjust the weight coefficients of each feature of the fused feature vector. Step 8: Input the fused feature vector and the weight coefficients of each feature into the pre-trained spatiotemporal fusion network model, perform comprehensive analysis and processing, and generate a judgment result on whether the target room is in an emergency situation. Step 6 specifically includes the following steps: Step 6.1: Identify and mark speech abrupt change points based on the following conditions: ;in, For dynamic threshold coefficients, The baseline standard deviation of the speech dynamic difference spectrum; Step 6.2: Identify and mark heart rate inflection points based on the following conditions: ;in, The standard deviation of the RR interval; This refers to the QRS group width; Step 6.3: The formula for measuring the synchronicity between the target room (target) and its neighboring room (j) is as follows: ;in, The time decay constant, This represents the maximum physiological response delay. For indicator functions; Using the above synchronization measurement formula, calculate the synchronization measure of the speech signals between the target room (target) and the adjacent room (j). and heart rate signal synchronicity measurement ; Step 6.4: Calculate the joint synchronicity of the speech signal and the heart rate signal: ;in, For speech weights; Step 6.5: Using topological feature vectors Calculate the path loss between the target room (target) and its neighboring room (j): ;like If , then it indicates a concrete wall; if , which indicates a lightweight partition wall; Computational space correction synchronization: ;in, The building attenuation factor is used; based on steps 6.1-6.5, the spatial correction synchronization between the target room and each adjacent room is obtained: ; Step 6.6: Calculate the time dimension score: ; Calculate the spatial dimension score: ; Step 6.7: Generate the results of the signal mutation correlation assessment: like If it is an isolated event, then it is judged as an isolated event; if If, then it is determined to be a related event; if If so, it is judged as a propagation event across multiple rooms.

2. The hotel emergency rescue method based on multimodal data fusion analysis and judgment according to claim 1, characterized in that... It also includes step 9: If the generated judgment result indicates that the target room is in an emergency situation, then the shortest path from the target room to the rescue location is calculated based on the topological relationship data, triggering the hotel's internal linkage alarm mechanism and generating an alarm signal.

3. The hotel emergency rescue method based on multimodal data fusion analysis and judgment according to claim 2, characterized in that... The heart rate signal in step 1 is acquired via non-contact millimeter-wave radar, specifically including: Step 1.1: Transmit frequency-modulated continuous wave millimeter wave signals to the target room and adjacent rooms; Step 1.2: After receiving the reflected signal, adaptive beamforming technology is used to separate the reflected signal components from different rooms; Step 1.3: For the reflected signal components of the target room, the chest cavity motion waveform is analyzed using a micro-Doppler feature extraction algorithm to generate a heart rate signal; Step 1.4: Dynamically suppress cross-room signal crosstalk based on the energy intensity of signal components in adjacent rooms.

4. The hotel emergency rescue method based on multimodal data fusion analysis and judgment according to claim 3, characterized in that... Step 1.4 specifically includes the following steps: Step 1.4.1: Calculate the reflected signal components of the target room in real time. Reflected signal components from adjacent rooms Energy ratio: Where N is the total number of topologically adjacent rooms; Step 1.4.2: Determine the number of rooms When any one At that time, an adaptive notch filter is activated for room i, and the number of rooms is updated. The adaptive notch filter is: in, The preset energy leakage threshold is used; the filter parameters are dynamically configured as follows: Center frequency locking Main frequency component; attenuation factor Controlling stopband depth; bandwidth factor Control and suppress bandwidth; Step 1.4.3: Coherently accumulate the filtered signal with the original signal from the target room: in, , where is the cumulative weighting coefficient; M is the coefficient that satisfies Number of rooms The original signal for the target room, This is the filtered signal.

5. A hotel emergency rescue method based on multimodal data fusion analysis and judgment according to claim 1, characterized in that... In step 4, when constructing a multi-level adjacent room network, the hotel building information model (BIM) database is accessed simultaneously to extract the acoustic attenuation coefficient of the walls and spatial geometric structure data, and a weighted topology map integrating physical propagation characteristics is generated; then, a topology feature vector is generated based on the weighted topology map; each dimension of the topology feature vector includes: the signal propagation path loss value between room nodes, and the spatial correlation strength factor based on historical emergency event data.

6. The hotel emergency rescue method based on multimodal data fusion analysis and judgment according to claim 1, characterized in that... In step 5, a normalization-based modal adaptive scaling algorithm is used, including: applying max-min normalization to the speech feature vector; applying max-min normalization to the heart rate feature vector; and applying graph embedding normalization to the topological feature vector.

7. A hotel emergency rescue method based on multimodal data fusion analysis and judgment according to claim 1, characterized in that... In step 8, the spatiotemporal fusion network model includes a two-stream adversarial training architecture: The spatial feature processing flow is a three-level graph convolutional neural network hierarchical structure. Each level performs a neighborhood node aggregation operation to process topological feature vectors. The temporal feature processing flow is a five-level dilated causal convolutional layer stacked structure used to process signal sequences; Adversarial discriminators are used to constrain the consistency of dual-stream feature distributions through KL divergence. The decision-making level generates emergency status assessment results based on a fully connected network.

8. A hotel emergency rescue system based on multimodal data fusion analysis and judgment, characterized in that: This system is used to implement the methods described in claims 1-7; the system includes the following modules: The acquisition module, deployed in the target room and adjacent rooms, includes a voice acquisition unit and a millimeter-wave radar unit, used to acquire voice signals and heart rate signals in real time. The hotel management system interface module is used to retrieve the topological relationship data between the target room and its adjacent rooms. The multimodal feature extraction module includes: a speech processing unit, used to perform framing and short-time Fourier transform on the speech signal to generate speech feature vectors; and a heart rate analysis unit, used to extract QRS group feature points through wavelet transform and empirical mode decomposition to generate heart rate feature vectors. The topology modeling unit is used to construct a multi-level adjacent room network based on topological relationship data and generate a topological feature vector. The feature fusion module is used to normalize the three types of feature vectors and concatenate them into a fused feature vector; The dynamic weight adjustment module is used to analyze the spatiotemporal synchronicity of signal mutations. When the correlation between the signal mutation in the target room and the adjacent rooms is lower than the threshold, the weight coefficients of each feature vector in the fused feature vector are automatically adjusted. The spatiotemporal fusion decision module integrates a pre-trained spatiotemporal fusion network model, receives the fusion feature vector and the weight coefficients of each feature vector, and generates the emergency status judgment result.

9. A hotel emergency rescue system for multimodal data fusion analysis and judgment according to claim 8, characterized in that... The system also includes an emergency response module, which responds to the emergency status assessment results and performs route planning, alarm linkage, and emergency notification issuance.

10. A hotel emergency rescue system for multimodal data fusion analysis and judgment according to claim 8, characterized in that... The system includes a gateway, a voice screen, and a vital signs detector. The voice screen is used to collect and process voice signals, the vital signs detector is used to collect and analyze heart rate signals, and the gateway is used to generate a topological feature vector and fuse it with the voice feature vector and heart rate feature vector. Based on the fused feature vector and the weight coefficients of each feature vector, an emergency status judgment result is generated.

Images