Hotel first-aid method and system based on multi-modal data fusion analysis and judgment

Abstract

The invention discloses a hotel first-aid method and system for multi-modal data fusion analysis and judgment, and relates to the field of general control systems or methods, and the method comprises the following steps: generating a voice feature vector and a heart rate feature vector through voice and heart rate signals collected in a target room and an adjacent room; constructing a multi-level adjacent room network according to the topological relation data, and generating a topological feature vector; and fusing the three feature vectors, analyzing signal mutation synchronism and spatial distribution characteristics of the target room and the adjacent room, when a signal mutation correlation evaluation result is an isolated event, adjusting a weight coefficient of the fusion feature vector, inputting the pre-spatio-temporal fusion network model, and judging whether the target room is in an emergency state or not. According to the invention, on the basis of multi-modal cooperative sensing, the weight coefficient of the fusion feature vector is dynamically adjusted through analysis of signal mutation, and the weight coefficient is input into the space-time fusion network for decision making, so that the accuracy and reliability of emergency state recognition are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of general control systems or methods, and in particular to a hotel first aid method and system for multimodal data fusion analysis and judgment. Background Art

[0002] In modern hotel management and security, ensuring guest safety is paramount. The ability to respond quickly to sudden health crises or emergencies is directly linked to a hotel's service quality and social responsibility. However, current emergency monitoring technologies for hotels generally suffer from significant limitations. Mainstream solutions rely on heart rate or voice sensors within a single room to identify abnormal conditions. However, in practice, sudden noises such as televisions, speakers, and dropped objects share similar spectral characteristics with actual emergency signals (such as falls, collisions, and cries for help), resulting in persistently high false alarm rates. Furthermore, the densely packed room layout of hotels leads to cross-conduction of signals between adjacent spaces. Emergency signals from a target room may be misinterpreted as incidents in adjacent rooms. When multiple people experience anomalies simultaneously, the system cannot pinpoint the true source of the risk. Existing technologies fail to effectively leverage the topological relationships of hotel rooms to construct spatial perception models and lack collaborative analysis mechanisms for cross-room signal mutations. This leads to a significant decrease in system reliability in complex interference environments. This passive, single-point monitoring architecture has become an industry-wide bottleneck restricting the accuracy of hotel emergency response. Summary of the Invention

[0003] The purpose of the present invention is to provide a hotel first aid method and system for multimodal data fusion analysis and judgment, which aims to overcome the above-mentioned problems existing in the prior art.

[0004] To achieve the purpose, the present invention provides the following technical solutions: A hotel first aid method for multimodal data fusion analysis and judgment includes the following steps: Step 1: collecting voice signals and heart rate signals from a target room and its adjacent rooms, and simultaneously obtaining topological relationship data of the target room and its adjacent rooms from a hotel management system; Step 2: performing preprocessing and framing operations on the voice signal, extracting features using a short-time Fourier transform method, and generating a voice feature vector representing the voice content; Step 3: performing window segmentation and wavelet transform processing on the heart rate signal, and extracting QRS wave group feature points in the heart rate signal in combination with an empirical mode decomposition method, and generating a heart rate feature vector representing vital signs; Step 4: constructing a multi-level adjacent room network centered on the target room based on the topological relationship data, and generating a topological feature vector representing spatial correlation; Step 5: 5: Normalize the generated speech feature vector, heart rate feature vector, and topology feature vector and concatenate them into a unified fusion feature vector; Step 6: Analyze the synchronization of signal mutations and spatial distribution characteristics between the target room and adjacent rooms in the time dimension for the speech features and heart rate features contained in the fusion feature vector to generate a signal mutation correlation evaluation result; Step 7: If the signal mutation correlation evaluation result is lower than the preset threshold and the signal mutations in the target room have no significant correlation with those in other rooms in terms of spatial distribution, adjust the weight coefficients of each feature in the fusion feature vector; Step 8: Input the fusion feature vector and the weight coefficients of each feature into the pre-trained spatiotemporal fusion network model, perform comprehensive analysis and processing, and generate a judgment result on whether the target room is in an emergency state.

[0005] Furthermore, step 9 is also included: if the generated judgment result indicates that the target room is in an emergency state, the shortest path from the target room to the rescue location is calculated based on the topological relationship data, the hotel's internal linkage alarm mechanism is triggered, and an alarm signal is generated.

[0006] Furthermore, the heart rate signal in step 1 is collected by a non-contact millimeter-wave radar, specifically including: step 1.1: transmitting a frequency-modulated continuous wave millimeter-wave signal to the target room and adjacent rooms; step 1.2: after receiving the reflected signal, using adaptive beamforming technology to separate the reflected signal components of different rooms; step 1.3: for the reflected signal component of the target room, using a micro-Doppler feature extraction algorithm to analyze the chest motion waveform to generate a heart rate signal; step 1.4: dynamically suppressing crosstalk between rooms based on the energy intensity of the signal components in the adjacent rooms.

[0007] Furthermore, the step 1.4 specifically includes the following steps: Step 1.4.1: Calculate the target room reflected signal component in real time Reflected signal components from adjacent rooms Energy ratio:

[0008] Where N is the total number of topologically adjacent rooms; Step 1.4.2: Set the number of rooms ; When any When , the adaptive notch filter is started for the room i and the number of rooms is updated ; The adaptive notch filter is:

[0009] in, is the preset energy leakage threshold; the filter parameters are dynamically configured as follows: Center frequency locking Main frequency component; attenuation factor , controls the stopband depth; bandwidth factor , controls the suppression bandwidth; Step 1.4.3: Coherently add the filtered signal to the original signal in the target room:

[0010] in, , is the cumulative weight coefficient; M is satisfied The number of rooms.

[0011] Furthermore, when constructing a multi-level adjacent room network in step 4, the hotel building information model (BIM) database of the hotel management system is synchronously accessed to extract the wall acoustic attenuation coefficient and spatial geometric structure data to generate a weighted topological map that integrates physical propagation characteristics; then, a topological feature vector is generated based on the weighted topological map; each dimension of the topological feature vector includes: a signal propagation path loss value between room nodes, and a spatial correlation strength factor based on historical emergency event data.

[0012] Furthermore, the above weighted topological graph ; Among them, the vertex set is a room node (target room + adjacent room); edge set is the room connectivity relationship, edge weight ;in, is the wall acoustic attenuation coefficient of the target room and adjacent rooms; is the straight-line distance between the target room and the adjacent room, The historical emergency event association frequency of the target room and adjacent rooms; The weighted topology graph is input into the graph neural network (GNN) and the following operations are performed to generate the topology feature vector: Aggregate neighborhood weights through two layers of graph convolution operations:

[0013] in, is the target room node, are adjacent room nodes, Target room node The neighbor set of For the target room node Total number of adjacent rooms; is the edge weight, representing the target room node Nodes in adjacent rooms the strength of the association between For the Layer feature vector, When is the initial feature of the adjacent room node; Aggregate output features for input to the next layer or as the final feature; For the activation function ReLU: ; Perform hierarchical aggregation on the target room node: The first convolution layer output: , that is, aggregating the features of first-degree adjacent rooms; The second convolution layer output: , i.e., aggregating the features of second-degree adjacent rooms; Embedding vector for target room node Perform learnable pooling and output a 32-dimensional topological feature vector The topological eigenvector After normalization, it is finally used in step 6 to calculate the path loss compensation factor in the spatial distribution characteristics.

[0014] Furthermore, in step 5, a modal adaptive scaling algorithm is used for normalization processing, including: using maximum and minimum normalization for speech feature vectors; using maximum and minimum normalization for heart rate feature vectors; and using graph embedding normalization for topological feature vectors.

[0015] Furthermore, in step 8, the spatiotemporal fusion network model includes a two-stream adversarial training architecture: The spatial feature processing flow is a three-level graph convolutional neural network hierarchy, where each level performs neighborhood node aggregation operations to process topological feature vectors. The time series feature processing flow is a five-level dilated causal convolution layer stack structure used to process signal sequences; Adversarial discriminator, constraining the consistency of dual-stream feature distribution through KL divergence; The decision layer generates emergency status judgment results based on the fully connected network.

[0016] A hotel emergency system for multimodal data fusion analysis and judgment, used to implement any of the above methods; the system includes the following modules: The acquisition module is deployed in the target room and adjacent rooms and includes a voice acquisition unit and a millimeter-wave radar unit for real-time acquisition of voice and heart rate signals. The hotel management system interface module is used to retrieve the topological relationship data between the target room and the adjacent rooms; The multimodal feature extraction module includes: a speech processing unit for performing frame segmentation and short-time Fourier transform on the speech signal to generate a speech feature vector; a heart rate analysis unit for extracting QRS complex feature points through wavelet transform and empirical mode decomposition to generate a heart rate feature vector; a topological modeling unit for constructing a multi-level adjacent room network based on topological relationship data to generate a topological feature vector; a feature fusion module for normalizing the three types of feature vectors and splicing them into a fused feature vector; a dynamic weight adjustment module for analyzing the spatiotemporal synchronization of signal mutations, and automatically adjusting the weight coefficients of each feature vector in the fused feature vector when the correlation between the target room signal mutation and the adjacent room is lower than the threshold; a spatiotemporal fusion decision module that integrates a pre-trained spatiotemporal fusion network model, receives the fused feature vector and the weight coefficients of each feature vector, and generates an emergency status judgment result. Furthermore, the system also includes an emergency response module for executing path planning, linkage alarm, and emergency notification issuance operations in response to the emergency status judgment result.

[0017] Furthermore, the system includes a gateway, a voice screen and a vital signs detector; the voice screen is used to collect and process voice signals, the vital signs detector is used to collect and analyze heart rate signals, and the gateway is used to generate a topological feature vector and fuse it with the voice feature vector and heart rate feature, and generate an emergency status judgment result based on the fused feature vector and the weight coefficient of each feature vector.

[0018] Compared with the prior art, the present invention has the following beneficial effects: First, the present invention innovatively constructs a multimodal collaborative perception architecture by synchronously collecting voice and heart rate signals from the target room and adjacent rooms and integrating the topological relationship data of the hotel rooms. Based on the analysis of the temporal synchronization and spatial distribution isolation of signal mutations, the weight coefficients of each feature of the fused feature vector are dynamically adjusted, and then the normalized feature vector and dynamic weight parameters are jointly input into the spatiotemporal fusion network for decision-making. This solution effectively overcomes key problems in hotel scenarios, such as noise conduction between adjacent rooms and sudden interference confusion, significantly reduces the false alarm rate in strong interference environments, and greatly improves the accuracy and reliability of emergency status recognition.

[0019] Secondly, the present invention adopts millimeter-wave radar to conduct non-contact detection of frequency-modulated continuous wave (FMCW) millimeter-wave signals, and innovatively cross-integrates the beam control in radar engineering, the acoustic characteristics of hotel buildings, and physiological signal processing to form a non-contact heart rate monitoring paradigm for complex scenarios with multiple rooms in hotels, so as to obtain purer heart rate signals in the target room, thereby providing a solid foundation for accurately analyzing and judging the emergency signals in the target room.

[0020] Thirdly, the present invention transforms the inherent physical characteristics of hotel buildings into dynamically calculable topological feature vectors, breaking through the limitation of traditional emergency systems where topological relationships are only used as logical connections. By deeply integrating the acoustic attenuation coefficient, spatial geometry data, and historical behavior patterns in the building information model (BIM), it constructs a weighted topological graph that integrates physical propagation loss and behavioral correlation strength for the first time, accurately depicting the true attenuation law of the signal in the complex wall structure of the hotel. More importantly, the graph neural network is innovatively used to perform neighborhood aggregation calculations on the weighted topology: through two layers of graph convolution operations, the feature vector of the target room is It not only aggregates the direct signal impact of the first-degree adjacent room, but also captures the indirect propagation effect of the second-degree adjacent room. The original solution solves the problem of neighbor number deviation caused by the hotel's star / chain heterogeneous layout. The final generated 32-dimensional topological feature vector It is input into the subsequent analysis module as a spatial propagation compensation factor to reduce the misjudgment rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the method in the present invention.

[0022] Figure 2 In the present invention, the structural frame of the system Figure 1 .

[0023] Figure 3 In the present invention, the structural frame of the system Figure 2 . DETAILED DESCRIPTION

[0024] The specific embodiments of the present invention are described below with reference to the accompanying drawings. In order to fully understand the present invention, many details are described below, but for those skilled in the art, the present invention can be implemented without these details.

[0025] like Figure 1 As shown, a hotel first aid method based on multimodal data fusion analysis and judgment includes the following steps: Step 1: Collect voice signals and heart rate signals from the target room and its adjacent rooms, and at the same time obtain the topological relationship data of the target room and its adjacent rooms from the hotel management system.

[0026] In one specific embodiment, the heart rate signal is collected via a non-contact millimeter-wave radar, specifically comprising the following steps: Step 1.1: Transmitting a frequency-modulated continuous-wave millimeter-wave signal to the target room and adjacent rooms. Step 1.2: After receiving the reflected signal, using adaptive beamforming technology to separate the reflected signal components from different rooms. Step 1.3: For the reflected signal component from the target room, using a micro-Doppler feature extraction algorithm to analyze the chest motion waveform and generate a heart rate signal. Step 1.4: Dynamically suppressing crosstalk between rooms based on the energy intensity of the signal components from adjacent rooms.

[0027] Preferably, the above step 1.4 specifically includes the following steps: Step 1.4.1: Calculate the target room reflected signal component in real time Reflected signal components from adjacent rooms Energy ratio:

[0028] Where N is the total number of topologically adjacent rooms.

[0029] Step 1.4.2: Set the number of rooms ; When any When , the adaptive notch filter is started for the room i and the number of rooms is updated ; The adaptive notch filter is:

[0030] in, is the preset energy leakage threshold; the filter parameters are dynamically configured as follows: Center frequency locking Main frequency component; attenuation factor , controls the stopband depth; bandwidth factor , controls the suppression bandwidth.

[0031] Step 1.4.3: Coherently add the filtered signal to the original signal in the target room:

[0032] in, , is the cumulative weight coefficient; M is satisfied The number of rooms.

[0033] It uses millimeter-wave radar to capture the millimeter-level micro-motion of people's chest movements in hotel rooms through non-contact detection of frequency-modulated continuous wave (FMCW) millimeter-wave signals, and innovatively couples radar signal processing technology with the structural characteristics of hotel buildings. That is, adaptive beamforming technology accurately separates the reflected signals of the target room and adjacent rooms through spatial filtering, essentially building a virtual signal cage with physical walls as a natural isolation barrier. At the same time, a dynamic crosstalk suppression mechanism is adopted, that is, real-time calculation based on the energy ratio formula, so that the system can intelligently identify the interference differences caused by concrete walls (high attenuation) and lightweight partition walls (low attenuation); when energy leakage is detected exceeding the threshold, the parameterized notch filter uses a mathematical mapping relationship to suppress the interference caused by concrete walls (high attenuation) and lightweight partition walls (low attenuation). Dynamically adjust the stopband depth. Ultimately, a pure heart rate signal is reconstructed through a coherent accumulation compensation algorithm, preserving the original physiological characteristics while reducing cross-room crosstalk errors. The specific technical solution in step 1 above integrates three independent technical domains: beam steering in radar engineering, hotel architectural acoustics, and physiological signal processing. This creates a non-contact heart rate monitoring paradigm tailored to hotel scenarios, enabling the acquisition of a purer heart rate signal in the target room. This, in turn, provides a solid foundation for accurately analyzing and identifying emergency signals in the target room.

[0034] Step 2: Perform preprocessing and framing operations on the speech signal, use the short-time Fourier transform method to extract features, and generate a speech feature vector that represents the speech content.

[0035] In a specific embodiment, the above step 2 specifically includes the following steps: Step 2.1: Enhance the voice signal by eliminating noise and channel distortion, completing the preprocessing of the voice signal. This step aims to eliminate interference from fixed hotel noise sources such as television sound and air conditioning noise.

[0036] Step 2.2: With a frame length of 30ms, a frame shift of 10ms, and a Hanning window function, the pre-processed speech signal is framed to obtain framed speech data, which can be expressed mathematically as follows:

[0037] in, is the frame number, is the number of frame shift samples, is the number of samples of the frame length, is the Hanning window function. This step allows us to accurately capture sudden cries for help (such as short pulses of "help").

[0038] Step 2.3: Perform short-time Fourier transform on the framed speech data: First, perform STFT calculation on the framed speech data to obtain the complex spectrum of the mth frame :

[0039] in, is the number of FFT points, covering a 30ms frame length; Secondly, the complex spectrum of the mth frame Perform power spectrum calculation:

[0040] The power spectrum is then passed through a 40-dimensional Mel filter bank to generate log-Mel spectrum features:

[0041] in, is the triangular window function of the i-th Mel filter; Then, calculate the dynamic difference spectrum components:

[0042] Get the final feature vector, that is, the speech feature vector: .

[0043] In this step, the STFT power spectrum converts the time-domain signal into a joint time-frequency representation, simultaneously capturing both the distress call semantics (frequency domain) and the intensity fluctuations (time domain). A Mel filter bank simulates the human hearing characteristics, focusing on the sensitive frequency bands of the human voice and suppressing irrelevant high-frequency noise. Dynamic differential calculations enhance the vibrato characteristics of painful moans, improving recognition accuracy. This step converts the original speech into a task-specific low-dimensional feature vector (Mel spectrum + dynamic differential), retaining only the acoustic patterns required for distress call detection (such as intensity fluctuations and vibrato), while discarding the phase information and full-band details that can reconstruct the semantics. This makes reverse engineering of the original speech difficult and prevents information leakage. Preferably, the entire process from voice acquisition to feature extraction is completed locally on the device, ensuring that the original audio does not leave the device and only the encrypted feature vectors are uploaded, completely blocking any leakage paths and further preventing information leakage.

[0044] Step 3: Perform window segmentation and wavelet transform on the heart rate signal, and use the empirical mode decomposition (EMD) method to extract the QRS complex feature points in the heart rate signal to generate a heart rate feature vector representing vital signs.

[0045] In a specific embodiment, the heart rate signal is window segmented and wavelet transformed. Specifically, the heart rate signal is window segmented with a window length of 5s-10s to ensure that 3-5 complete heart cycles are covered. Then, the db6 wavelet base is used for wavelet transformation to match the QRS complex morphology. The final heart rate feature vector includes but is not limited to the standard deviation of the RR interval. , and QRS complex width .

[0046] Step 4: Based on the topological relationship data, a multi-level adjacent room network centered on the target room is constructed to generate a topological feature vector representing the spatial correlation.

[0047] In a specific embodiment, when constructing the multi-level adjacent room network in step 4 above, the hotel building information model (BIM) database is accessed simultaneously to extract the wall acoustic attenuation coefficient and spatial geometry data to generate a weighted topological map that integrates physical propagation characteristics. A topological feature vector is then generated based on the weighted topological map. Each dimension of the topological feature vector includes: (a) Signal propagation path loss value between room nodes; (b) Spatial correlation intensity factor based on historical emergency incident data.

[0048] As an advantage, the method comprises the following steps: Step 4.1: When constructing a multi-level adjacent room network, access the hotel building information model (BIM) database, extract the wall acoustic attenuation coefficient and spatial geometry data, and generate a weighted topology map that integrates physical propagation characteristics. ; Among them, the vertex set is a room node, including the target room node and its adjacent room nodes; the edge set is the room connectivity relationship, edge weight ;in, is the wall acoustic attenuation coefficient of the target room and adjacent rooms; is the straight-line distance between the target room and the adjacent room, The historical emergency event association frequency of the target room and adjacent rooms; Step 4.2: Input the above weighted topology graph into the graph neural network (GNN) and perform the following operations to generate the topology feature vector: Step 4.2.1: Aggregate neighborhood weights through two layers of graph convolution operations:

[0049] in, is the target room node, are adjacent room nodes, Target room node The neighbor set of For the target room node Total number of adjacent rooms; is the edge weight, representing the target room node Nodes in adjacent rooms the strength of the association between For the Layer feature vector, When is the initial feature of the adjacent room node; Aggregate output features for input to the next layer or as the final feature; For the activation function ReLU: .

[0050] Step 4.2.2: Perform hierarchical aggregation on the target room node: The first convolution layer output: , that is, aggregating the features of first-degree adjacent rooms; The second convolution layer output: , that is, aggregating the features of second-degree adjacent rooms.

[0051] Step 4.2.3: Embed the target room node vector Perform learnable pooling and output a 32-dimensional topological feature vector The topological eigenvector After normalization, it is finally used in step 6 to calculate the path loss compensation factor in the spatial distribution characteristics.

[0052] In this step, the inherent physical characteristics of the hotel building are converted into dynamically calculable topological feature vectors, breaking through the limitation of traditional emergency systems where topological relationships are only logical connections. By deeply integrating the acoustic attenuation coefficient, spatial geometry data, and historical behavior patterns in the building information model (BIM), a weighted topological graph is constructed that integrates physical propagation loss and behavioral correlation strength, accurately depicting the actual attenuation law of the signal in the complex wall structure of the hotel. The graph neural network is innovatively used to perform neighborhood aggregation calculations on the weighted topology: through two layers of graph convolution operations, the feature vector of the target room is It not only aggregates the direct signal impact of the first-degree adjacent room, but also captures the indirect propagation effect of the second-degree adjacent room. The original solution solves the problem of neighbor number deviation caused by the hotel's star / chain heterogeneous layout. The final generated 32-dimensional topological feature vector It is input into the subsequent analysis module as a spatial propagation compensation factor to reduce the misjudgment rate.

[0053] Step 5: Normalize the generated speech feature vector, heart rate feature vector, and topology feature vector and concatenate them into a unified fusion feature vector.

[0054] In a specific embodiment, step 5 adopts a modal adaptive scaling algorithm for normalization processing, including: using maximum and minimum normalization for speech feature vectors; using maximum and minimum normalization for heart rate feature vectors; using maximum and minimum normalization for topological feature vectors Use graph embedding normalization. Finally, the normalized speech features, heart rate features, and topology features are combined to form a new fused feature vector.

[0055] Step 6: Based on the speech features and heart rate features contained in the fused feature vector, analyze the synchronization of signal mutations and the spatial distribution characteristics in the time dimension between the target room and the adjacent rooms to generate a signal mutation correlation evaluation result.

[0056] In a specific implementation, step 6 specifically includes the following steps: Step 6.1: Determine and mark the speech mutation points based on the following conditions:

[0057] in, Dynamic threshold coefficient, including but not limited to daytime (such as 7 am to 7 pm) 3.5, night time (e.g. 19:00 to 7:00) reduced to 2.5; is the baseline standard deviation of the speech dynamic difference spectrum.

[0058] Step 6.2: Determine and mark the heart rate mutation point based on the following conditions:

[0059] in, is the standard deviation of the RR interval (interval between adjacent heart beats); is the QRS complex width (ventricular depolarization time).

[0060] Step 6.3: The synchronization measurement formula between the target room target and the adjacent room j is as follows:

[0061] in, is the time decay constant, The maximum physiological response delay, including but not limited to s and s; is the indicator function (if the time difference exceeds the limit, it is set to zero, i.e., it does not meet the The indicator function is reset to zero); Using the above synchronization measurement formula, calculate the synchronization measurement of the speech signal of the target room target and the adjacent room j , and heart rate signal synchronization metrics .

[0062] Step 6.4: Calculate the joint synchronization of the speech signal and the heart rate signal:

[0063] in, For voice weight, including but not limited to .

[0064] Step 6.5: Using Topological Eigenvectors Calculate the path loss between the target room target and the adjacent room j:

[0065] like , it indicates a concrete wall (high attenuation); if , it means lightweight partition wall (low attenuation); Compute spatially corrected synchronicity:

[0066] in, is the building attenuation factor, including but not limited to .

[0067] Based on steps 6.1 to 6.5, the spatial correction synchronization of the target room target and each adjacent room is obtained: .

[0068] Step 6.6: Calculate the time dimension score: ; Calculate the spatial dimension score:

[0069] Step 6.7: Generate signal mutation correlation assessment results, specifically: like , it is judged as an isolated incident; if , it is judged as a related event; if , it is judged as a propagation event across multiple rooms.

[0070] Step 7: If the signal mutation correlation evaluation result is lower than the preset threshold, and the signal mutation in the target room has no significant spatial correlation with other rooms (i.e., the signal mutation correlation evaluation result is an isolated event), adjust the weight coefficients of each feature in the fused feature vector. If the signal mutation correlation evaluation result is a correlation event or a propagation event, retain the default weight coefficients.

[0071] In a specific embodiment, the basic weight distribution (ie, the default weight coefficient) is as follows: The weight coefficient of the speech feature is 0.4; The weight coefficient of the heart rate feature is 0.5; The weight coefficient of topological features is 0.1.

[0072] When the signal mutation correlation assessment result is an isolated event, the weight coefficient is adjusted as follows: Weight coefficient of speech features for ; Weight coefficient of heart rate feature for ; Weight coefficient of topological features is 0.1; in, , is the weight attenuation factor of speech features, , is the weight enhancement factor of the heart rate feature; Weight coefficient for speech features , weight coefficient of heart rate feature and the weight coefficient of topological features Perform normalization:

[0073] Step 8: The fused feature vector and the weight coefficients of each feature are input into the pre-trained spatiotemporal fusion network model, and comprehensive analysis and processing are performed to generate a judgment result on whether the target room is in an emergency state.

[0074] In a specific embodiment, step 8 specifically includes the following steps: Step 8.1: Construct weighted feature vector:

[0075] Step 8.2: Input the weighted feature vector into the pre-trained spatiotemporal fusion network model for spatiotemporal fusion network processing to generate a judgment result on whether the target room is in an emergency state. The judgment results are divided into emergency state and non-emergency state. Among them, the spatiotemporal fusion network model includes a two-stream adversarial training architecture, which is as follows:

[0076] Among them, the three-level graph convolutional neural network hierarchy of the spatial feature processing flow performs neighborhood node aggregation operations at each level to process topological feature vectors: the first level aggregates directly adjacent rooms, the second level expands to rooms on the same floor, and the third level incorporates vertically adjacent rooms.

[0077] The time series feature processing flow is a five-level dilated causal convolution layer stack structure used to process signal sequences; Adversarial discriminator, constraining the consistency of dual-stream feature distribution through KL divergence; The decision layer generates emergency status judgment results based on a fully connected network, which are divided into emergency status and non-emergency status. The decision layer includes but is not limited to using a Softmax function to convert feature vectors into an emergency status probability distribution. When the emergency category probability exceeds a preset threshold, the output result is emergency status; otherwise, the output result is non-emergency status.

[0078] Step 9: If the generated judgment result indicates that the target room is in an emergency state, the shortest path from the target room to the rescue location is calculated based on the topological relationship data, triggering the hotel's internal linkage alarm mechanism to generate an alarm signal; an emergency notification is sent to the relevant areas through the hotel's communication system, and an event log is recorded for subsequent optimization processing.

[0079] like Figure 1 and Figure 2 As shown, the present invention also discloses a hotel emergency system for multimodal data fusion analysis and judgment, which is used to implement the above method and includes the following modules: The acquisition module is deployed in the target room and adjacent rooms and includes a voice acquisition unit and a millimeter-wave radar unit for real-time acquisition of voice and heart rate signals. A hotel management system interface module configured to retrieve topological relationship data between a target room and adjacent rooms; The multimodal feature extraction module includes a speech processing unit, a heart rate analysis unit, and a topology modeling unit. The speech processing unit is used to perform framing and short-time Fourier transform on the speech signal to generate a speech feature vector; the heart rate analysis unit is used to extract QRS wave group feature points through wavelet transform and empirical mode decomposition to generate a heart rate feature vector; the topology modeling unit is used to construct a multi-level adjacent room network based on topological relationship data to generate a topological feature vector; the feature fusion module is used to normalize the three types of feature vectors and splice them into a fused feature vector; the dynamic weight adjustment module is used to analyze the spatiotemporal synchronization of signal mutations, and automatically adjust the weight coefficients of each feature vector in the fused feature vector when the correlation between the target room signal mutation and the adjacent room is lower than the threshold; the spatiotemporal fusion decision module integrates the pre-trained spatiotemporal fusion network model, receives the fused feature vector and the weight coefficients of each feature vector, and generates the emergency status judgment result. The emergency response module is used to respond to the emergency status judgment results and perform path planning, linkage alarm and emergency notification operations.

[0080] like Figure 3 As shown, in a specific embodiment, the system includes a gateway 300, a voice screen [1] 100 and a vital sign detector 200. The voice screen is used to collect and process voice signals, and the vital sign detector is used to collect and analyze heart rate signals. The gateway is responsible for edge computing, which is used to generate a topological feature vector and fuse it with the voice feature vector and heart rate feature, and generate an emergency status judgment result based on the fused feature vector and the weight coefficient of each feature vector.

[0081] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A hotel first aid method based on multimodal data fusion analysis and judgment, characterized by , including the following steps: Step 1: Collect voice signals and heart rate signals from the target room and its adjacent rooms, and obtain the topological relationship data of the target room and its adjacent rooms from the hotel management system; Step 2: Perform preprocessing and framing operations on the voice signal, use the short-time Fourier transform method to extract features, and generate a voice feature vector that represents the voice content; Step 3: Perform window segmentation and wavelet transform on the heart rate signal, and use the empirical mode decomposition method to extract the QRS complex feature points in the heart rate signal to generate a heart rate feature vector representing vital signs; Step 4: Based on the topological relationship data, a multi-level adjacent room network centered on the target room is constructed to generate a topological feature vector representing spatial correlation; Step 5: The generated speech feature vector, heart rate feature vector, and topological feature vector are normalized and spliced ​​into a unified fusion feature vector; Step 6: Based on the speech features and heart rate features contained in the fusion feature vector, the signal mutation synchronization and spatial distribution characteristics of the target room and the adjacent rooms in the time dimension are analyzed to generate a signal mutation correlation evaluation result; Step 7: If the signal mutation correlation evaluation result is lower than the preset threshold and the signal mutation in the target room has no significant correlation with other rooms in terms of spatial distribution, the weight coefficient of each feature of the fusion feature vector is adjusted; Step 8: The fusion feature vector and the weight coefficient of each feature are jointly input into the pre-trained spatiotemporal fusion network model to perform comprehensive analysis and processing to generate a judgment result on whether the target room is in an emergency state.

2. A hotel first aid method based on multimodal data fusion analysis and judgment according to claim 1, characterized in that ,It also includes step 9: If the generated judgment result shows that the target room is in an emergency state, the shortest path from the target room to the rescue location is calculated based on the topological relationship data, and the hotel's internal linkage alarm mechanism is triggered to generate an alarm signal.

3. A hotel first aid method based on multimodal data fusion analysis and judgment according to claim 2, characterized in that The heart rate signal in step 1 is collected by a non-contact millimeter-wave radar, specifically including: step 1.1: transmitting a frequency-modulated continuous wave millimeter-wave signal to the target room and adjacent rooms; step 1.2: after receiving the reflected signal, using adaptive beamforming technology to separate the reflected signal components of different rooms; step 1.3: for the reflected signal component of the target room, using a micro-Doppler feature extraction algorithm to analyze the chest motion waveform and generate a heart rate signal; step 1.4: dynamically suppressing crosstalk between rooms based on the energy intensity of the signal components in adjacent rooms.

4. A hotel first aid method based on multimodal data fusion analysis and judgment according to claim 3, characterized in that , the step 1.4 specifically includes the following steps: Step 1.4.1: Calculate the target room reflected signal component in real time Reflected signal components from adjacent rooms Energy ratio: Where N is the total number of topologically adjacent rooms; Step 1.4.2: Set the number of rooms ; When any When , the adaptive notch filter is started for the room i and the number of rooms is updated ; The adaptive notch filter is: in, is the preset energy leakage threshold; the filter parameters are dynamically configured as follows: Center frequency locking Main frequency component; attenuation factor , controls the stopband depth; bandwidth factor , controls the suppression bandwidth; Step 1.4.3: Coherently add the filtered signal to the original signal in the target room: in, , is the cumulative weight coefficient; M is satisfied The number of rooms.

5. The hotel first aid method of multimodal data fusion analysis and judgment according to claim 1 is characterized in that When constructing a multi-level adjacent room network in step 4, the hotel building information model (BIM) database is accessed synchronously to extract the wall acoustic attenuation coefficient and spatial geometric structure data to generate a weighted topological map that integrates physical propagation characteristics; then a topological feature vector is generated based on the weighted topological map; each dimension of the topological feature vector includes: the signal propagation path loss value between room nodes, and the spatial correlation intensity factor based on historical emergency event data.

6. A hotel first aid method based on multimodal data fusion analysis and judgment according to claim 1, characterized in that In the step 5, the modal adaptive scaling algorithm is used for normalization processing, including: using maximum and minimum normalization for speech feature vectors; using maximum and minimum normalization for heart rate feature vectors; and using graph embedding normalization for topological feature vectors.

7. The hotel first aid method of multimodal data fusion analysis and judgment according to claim 1 is characterized in that In step 8, the spatiotemporal fusion network model includes a two-stream adversarial training architecture: The spatial feature processing flow is a three-level graph convolutional neural network hierarchy, where each level performs neighborhood node aggregation operations to process topological feature vectors. The time series feature processing flow is a five-level dilated causal convolution layer stack structure used to process signal sequences; Adversarial discriminator, constraining the consistency of dual-stream feature distribution through KL divergence; The decision layer generates emergency status judgment results based on the fully connected network.

8. A hotel emergency system based on multimodal data fusion analysis and judgment, characterized by: The system is used to implement the method described in claims 1-7; the system includes the following modules: The acquisition module is deployed in the target room and adjacent rooms and includes a voice acquisition unit and a millimeter-wave radar unit for real-time acquisition of voice and heart rate signals. The hotel management system interface module is used to retrieve the topological relationship data between the target room and the adjacent rooms; The multimodal feature extraction module includes: a speech processing unit for performing frame segmentation and short-time Fourier transform on the speech signal to generate a speech feature vector; a heart rate analysis unit for extracting QRS wave group feature points through wavelet transform and empirical mode decomposition to generate a heart rate feature vector; A topological modeling unit, used to construct a multi-level adjacent room network based on topological relationship data and generate a topological feature vector; Feature fusion module, used to normalize the three types of feature vectors and splice them into a fused feature vector; The dynamic weight adjustment module is used to analyze the spatiotemporal synchronization of signal mutations. When the correlation between the signal mutation in the target room and the adjacent room is lower than the threshold, the weight coefficient of each eigenvector in the fused feature vector is automatically adjusted; the spatiotemporal fusion decision module integrates the pre-trained spatiotemporal fusion network model, receives the fused feature vector and the weight coefficients of each eigenvector, and generates the emergency status judgment result.

9. A hotel emergency system for multimodal data fusion analysis and judgment according to claim 8, characterized in that ,The system also includes an emergency response module for ,performing path planning, linkage alarm and emergency notification ,issuance in response to the emergency status judgment result.

10. The hotel emergency system for multimodal data fusion analysis and judgment according to claim 8 is characterized in that The system includes a gateway, a voice screen and a vital signs detector; the voice screen is used to collect and process voice signals, the vital signs detector is used to collect and analyze heart rate signals, and the gateway is used to generate a topological feature vector and fuse it with the voice feature vector and heart rate feature, and generate an emergency status judgment result based on the fused feature vector and the weight coefficient of each feature vector.

Images