A multi-modal information fusion method in clinical nursing

Abstract

The application relates to the field of artificial intelligence technology and discloses a multi-modal information fusion method in clinical nursing, which comprises the following steps: acquiring multi-modal data composed of high-frequency signal flow and low-frequency state sequence; extracting a sub-threshold membrane potential distribution gradient by using a pulse neural network branch to generate a global state representation value; determining a dynamic activation threshold of an artificial neural network branch according to the global state representation value; when the dynamic activation threshold is reduced to a preset trigger level, activating the artificial neural network branch to fuse features, extracting sub-threshold topological features and outputting an analytical vector, and determining a channel mask vector to filter the high-frequency signal flow. The application uses the dynamic characteristics of a pulse neuron to guide feature screening, solves the entropy value dislocation problem of high-frequency signals and low-frequency semantic events, and enhances the reasoning certainty of information fusion under complex working conditions.

Description

Technical Field

[0001] This invention relates to a multimodal information fusion method in clinical nursing, belonging to the field of artificial intelligence technology. Background Technology

[0002] When processing multimodal information, current computer systems generally adopt parallel computing architectures based on artificial neural networks. By aligning physical time-series signals and semantic discrete data through a unified time window, dense feature coupling of multidimensional features is achieved. However, this clock-driven computing method faces significant technical limitations when dealing with sparse event streams with long-tail distribution characteristics. Especially in long-term monitoring environments, effective features account for a low proportion of the total data stream, but the computing engine needs to maintain feature extraction and model updates around the clock, resulting in severe redundant computing power consumption for edge computing devices, which makes it difficult to meet the engineering requirements of low energy efficiency operation.

[0003] Existing feature alignment methods, achieved through linear interpolation or resampling, disrupt the original dynamic chain between heterogeneous signals. There is a millisecond-level trigger-response logic between the microscopic fluctuations of physical characteristics and the asynchronous semantics of text records. Traditional frame-level alignment mechanisms easily obscure subthreshold evolutionary precursors, leading to limited model stability when processing highly concealed temporal features and difficulty in reconstructing the true causal relationship between physical symptoms and human intervention. Conventional improvements such as simply increasing the sampling frequency or adding network layers cannot address these bottlenecks at the mechanism level. Such solutions not only exacerbate the system's heat dissipation pressure and deployment costs but also cause core features to be submerged in massive sampling noise, resulting in response delays when facing sudden physiological changes due to the deterioration of the signal-to-noise ratio in the feature space. The industry is exploring how to construct an asynchronous [system / mechanism] that balances perceptual sensitivity and computational efficiency. The fusion framework still lacks effective model architecture support; in terms of underlying information fusion control methods, existing technologies have limitations. For example, Chinese utility model patent with authorization announcement number CN212565113U discloses a nursing information acquisition device based on visual multimodal fusion technology. This technology optimizes the mechanical locking structure of the mounting base and camera device to improve hardware deployment stability. However, the multimodal data processing logic still uses a linear acquisition scheme. Based on the improvement of the overall physical device stability, it ignores the non-stationarity and asynchronous characteristics between high-frequency physiological electrical signals and low-frequency semantic records in clinical monitoring. It lacks the dynamic transformation of the underlying computing architecture. When the solution faces sensor artifact interference or sudden changes in physiological state, it cannot achieve on-demand allocation of computing resources at the algorithm layer. It is difficult to capture the precursors of micro-evolution below the ignition threshold. The mismatch between hardware form and algorithm mechanism makes it impossible for the system to take into account both sensing sensitivity and low-energy operation in clinical practice.

[0004] Therefore, how to provide an asynchronous information processing framework that can achieve dynamic coupling while taking into account computational energy efficiency and causal locking is the technical problem to be solved by this invention. Summary of the Invention

[0005] To address the problems mentioned in the background art, the technical solution of the present invention is as follows: A multimodal information fusion method in clinical nursing, comprising the following steps: Step 101: Obtain multimodal data to be processed, which includes high-frequency time-series signal streams and low-frequency unstructured state sequences. Step 102: Input the high-frequency time-series signal stream into the first computing branch, use the spiking neural network in the first computing branch to extract the membrane potential state matrix of the spiking neuron set at time 102, and calculate the membrane potential distribution gradient of the spiking neuron set in the subthreshold state to generate a global state characterization value. Step 103: Perform cross-modal tensor gating modulation on the second computational branch based on the global state representation value. The second computational branch uses an artificial neural network. Determine the dynamic activation threshold of the second computational branch using a negative exponential function and based on the global state representation value. Step 104: When the dynamic activation threshold is reduced to the preset trigger level, the second computing branch is activated to perform nonlinear feature fusion on the multimodal data to be processed, and tensor splicing operation is performed on the high-frequency time-series signal stream to extract subthreshold topological evolution features, and the second computing branch outputs an analytical vector. Step 105: Based on the parsed vector, extract the channel mask vector corresponding to the current event type from the preset logical mapping table, and perform channel-level dot product filtering on the high-frequency timing signal stream to shield signal interference terms.

[0006] Preferably, the calculation of the global state characterization value in step 102 includes the following steps: Step 201, performing spatial dimension normalization processing on the membrane potential state matrix and extracting the membrane potential evolution gradients of the spiking neurons that have not crossed the ignition threshold; Step 202, inputting the membrane potential evolution gradients into a preset nonlinear mapping function to calculate the global state characterization value that characterizes the energy evolution intensity of the high-frequency time-series signal flow before the triggering critical point.

[0007] Preferably, the dynamic activation threshold θ(t) of the second computational branch in step 103 follows the following parameterized association rule: , where θ(t) is the dynamic activation threshold at time t; α is the preset basic activation threshold of the artificial neural network; e is the natural constant; α is the preset sensitivity adjustment factor; V(t) is the value of the first computational branch at time t. Output global state representation value.

[0008] Preferably, while performing step 101, the following steps are also included: Step 401, opening a circular buffer in the data buffer area, and continuously storing the high-frequency time-series signal stream into the circular buffer according to a preset sampling frequency; Step 402, monitoring the excitation flow density in the low-frequency unstructured state sequence in real time, wherein the excitation flow density is determined by the number of external non-continuous excitation data entries enqueued per unit time.

[0009] Preferably, the nonlinear feature fusion in step 104 includes the following steps: Step 501, determining the current event type based on the analytical vector, and retrieving the corresponding basic backtracking offset from the preset offset table according to the event type; Step 502, using the scaling coefficient generated based on the excitation flow density to linearly correct the basic backtracking offset and determine the target backtracking window; Step 503, extracting the temporal component segment corresponding to the target backtracking window from the circular buffer, and performing semantic anchor alignment between the temporal component segment and the low-frequency unstructured state sequence.

[0010] Preferably, the channel-level dot product filtering in step 105 includes the following steps: Step 601, analyzing the prior knowledge contained in the vector to identify the target feature dimension that is relevant to the current event; Step 602, generating a binary mask vector based on the target feature dimension, wherein the corresponding bit of the relevant dimension is set to 1 and the corresponding bit of the irrelevant dimension is set to 0; Step 603, calculating the dot product between the binary mask vector and the feature vector of the multidimensional component in the high-frequency time-series signal stream, thereby shielding the signal interference terms of the irrelevant dimension.

[0011] Preferably, the cross-modal tensor gating modulation performed in step 103 further includes the following steps: step 701, periodically comparing the global state representation value with the preset state monitoring threshold; step 702, when the global state representation value exceeds the state monitoring threshold for three consecutive sampling periods, forcibly locking the dynamic activation threshold to the lowest trigger level.

[0012] Preferably, the nonlinear feature fusion in step 104 further includes the following steps: step 801, dynamically reconstructing the weight matrix of the second computational branch and the synaptic weights of the spiking neurons in the first computational branch; step 802, adjusting the leakage current time constant of the spiking neurons in the first computational branch in reverse according to the convergence speed of the nonlinear feature fusion.

[0013] Preferably, after step 503, the following steps are also included: step 901, calculating the spatiotemporal Euclidean distance between the temporal component fragment and the semantic anchor point; step 902, evaluating the confidence of multimodal information fusion based on the spatiotemporal Euclidean distance, and outputting a state offset index that characterizes the state evolution trend of the monitored source entity.

[0014] Preferably, the activation of the second computation branch in step 104 is subject to the following closed-loop logic constraint: within the sampling period when the dynamic activation threshold is reduced to the preset trigger level, it is determined whether the time change rate of the global state representation value is continuously positive; if it is positive, the forward propagation operation of the artificial neural network is activated.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. In multimodal information fusion, by constructing an asynchronous pulse coding layer and a dual-path heterogeneous computing topology, continuous time-series input is converted into a sparse pulse sequence, and the membrane potential accumulation is performed by the first computing branch containing spiking neurons. Since the global membrane potential state value generated by the first computing branch directly participates in regulating the activation response threshold of the second computing branch, this mechanism ensures that the artificial neural network with deep inference capabilities maintains a silent or low-power state during most signal stability periods, and only starts forward propagation at the moment when the potential accumulation breaks through the dynamic threshold. This computing resource scheduling method based on neurodynamic gating realizes the logical transformation from clock-driven to event-driven, effectively reducing the standby power consumption and redundant computing overhead of edge computing nodes in long-term monitoring environments.

[0016] 2. By reading the instantaneous membrane potential values ​​of all spiking neurons in the first computational branch within the same microsecond of trigger ignition and recombining them into a subthreshold distribution tensor, this invention solves the information truncation problem in the asynchronous coupling process of heterogeneous networks. This subthreshold distribution tensor objectively represents the energy evolution gradient of the physical signal before the trigger critical point. By performing a Hadamard product operation with the initial semantic tensor, the system imprints the microsecond-level physical evolution precursors in situ into the discrete semantic feature space. This physical-level data coupling mechanism avoids the physical discarding of subthreshold topological information by the traditional discrete pulse triggering mode, and enhances the extraction depth and inference stability of artificial neural networks for temporal features with hidden precursors.

[0017] 3. Utilizing a semantic anchor-based backtracking calibration mechanism, the system opens an independently operating circular buffer within the monitoring terminal and dynamically establishes the backtracking offset based on the event type parsed from the semantic stream. By using the scaling coefficient generated by the system's operation stream density to correct the backtracking window, the locking range of physiological parameter segments can be adaptively adjusted according to the environmental task load. This mechanism logically offsets the objective lag in manual recording, ensuring precise alignment of physiological mutation facts and supplementary semantic records on the timeline. Thus, without adding external timing hardware, the true causal logic between physiological abnormalities and intervention actions can be restored through parameterized adaptive control of the data stream window. Attached Figure Description

[0018] Figure 1 This is a flowchart of the dynamic fusion of nursing data using the dual-branch neural network of the present invention; Figure 2 This is a schematic diagram illustrating the semantic-driven temporal feature alignment and state evolution evaluation of this invention.

[0019] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] A multimodal information fusion method for clinical nursing includes the following steps: Step 101: Obtain multimodal data to be processed, which includes high-frequency time-series signal streams and low-frequency unstructured state sequences. Step 102: Input the high-frequency time-series signal stream into the first computing branch, use the spiking neural network in the first computing branch to extract the membrane potential state matrix of the spiking neuron set at time 102, and calculate the membrane potential distribution gradient of the spiking neuron set in the subthreshold state to generate a global state characterization value. Step 103: Perform cross-modal tensor gating modulation on the second computational branch based on the global state representation value. The second computational branch uses an artificial neural network. Determine the dynamic activation threshold of the second computational branch using a negative exponential function and based on the global state representation value. Step 104: When the dynamic activation threshold is reduced to the preset trigger level, the second computing branch is activated to perform nonlinear feature fusion on the multimodal data to be processed, and tensor splicing operation is performed on the high-frequency time-series signal stream to extract subthreshold topological evolution features, and the second computing branch outputs an analytical vector. Step 105: Based on the parsed vector, extract the channel mask vector corresponding to the current event type from the preset logical mapping table, and perform channel-level dot product filtering on the high-frequency timing signal stream to shield signal interference terms.

[0022] Preferably, the calculation of the global state characterization value in step 102 includes the following steps: Step 201, performing spatial dimension normalization processing on the membrane potential state matrix and extracting the membrane potential evolution gradients of the spiking neurons that have not crossed the ignition threshold; Step 202, inputting the membrane potential evolution gradients into a preset nonlinear mapping function to calculate the global state characterization value that characterizes the energy evolution intensity of the high-frequency time-series signal flow before the triggering critical point.

[0023] Preferably, the dynamic activation threshold θ(t) of the second computational branch in step 103 follows the following parameterized association rule: , where θ(t) is the dynamic activation threshold at time t; α is the preset basic activation threshold of the artificial neural network; e is the natural constant; α is the preset sensitivity adjustment factor; V(t) is the value of the first computational branch at time t. Output global state representation value.

[0024] Preferably, while performing step 101, the following steps are also included: Step 401, opening a circular buffer in the data buffer area, and continuously storing the high-frequency time-series signal stream into the circular buffer according to a preset sampling frequency; Step 402, monitoring the excitation flow density in the low-frequency unstructured state sequence in real time, wherein the excitation flow density is determined by the number of external non-continuous excitation data entries enqueued per unit time.

[0025] Preferably, the nonlinear feature fusion in step 104 includes the following steps: Step 501, determining the current event type based on the analytical vector, and retrieving the corresponding basic backtracking offset from the preset offset table according to the event type; Step 502, using the scaling coefficient generated based on the excitation flow density to linearly correct the basic backtracking offset and determine the target backtracking window; Step 503, extracting the temporal component segment corresponding to the target backtracking window from the circular buffer, and performing semantic anchor alignment between the temporal component segment and the low-frequency unstructured state sequence.

[0026] Preferably, the channel-level dot product filtering in step 105 includes the following steps: Step 601, analyzing the prior knowledge contained in the vector to identify the target feature dimension that is relevant to the current event; Step 602, generating a binary mask vector based on the target feature dimension, wherein the corresponding bit of the relevant dimension is set to 1 and the corresponding bit of the irrelevant dimension is set to 0; Step 603, calculating the dot product between the binary mask vector and the feature vector of the multidimensional component in the high-frequency time-series signal stream, thereby shielding the signal interference terms of the irrelevant dimension.

[0027] Preferably, the cross-modal tensor gating modulation performed in step 103 further includes the following steps: step 701, periodically comparing the global state representation value with the preset state monitoring threshold; step 702, when the global state representation value exceeds the state monitoring threshold for three consecutive sampling periods, forcibly locking the dynamic activation threshold to the lowest trigger level.

[0028] Preferably, the nonlinear feature fusion in step 104 further includes the following steps: step 801, dynamically reconstructing the weight matrix of the second computational branch and the synaptic weights of the spiking neurons in the first computational branch; step 802, adjusting the leakage current time constant of the spiking neurons in the first computational branch in reverse according to the convergence speed of the nonlinear feature fusion.

[0029] Preferably, after step 503, the following steps are also included: step 901, calculating the spatiotemporal Euclidean distance between the temporal component fragment and the semantic anchor point; step 902, evaluating the confidence of multimodal information fusion based on the spatiotemporal Euclidean distance, and outputting a state offset index that characterizes the state evolution trend of the monitored source entity.

[0030] Preferably, the activation of the second computation branch in step 104 is subject to the following closed-loop logic constraint: within the sampling period when the dynamic activation threshold is reduced to the preset trigger level, it is determined whether the time change rate of the global state representation value is continuously positive; if it is positive, the forward propagation operation of the artificial neural network is activated.

[0031] Example 1: Addressing the entropy misalignment problem between high-frequency physiological time-series signal streams and low-frequency discrete nursing event records in multimodal physiological parameter monitoring scenarios within clinical intensive care units, the system acquires a 100Hz high-frequency time-series signal stream from sensors in real time and inputs it into the first computational branch. The spiking neural network in the first computational branch... The membrane potential state matrix M(t) reflecting the dynamic state of the neuronal cluster is extracted. By performing spatial dimension normalization on the membrane potential state matrix M(t), the subthreshold membrane potential distribution gradient that has not crossed the ignition threshold is captured, thereby generating a global state representation value V(t) that characterizes the energy intensity of signal evolution. This step provides a logical prior in the physical dimension for the computational awakening of the second computational branch by transforming micro-potential fluctuations into global gating parameters.

[0032] To address the significant redundant computing power consumption during stable periods in existing intensive computing methods and ensure the effective capture of hidden risk characteristics, the system utilizes the global state representation value V(t) to perform cross-modal tensor gating modulation on the second computational branch employing an artificial neural network, based on parameterized association rules. Dynamically calculate the dynamic activation threshold of the second computational branch. ,in Let be the dynamic activation threshold at time t. The system uses a preset base activation threshold, where e is a natural constant, α is a preset sensitivity adjustment factor, and V(t) is the global state representation value output by the first computational branch. This mapping logic enables the system to maintain the activation threshold at a level that suppresses high-energy-consuming computation when the physiological state is stable. When subthreshold energy accumulation is detected, the threshold is instantaneously lowered to wake up the deep feature extraction logic. During the sampling period when the dynamic activation threshold is lowered to the preset trigger level, the computational engine extracts the instantaneous subthreshold distribution tensor in the first computational branch and performs a Hadamard product operation with the initial semantic tensor obtained by converting it with the low-frequency unstructured state sequence. This generates a modulation feature tensor containing physical spatiotemporal causal locking features and inputs it into the second computational branch to perform nonlinear feature fusion. Finally, the second computational branch outputs an analytical vector pointing to a specific care risk level and determines the channel mask vector accordingly. This mechanism reduces node computational power consumption while capturing hidden precursor features by changing the physical boundary of spatiotemporal alignment.

[0033] Example 2: In a real-time monitoring and verification scenario involving continuous reception of clinical multimodal physiological signals, the system constructs a signal processing platform including a high-speed vector operation unit. The data source is a publicly available heart rate RR interval sequence as the physical input to the high-frequency time-series signal stream. Simultaneously acquired discrete nursing behavior records are used as a low-frequency unstructured state sequence. To simulate electromagnetic noise interference and baseline drift caused by human movement in a real ward environment, Gaussian additive noise with signal-to-noise ratios of 20dB, 15dB, and 10dB is superimposed on the high-frequency time-series signal stream. The system sets the high-frequency sequence sampling frequency to 120Hz and uses this to establish an original feature library. The operation and control unit executes the parameter calibration procedure and sets the basic activation threshold of the second computational branch. The sensitivity adjustment factor α is 0.5, and the value of the sensitivity adjustment factor α is determined based on the root mean square volatility of the high-frequency time-series signal stream during the silent period. This balances the sensitivity of the feature capture and the computational load. During the signal stability period at time t of 10s, the global state characterization value V(10) of the first computation branch is 0.5, according to the parameterized correlation rule. The current dynamic activation threshold θ(10) is calculated to be 7.788. Since this value has not dropped below the preset trigger level of 1.5, the second calculation branch maintains the suppression state, thereby blocking the invalid feedforward calculation cycle.

[0034] When the monitored environment enters the signal fluctuation period at time t=45s, the first computational branch captures the hidden energy jump in the high-frequency sequence. The system extracts the membrane potential distribution gradient in the subthreshold state of the spiking neuron set and calculates the global state representation value V(45) as 4.2. At this time, the operation control unit calculates the dynamic activation threshold θ(45) as 1.22. This value drops below the preset trigger level and activates the forward propagation operation of the second computational branch. The system extracts the subthreshold distribution tensor at the current time and performs a Hadamard product operation with the initial semantic tensor to generate a modulation feature tensor. To verify the determinism of this technical solution in complex environments, a first comparative sample group using continuous fully connected operation and a second comparative sample group with a fixed activation threshold of 3 and without using subthreshold tensors to participate in feature modulation are set up. Under signal-to-noise ratio (SNR) conditions, the false alarm rate of the proposed sample group was 1.2%, compared to 1.5% for the first comparative sample group and 4.5% for the second comparative sample group. When the SNR decreased to 15dB, the false alarm rate of the proposed sample group was 2.8%, while the false alarm rate of the second comparative sample group increased to 18.4%. Furthermore, under extreme interference conditions of 10dB, the second comparative sample group exhibited a nonlinear surge in false alarms due to a lack of threshold adjustment capability, reaching a decision deviation of 42.7%, while the proposed sample group still kept the false alarm rate below 5.1%. At the same time, measurement data showed that the computational power consumption of the proposed sample group throughout the entire test cycle was only 32% to 35% of the total power consumption of the first comparative sample group. The data results indicate that the system maintained a gradual evolution of decision accuracy during environmental SNR fluctuations and suppressed redundant computing power overhead.

[0035] Example 3: Addressing the engineering obstacles posed by the lack of specific physical constraints on parameter setting and subthreshold gradient extraction paths for cross-modal tensor gating modulation logic in clinical multimodal information fusion, the system deploys a test platform including a historical physiological feature database and an offline tensor computation core. It retrieves an annotated multi-lead physiological signal test set and divides it into a stable baseline segment containing only background noise from the human body at rest and an abnormal prodromal segment containing high-frequency aberrations prior to ventricular fibrillation. The stable baseline segment and the abnormal prodromal segment are respectively input into the first computational branch. The spiking neural network in the first computational branch extracts the spiking neuron set at time... The computation control unit extracts the instantaneous membrane potential values ​​of each spiking neuron in the membrane potential state matrix M(t). By subtracting the resting potential physical reference of the corresponding neuron and dividing the resulting difference by the difference between the ignition threshold of the neuron and the resting potential physical reference, a spatial dimension normalized tensor is generated. The computation engine calculates the L2 norm of the spatial dimension normalized tensor to extract the quantized membrane potential distribution gradient. The membrane potential distribution gradient is then input into a preset nonlinear activation function for compression mapping, and the global state representation value V(t) that converges to a specific value range is output.

[0036] The system determines the dynamic activation threshold θ(t) based on the boundary approximation calibration procedure and calculates the preset basic activation threshold in the association rules. Along with the sensitivity adjustment factor α, the offline tensor computation core processes stationary baseline segments to extract all global state representation values ​​output within the computation cycle, and selects the maximum value among them as the baseline extremum. The product of the preset trigger level and the scalar amplification factor is set as the preset basic activation threshold. This setting ensures that the dynamic activation threshold calculated based on the product remains above the preset trigger level under baseline extreme value input conditions. The operation control unit processes abnormal precursor segments and extracts the global state characterization value sequence within the high-frequency abnormality feature occurrence cycle. The arithmetic mean of the sequence is calculated as the event feature scalar. Based on this, a parameter optimization algorithm is run to increment the sensitivity adjustment factor α with a preset step size until the dynamic activation threshold obtained by substituting the event feature scalar into the formula decreases to equal the product of the preset trigger level and the preset convergence constant. At this point, the current value is locked as the sensitivity adjustment factor α for actual operation. Using the above tensor normalization process and boundary approximation calibration procedure, the system constructs a quantization mapping path from the underlying neuron potential to the global gated variable at the numerical calculation layer. This enables the wake-up boundary of the second calculation branch to obtain a numerical benchmark based on the statistical distribution characteristics of a specific test set, eliminating the empirical setting error in the abnormal signal discrimination trigger logic.

[0037] The first computational branch extracts the membrane potential state matrix M(t) of the spiking neuron set at time t, where M(t) is the set of potential values ​​composed of N spiking neurons. By calculating the deviation between the instantaneous membrane potential value of each neuron in the membrane potential state matrix M(t) and the preset resting potential benchmark, and dividing it by the difference between the ignition threshold and the resting potential benchmark, a normalized tensor reflecting the polarization degree of the neuron is obtained. The normalized tensor is input into a preset nonlinear mapping function to generate a global state representation value V(t), where V(t) is a scalar representing the energy evolution intensity of the high-frequency time-series signal flow before the triggering critical point. The sensitivity adjustment factor α is used to control the response. The slope and subthreshold micro-potential fluctuations are converted into deterministic control signals to adjust the activation boundary of the second computational branch. When the system performs cross-modal feature fusion, it uses a linear projection matrix to perform spatial transformation on the subthreshold distribution tensor output by the first computational branch, so that the physical feature dimension after transformation is consistent with the initial semantic tensor corresponding to the second computational branch. The polarization gradient distribution of the physical dimension is injected into the semantic feature space using the Hadamard product operation to generate a modulation feature tensor containing physical spatiotemporal correlation features. By establishing the mapping relationship between physical layer energy evolution and semantic nodes, the second computational branch obtains physical layer subthreshold weight guidance when performing feedforward calculation.

[0038] Example 4: In clinical deployment scenarios facing differences in sensor responses across devices, the system performs baseline parameter calibration for monitoring nodes. The data acquisition module acquires a baseline test signal stream lasting 24 hours. This baseline test signal stream includes the routine vital signs sequence of the target bed and discrete care behavior logs, excluding risk events that are diagnosed. The first calculation branch receives the baseline test signal stream and calculates the global state representation value V(t) for each sampling period. The operation control unit extracts the global state representation value sequence generated throughout the calibration period and calculates its time dimension arithmetic mean as the baseline median.

[0039] The system sets the baseline median as the reference point for the target environment and sets the initial value of the sensitivity adjustment factor to 0.1. The operation control unit increments the sensitivity adjustment factor in steps of 0.05, and after each increment, it inputs the benchmark test signal stream into the model containing the dynamic activation threshold calculation logic. When the number of wake-ups of the calculated second calculation branch exceeds the preset wake-up limit of 5 times per day, the increment stops. The operation control unit extracts the value corresponding to the previous step and solidifies it as the sensitivity adjustment factor of the deployment node. The system reads the discrete action timestamps in the benchmark test signal stream and counts the total frequency of various discrete actions occurring within 24 hours. The operation control unit sorts various actions according to the total frequency and selects the three most frequent actions. In the preset offset table, the basic backtracking offset corresponding to these three actions is reduced to 80%, 70%, and 60% of the initial value, respectively, to customize the feature extraction window of the target bed.

[0040] Example 5: In a clinical intensive care scenario facing concurrent access of heterogeneous sensors and dynamic fluctuations in data flow density, the system implements parameter calibration procedures and initial state definition procedures to eliminate uncertainties in computing power allocation. The calibration procedure defines the dynamic initial state of the spiking neurons in the first computing branch and sets the physical reference for the resting potential of each neuron to -65mV and the ignition threshold of each neuron to -50mV. The system monitors the excitation flux density in the low-frequency unstructured state sequence in real time. The excitation flow density is determined by the number of external discontinuous excitation data entries stored in the circular buffer per unit time. The system selects the average excitation flow density of the deployment environment during the quiescent period as the benchmark value. The arithmetic control unit is based on the formula Calculate the scaling factor γ used to adjust the base backoff offset, where γ is the scaling factor. For a moment The monitored excitation current density, The scaling factor γ is used as a preset average excitation flow density benchmark to participate in the real-time correction of the time parameters in the preset offset table, thereby reducing the redundant computing load of the second computing branch by reducing the backtracking window length when the excitation data is enqueued at high frequency.

[0041] When the global state representation value V(t) output by the first computational branch drives the dynamic activation threshold to drop below the preset trigger level, the system initiates a cross-modal feature tensor projection procedure to establish the correlation between the neuron's local energy gradient and global semantics. The computation control unit extracts the subthreshold distribution tensor at time t. Its tensor dimension is determined by the spatial distribution of neurons in the last layer of the spiking neural network. The system receives the initial semantic tensor obtained by transforming a low-frequency unstructured state sequence. The computation engine utilizes the bilinear mapping algorithm to transform the subthreshold distribution tensor Projected onto the initial semantic tensor The feature space, and according to the formula Perform element-wise Hadamard product operations to generate modulation feature tensors. ,in For the generated modulation feature tensor, For the subthreshold distribution tensor after projection mapping, As the initial semantic tensor, the symbol ∘ represents the Hadamard product operator. This logic enables the evolution gradient of local membrane potential below the ignition threshold to directly guide the weight of the physical dimension of discrete semantic features, realizing feature coupling from continuous signals to the underlying semantic space. The modulated feature tensor is input into the artificial neural network of the second computational branch to carry out feedforward computation. The system uses convolutional kernel sequences to extract cross-modal correlation features in the feature fusion layer and the pooling unit performs dimensionality reduction on the feature mapping before outputting an analytical vector. The operation control unit determines the current nursing event category based on the numerical distribution of the analytical vector. By implementing the above excitation flow density calibration and tensor space alignment procedures, the system establishes a feature filtering mechanism based on physical energy evolution in the asynchronous data flow environment, so that the inference output of the second computational branch remains within the preset accuracy range when facing signal-to-noise ratio fluctuations.

[0042] The system monitors the excitation flow density in low-frequency unstructured state sequences in real time to generate a scaling factor γ, where γ is a dimensionless parameter used to correct the length of the time window. Specifically, it is calculated by dividing the number of external excitation data entries in the current sampling period by the preset baseline flow density mean. The quotient is used as a correction factor to determine the target backtracking window based on the base backtracking offset in the preset offset table. When the excitation flow density increases, the target backtracking window length is reduced by the scaling factor γ, and redundant computational overhead is suppressed under high load conditions. The system calculates the spatiotemporal Euclidean distance between the extracted time-series component segments and the semantic anchors to assess the fusion confidence. The semantic anchors are the nursing event start points with timestamps in the low-frequency unstructured state sequences. Through parameterized adaptive control of the data flow window, the micro-fluctuation characteristics of physiological parameters and the asynchronously entered semantic records are aligned on the time axis, eliminating the impact of manual recording lag on causal logic locking.

[0043] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal information fusion method in clinical nursing, characterized in that, Includes the following steps: Step 101: Obtain multimodal data to be processed, which includes high-frequency time-series signal streams and low-frequency unstructured state sequences. Step 102: Input the high-frequency time-series signal stream into the first computing branch, use the spiking neural network in the first computing branch to extract the membrane potential state matrix of the spiking neuron set at time 102, and calculate the membrane potential distribution gradient of the spiking neuron set in the subthreshold state to generate a global state characterization value. Step 103: Perform cross-modal tensor gating modulation on the second computational branch based on the global state representation value. The second computational branch uses an artificial neural network. The dynamic activation threshold θ(t) of the second computational branch is determined by using a negative exponential function and based on the global state representation value. Step 104: When the dynamic activation threshold is reduced to the preset trigger level, the second computing branch is activated to perform nonlinear feature fusion on the multimodal data to be processed, and tensor splicing operation is performed on the high-frequency time-series signal stream to extract subthreshold topological evolution features, and the second computing branch outputs an analytical vector. Step 105: Based on the parsed vector, extract the channel mask vector corresponding to the current event type from the preset logical mapping table, and perform channel-level dot product filtering on the high-frequency timing signal stream to shield signal interference terms.

2. The multimodal information fusion method in clinical nursing according to claim 1, characterized in that, Step 102, calculating the global state representation value, includes the following steps: Step 201, performing spatial dimension normalization on the membrane potential state matrix and extracting the membrane potential evolution gradients of the spiking neurons that have not crossed the ignition threshold; Step 202, inputting the membrane potential evolution gradients into a preset nonlinear mapping function to calculate the global state representation value that represents the energy evolution intensity of the high-frequency time-series signal flow before the triggering critical point.

3. The multimodal information fusion method in clinical nursing according to claim 2, characterized in that, In step 103, the dynamic activation threshold θ(t) of the second computational branch is determined according to the following parameterized association rule: , where θ(t) is the dynamic activation threshold at time t; α is the preset basic activation threshold of the artificial neural network; e is the natural constant; α is the preset sensitivity adjustment factor; V(t) is the value of the first computational branch at time t. Output global state representation value.

4. The multimodal information fusion method in clinical nursing according to claim 1, characterized in that, While performing step 101, the following steps are also included: Step 401, opening a circular buffer in the data buffer area and continuously storing the high-frequency time-series signal stream into the circular buffer according to the preset sampling frequency; Step 402, monitoring the excitation flow density in the low-frequency unstructured state sequence in real time, the excitation flow density being determined by the number of external non-continuous excitation data entries enqueued per unit time.

5. The multimodal information fusion method in clinical nursing according to claim 4, characterized in that, Step 104 involves performing nonlinear feature fusion, which includes the following steps: Step 501, determining the current event type based on the analytical vector, and retrieving the corresponding basic backtracking offset from the preset offset table according to the event type; Step 502, linearly correcting the basic backtracking offset using the scaling coefficient generated based on the excitation flow density to determine the target backtracking window; Step 503, extracting the temporal component segment corresponding to the target backtracking window from the circular buffer, and performing semantic anchor alignment between the temporal component segment and the low-frequency unstructured state sequence.

6. The multimodal information fusion method in clinical nursing according to claim 1, characterized in that, Step 105 performs channel-level dot product filtering, which includes the following steps: Step 601, analyze the prior knowledge contained in the vector to identify the target feature dimension that is relevant to the current event; Step 602, generate a binary mask vector based on the target feature dimension, where the relevant dimension has a value of 1 and the irrelevant dimension has a value of 0; Step 603, calculate the dot product between the binary mask vector and the feature vector of the multi-dimensional component in the high-frequency time-series signal stream to mask the signal interference terms of the irrelevant dimension.

7. The multimodal information fusion method in clinical nursing according to claim 1, characterized in that, Step 103, which involves performing cross-modal tensor gating modulation, further includes the following steps: Step 701, periodically comparing the global state representation value with a preset state monitoring threshold; Step 702, when the global state representation value exceeds the state monitoring threshold for three consecutive sampling periods, forcibly locking the dynamic activation threshold to the lowest trigger level.

8. The multimodal information fusion method in clinical nursing according to claim 1, characterized in that, The nonlinear feature fusion in step 104 also includes the following steps: Step 801, dynamically reconstructing the weight matrix of the second computational branch and the synaptic weights of the spiking neurons in the first computational branch; Step 802, adjusting the leakage current time constant of the spiking neurons in the first computational branch in reverse according to the convergence speed of the nonlinear feature fusion.

9. A multimodal information fusion method in clinical nursing according to claim 5, characterized in that, Step 503 is followed by the following steps: Step 901, calculate the spatiotemporal Euclidean distance between the temporal component fragment and the semantic anchor; Step 902, evaluate the confidence of multimodal information fusion based on the spatiotemporal Euclidean distance, and output the state offset index that characterizes the state evolution trend of the monitoring source entity.

10. A multimodal information fusion method in clinical nursing according to claim 1, characterized in that, In step 104, the activation of the second computation branch is subject to the following closed-loop logic constraint: within the sampling period when the dynamic activation threshold is reduced to the preset trigger level, it is determined whether the time change rate of the global state representation value is continuously positive; if it is positive, the forward propagation operation of the artificial neural network is activated.

Images