An abnormal behavior early warning method, device, equipment and medium

Abstract

The application discloses an abnormal behavior early warning method and device, equipment and medium, and relates to the technical field of artificial intelligence, which comprises the following steps: collecting target data corresponding to a patient; determining a spatial feature vector based on a video frame through a depth separable convolution and a preset coordinate attention module, determining a light flow feature sequence corresponding to the video frame, and determining a time feature vector based on the light flow feature sequence and a one-dimensional hollow convolution network; determining a fusion feature vector based on the spatial feature vector and the time feature vector, performing multi-modal fusion on a radar feature vector corresponding to millimeter wave radar data and a pressure feature vector corresponding to pressure sensing mattress data, and obtaining a multi-modal fusion feature; inputting the multi-modal fusion feature into a trained hidden Markov model to obtain a probability value of a patient's abnormal behavior in a future target time period, and determining an early warning operation based on the size relationship between the probability value and a preset threshold. Real-time detection and early warning of abnormal behavior are realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an abnormal behavior early warning method, device, equipment, and medium. Background Technology

[0002] In healthcare facilities (including hospital wards, ICUs, nursing homes, and rehabilitation centers), incidents of abnormal patient behavior such as falls, bed falls, extubation, loss of consciousness, and agitation / harm occur frequently, posing a serious threat to patient safety. With the continued exacerbation of the nursing staff shortage, intelligent early warning technology based on video surveillance has become an important auxiliary tool.

[0003] Currently, mainstream abnormal behavior detection solutions mainly face the following problems: Problem 1: Insufficient detection accuracy, and alarms can only be triggered after the fact. Existing solutions mostly use standard CNNs (Convolutional Neural Networks) to extract features from single frames or short frame sequences, which has limited ability to distinguish between normal nursing procedures and abnormal behaviors. In complex nursing scenarios, the false alarm rate remains high, severely impacting nurses' trust in and willingness to use the alert system. More importantly, these solutions only trigger alarms after the abnormal behavior has occurred, representing a reactive response—there is often a gap of several minutes between the occurrence of the abnormality and the nurse's arrival, missing the optimal intervention window.

[0004] Question 2: Insufficient real-time performance, relying on cloud deployment: Many solutions involve backbone network parameters in the tens of millions, requiring cloud deployment, which results in inference latency that cannot meet the requirements for real-time bedside early warning. Furthermore, cloud deployment introduces bandwidth bottlenecks and data privacy risks.

[0005] Question 3: Poor environmental adaptability: When faced with common situations such as changes in lighting, camera obstruction, or drawn curtains, the accuracy of vision-based solutions drops significantly. Most existing solutions lack the ability to integrate data from multiple sensors, making it difficult to adapt to changing nursing environments.

[0006] Question 4: Lack of independent and controllable data security safeguards: Existing solutions generally rely on foreign operating systems and chips, and encryption components mostly use international algorithms, leaving core components dependent on foreign suppliers. Patients' sensitive data lacks independent and controllable security measures during transmission and storage.

[0007] Overall, existing technologies have shortcomings in terms of detection accuracy, real-time performance, and environmental adaptability, and in particular, they lack the ability to make predictions in advance. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide an abnormal behavior early warning method, device, equipment, and medium, which can accurately detect abnormal behavior and trigger an early warning before the occurrence of abnormal events, realizing real-time, high-precision, interpretable, and autonomously controllable early warning of abnormal medical and nursing behaviors on edge devices. The specific solution is as follows: Firstly, this application discloses an abnormal behavior early warning method, including: The system collects target data corresponding to the patient through target sensors, including video acquisition sensors, millimeter-wave radar, and pressure-sensing mattresses; the target data includes video frames from the patient's bedside, millimeter-wave radar data, and pressure-sensing mattress data. The spatial feature vector is determined based on the video frame by using depthwise separable convolution and a preset coordinate attention module, the optical flow feature sequence corresponding to the video frame is determined, and the temporal feature vector is determined based on the optical flow feature sequence and a one-dimensional dilated convolutional network. Based on the spatial feature vector and the temporal feature vector, a fusion feature vector is determined. The fusion feature vector, the radar feature vector corresponding to the millimeter-wave radar data, and the pressure feature vector corresponding to the pressure sensor mattress data are then fused in a multimodal manner to obtain a multimodal fusion feature. The multimodal fusion features are input into a trained Hidden Markov Model to obtain the probability value of abnormal patient behavior within a future target time period. Based on the relationship between the probability value and a preset threshold, a corresponding early warning operation is determined.

[0009] Optionally, the millimeter-wave radar data includes the distance between the patient and the radar, the patient's movement speed, and the patient's azimuth angle change; the pressure-sensing mattress data includes pressure distribution data; Accordingly, after collecting the target data corresponding to the patient through the target sensor, the process further includes: The time corresponding to the video frame is determined as the time base. Based on the time base, the millimeter-wave radar data and the pressure-sensing mattress data are subjected to timestamp interpolation and alignment operations to complete the time synchronization of the target data.

[0010] Optionally, determining the optical flow feature sequence corresponding to the video frame includes: Extract the region of interest from the video frame, and calculate the optical flow feature sequence of the region of interest based on the Lucas-Cannard method.

[0011] Optionally, determining the fused feature vector based on the spatial feature vector and the temporal feature vector includes: The spatial feature vector and the temporal feature vector are concatenated to obtain the concatenated feature vector; The concatenated feature vectors are weighted and fused to obtain the fused feature vector.

[0012] Optionally, before inputting the multimodal fused features into the trained Hidden Markov Model, the method further includes: Historical target data is acquired, and the historical target data within a preset time period is labeled to indicate whether there is any abnormal behavior. The Hidden Markov Model is then trained based on the labeled data to obtain a trained Hidden Markov Model.

[0013] Optionally, determining the corresponding early warning operation based on the relationship between the probability value and the preset threshold includes: If the probability value is less than the first preset threshold, the patient's behavior is determined to be normal, and a level one warning is issued; the level one warning is a screen display warning. If the probability value is greater than or equal to the first preset threshold and less than the second preset threshold, it is determined that the patient will exhibit abnormal behavior, and a secondary warning is issued; the secondary warning is to push warning information to staff and issue an audio-visual warning. If the probability value is greater than or equal to the second preset threshold, it is determined that the patient has abnormal behavior and a level three warning is issued. The level three warning is to issue an audible and visual alarm, notify the staff by telephone, and push the on-site video to the staff.

[0014] Optionally, the method further includes: If it is determined that the patient will exhibit abnormal behavior or that the patient has abnormal behavior, the current video frame image, warning level, probability value, spatial feature vector, temporal feature vector, millimeter-wave radar data, and pressure sensor mattress data are input into the target large language model to obtain the corresponding natural language warning report. The natural language warning report will be fed back to the staff member.

[0015] Secondly, this application discloses an abnormal behavior early warning device, comprising: The data acquisition module is used to acquire target data corresponding to the patient through target sensors; the target sensors include a video acquisition sensor, a millimeter-wave radar, and a pressure-sensing mattress; the target data includes video frames at the patient's bedside, millimeter-wave radar data, and pressure-sensing mattress data. The feature vector determination module is used to determine the spatial feature vector based on the video frame through a depthwise separable convolution and a preset coordinate attention module, determine the optical flow feature sequence corresponding to the video frame, and determine the temporal feature vector based on the optical flow feature sequence and a one-dimensional dilated convolutional network. The multimodal fusion module is used to determine a fusion feature vector based on the spatial feature vector and the temporal feature vector, and to perform multimodal fusion of the fusion feature vector, the radar feature vector corresponding to the millimeter-wave radar data, and the pressure feature vector corresponding to the pressure sensor mattress data to obtain a multimodal fusion feature. The early warning operation determination module is used to input the multimodal fusion features into a trained hidden Markov model to obtain the probability value of abnormal patient behavior within a future target time period, and to determine the corresponding early warning operation based on the relationship between the probability value and a preset threshold.

[0016] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute computer programs to implement abnormal behavior warning methods as described above.

[0017] Fourthly, this application discloses a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the aforementioned abnormal behavior warning method.

[0018] This application first collects target data corresponding to the patient using target sensors, including a video acquisition sensor, millimeter-wave radar, and a pressure-sensing mattress. The target data includes video frames from the patient's bedside, millimeter-wave radar data, and pressure-sensing mattress data. A spatial feature vector is determined based on the video frames using depthwise separable convolution and a preset coordinate attention module. An optical flow feature sequence corresponding to the video frames is then determined. A temporal feature vector is determined based on the optical flow feature sequence and a one-dimensional dilated convolutional network. A fusion feature vector is determined based on the spatial and temporal feature vectors. The fusion feature vector, the radar feature vector corresponding to the millimeter-wave radar data, and the pressure feature vector corresponding to the pressure-sensing mattress data are then fused in a multimodal manner to obtain a multimodal fusion feature. This multimodal fusion feature is input into a trained Hidden Markov Model to obtain the probability value of abnormal patient behavior within a future target time period. Based on the relationship between the probability value and a preset threshold, a corresponding warning operation is determined. As can be seen, this application utilizes multi-source asynchronous fusion of video, millimeter-wave radar, and pressure mattresses, combined with spatiotemporal features extracted by depthwise separable convolution, coordinate attention, and one-dimensional dilated temporal convolutional networks, to obtain highly discriminative behavioral representations even under parameter constraints. Furthermore, it dynamically estimates future risk probabilities using a hidden Markov model and triggers early warning responses in stages based on preset thresholds, thus providing an effective intervention window before abnormal events such as falls from beds or extubation actually occur. The overall solution balances real-time performance, accuracy, and deployment economy, significantly improving the advance warning, recall rate, and practicality of abnormal behavior warnings in medical settings compared to traditional post-event alarms or single-modal systems. Through the systematic design of multimodal sensor collaboration and lightweight deep learning models, it achieves low-cost, high-precision, and early warning-enabled real-time monitoring of abnormal patient behavior in an edge computing environment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 This is a flowchart of an abnormal behavior early warning method disclosed in this application; Figure 2 This is a schematic diagram of a threshold adaptive mechanism disclosed in this application; Figure 3 This is a schematic diagram of an encryption system disclosed in this application; Figure 4 This is a schematic diagram of the structure of an abnormal behavior early warning device disclosed in this application; Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Currently, mainstream abnormal behavior detection solutions mainly face the following problems: Existing solutions mostly use standard CNNs to extract features from single frames or short frame sequences, which has limited ability to distinguish between normal nursing operations and abnormal behaviors. In complex nursing scenarios, the false alarm rate remains high, severely impacting nurses' trust in and willingness to use the early warning system. More importantly, these solutions only trigger alarms after the abnormal behavior has occurred, which is a passive response—there is often a gap of several minutes between the occurrence of the abnormality and the arrival of the nurse, missing the optimal intervention opportunity. Many solutions have backbone network parameters exceeding tens of millions, requiring cloud deployment, and the inference latency is insufficient to meet the requirements of real-time bedside early warning. Moreover, cloud deployment also brings bandwidth bottlenecks and data privacy risks. In common situations such as changes in lighting, camera obstruction, or curtains being drawn, the detection accuracy of vision-based solutions drops significantly. Most existing solutions lack the ability to integrate data from multiple sensors, making it difficult to adapt to changing nursing environments. Existing solutions generally rely on foreign operating systems and chips, and encryption components mostly use international algorithms, leaving core components subject to foreign control. Sensitive patient data lacks autonomous and controllable security measures during transmission and storage. To address the aforementioned technical problems, this invention discloses an abnormal behavior early warning method, device, equipment, and medium, which can accurately detect abnormal behavior and trigger early warnings before abnormal events occur, realizing real-time, high-precision, interpretable, and autonomously controllable early warning of abnormal medical and nursing behaviors on edge devices.

[0023] See Figure 1 As shown, this embodiment of the invention discloses an abnormal behavior early warning method, including: Step S11: Collect target data corresponding to the patient through target sensors; the target sensors include video acquisition sensors, millimeter-wave radar and pressure-sensing mattress; the target data includes video frames at the patient's bedside, millimeter-wave radar data and pressure-sensing mattress data.

[0024] In this embodiment, when collecting target data corresponding to the patient, relying solely on vision is prone to failure when there is occlusion or changes in lighting. Therefore, this solution additionally introduces millimeter-wave radar and a pressure-sensing mattress as auxiliary sensing methods. Equipment: Ordinary camera (video), millimeter-wave radar (distance / speed), pressure-sensing mattress (pressure distribution). In a specific embodiment, the camera: takes 30 photos per second, each photo is sent to an edge box. The millimeter-wave radar: measures the patient's distance and speed 100 times per second, for example, "0.2 meters from the bed surface, speed 0". The pressure mattress: measures the pressure distribution 10 times per second, for example, "center of gravity at the waist, maximum pressure zone 150 mmHg". Specifically, the target sensors include a video acquisition sensor, millimeter-wave radar, and a pressure-sensing mattress; the target data includes video frames from beside the patient's bed, millimeter-wave radar data, and pressure-sensing mattress data. The millimeter-wave radar data includes the distance between the patient and the radar, the patient's movement speed, and the patient's azimuth change; the pressure-sensing mattress data includes pressure distribution data. A millimeter-wave radar, installed above the bed, acquires patient distance, speed, and angle information, and extracts radar feature vectors. A pressure-sensing mattress collects statistical features such as pressure center location and distribution entropy, generating pressure feature vectors. Furthermore, since the three modalities acquire data asynchronously at different sampling frequencies, the time corresponding to the video frame is determined as the time base. Based on this time base, timestamp interpolation and alignment operations are performed on the millimeter-wave radar data and the pressure-sensing mattress data to achieve time synchronization of the target data. This ensures that the modalities are temporally corresponding during fusion.

[0025] Step S12: Determine the spatial feature vector based on the video frame using depthwise separable convolution and a preset coordinate attention module, determine the optical flow feature sequence corresponding to the video frame, and determine the temporal feature vector based on the optical flow feature sequence and a one-dimensional dilated convolutional network.

[0026] This application presents a lightweight, dual-stream parallel convolutional neural network (Spatial-Temporal-Locally Aligned Convolutional Neural Network, ST-LACNN) for extracting spatiotemporal features from videos. The spatial stream takes the current video frame as input, replacing standard convolutions with depthwise separable convolutions to significantly compress the number of parameters; a coordinate attention module is embedded in the deep network, enabling the model to simultaneously perceive spatial relationships and inter-channel dependencies. The temporal stream takes optical flow features from multiple consecutive frames as input, aggregating motion information across multiple time scales through a one-dimensional dilated temporal convolutional network (1D-TCN). Finally, spatial and temporal features are adaptively weighted and fused using an SE-Fusion (Squeeze-and-Excitation) channel attention module to generate a highly discriminative 384-dimensional fused feature vector. The overall model's parameter count is kept low, meeting the frame rate requirements for real-time inference on domestically produced edge devices. ST-LACNN employs a dual-stream parallel architecture, processing spatial static features and temporal motion features separately, and finally fusing them through a channel attention module. The model is relatively small and can meet real-time frame rate requirements on domestic edge computing platforms.

[0027] First, the spatial flow takes the current video frame as input and uses depthwise separable convolutions as the basic units to reduce computational overhead. Specifically, the input image is progressively downsampled through four convolutional stages, with the number of channels gradually increasing. Coordinate attention modules are added to the two deeper stages, allowing the model to simultaneously focus on key spatial locations (such as patient limbs and the bedside area) and height information (distinguishing between actions on and off the bed), ultimately outputting a spatial feature vector. Then, regions of interest (ROIs) are extracted from the video frames, and optical flow feature sequences of these ROIs are calculated based on the Lucas-Cannard method. The temporal flow takes the optical flow feature sequences of T frames (default T=16, time window approximately 0.64 seconds) as input and aggregates temporal motion information through a one-dimensional temporal convolutional network (1D-TCN). Specific implementation: (1) Inter-frame motion feature extraction: The Lucas-Kanade optical flow algorithm is used to calculate the motion vector of key points in the patient's ROI (Region of Interest) region (located by spatial flow detection); (2) One-dimensional dilated convolution aggregation: A three-layer one-dimensional dilated convolutional network is used with dilation rates of 1, 2 and 4 to capture motion patterns at different time scales and output a 128-dimensional motion feature vector.

[0028] Step S13: Determine the fusion feature vector based on the spatial feature vector and the temporal feature vector, and perform multimodal fusion on the fusion feature vector, the radar feature vector corresponding to the millimeter-wave radar data, and the pressure feature vector corresponding to the pressure sensor mattress data to obtain the multimodal fusion feature.

[0029] In this embodiment, the spatial feature vector and the temporal feature vector are concatenated to obtain a concatenated feature vector; the concatenated feature vector is then weighted and fused to obtain the fused feature vector. After concatenation of the spatial and temporal features, adaptive weighting is performed by a channel attention module to obtain the fused feature vector. The weights of the two features are automatically adjusted according to the current environment—biasing towards spatial features during the day when there is good lighting, and biasing towards motion features at night when there is insufficient lighting. In addition, the fused feature vector, the radar feature vector corresponding to the millimeter-wave radar data, and the pressure feature vector corresponding to the pressure sensor mattress data are fused in a multimodal manner to obtain a multimodal fused feature.

[0030] Step S14: Input the multimodal fusion features into the trained Hidden Markov Model to obtain the probability value of abnormal patient behavior within the future target time period, and determine the corresponding early warning operation based on the relationship between the probability value and the preset threshold.

[0031] In this embodiment, before inputting the multimodal fusion features into the trained Hidden Markov Model, historical target data is first acquired, and the presence of abnormal behavior in the historical target data within a preset time period is marked. The Hidden Markov Model is then trained based on the marked data to obtain the trained Hidden Markov Model.

[0032] Specifically, the training data is labeled using a bag-level annotation method: only a segment of video is labeled to determine if it contains anomalies, without requiring frame-by-frame annotation. A video segment is considered a "bag," and each frame is an "instance." If at least one frame in the bag contains an anomaly, the entire segment is labeled as a positive sample. During training, the attention module automatically learns the contribution of each frame to the bag-level label and identifies anomalous keyframes. This method significantly reduces the annotation workload. When deployed to new scenarios, the model needs to continuously learn but cannot forget old knowledge. This solution uses an elastic weight consolidation mechanism to address this issue: during fine-tuning, stronger regularization constraints are applied to parameters more important in the old task, limiting their variation. Simultaneously, an old model is retained as a "teacher," and the output of the new model is kept consistent with the old model through loss distillation. Inference and fine-tuning are performed entirely on edge devices, and the training data remains within the domain. Regarding the training data, a multi-instance learning framework is used for weakly supervised training: only video segments are labeled with bag-level annotations (whether they contain anomalies), and the MIL (multi-instance learning) attention network automatically learns frame-level weights to locate anomalous keyframes, significantly reducing the annotation workload. When deployed to new scenarios, a combination of elastic weight consolidation and knowledge distillation allows the model to quickly adapt to the new environment while retaining knowledge from the old scenarios. After optimization through weakly supervised training and incremental learning, the false positive rate is significantly reduced compared to traditional methods.

[0033] Unlike traditional binary classification of normal / abnormal behavior, this invention models abnormal behavior as a three-stage progressive process: latency (L), critical period (C), and outbreak period (E). The system uses a Hidden Markov Model to estimate the probability P_risk of entering the outbreak period within a few future frames based on the current observation sequence. Based on the dynamic changes in P_risk, the system automatically matches differentiated early warning response strategies: in phase L, only recording is performed without alarm; in phase C, an early warning prompt is sent to the nurses' station; and in phase E, emergency broadcasts and automatic alarms are triggered. Simultaneously, the threshold is not fixed but automatically adjusted according to the patient's personal habits and circadian rhythm—for example, the trigger threshold is appropriately lowered at night and raised during the day to reduce false alarms caused by normal nursing actions. Specifically: Latent period (L phase): Abnormal behavior has not yet occurred, but warning signs may appear, such as the patient frequently turning over or limbs protruding from the bed edge. At this time, the risk level (P_risk) is low and still within the range of normal nursing care procedures.

[0034] Critical phase (Phase C): Prodromal symptoms continue to accumulate, and abnormal behavior may occur at any time. P_risk enters the moderate to high range, requiring attention and preventive intervention from nursing staff.

[0035] Outbreak Phase (E Phase): Abnormal behavior has actually occurred, such as falling out of bed or pulling out of a tube. P_risk enters the high-value range and requires immediate action.

[0036] The system uses a hidden Markov model to estimate the state transition probability and predicts the probability P_risk of entering period E in the next few frames based on the current observation sequence.

[0037] If the probability value is less than a first preset threshold, the patient's behavior is determined to be normal, and a Level 1 warning is issued; the Level 1 warning is a screen display warning. If the probability value is greater than or equal to the first preset threshold but less than a second preset threshold, the patient is determined to exhibit abnormal behavior, and a Level 2 warning is issued; the Level 2 warning involves sending warning information to staff and issuing an audible and visual warning. If the probability value is greater than or equal to the second preset threshold, the patient is determined to exhibit abnormal behavior, and a Level 3 warning is issued; the Level 3 warning involves issuing an audible and visual alarm, notifying staff by phone, and sending on-site video to staff. Based on the dynamic changes of P_risk, the system automatically matches differentiated warning response strategies as shown in Table 1.

[0038] Table 1

[0039] In one specific embodiment, if a C-stage warning is issued, the nurse station screen will display a message: "Bed 3, critical stage warning, increased risk of falling out of bed"; the nurse's mobile app will push the same message along with a current video (de-sensitized, faces blurred); the bedside equipment will begin flashing a yellow indicator light; and a voice prompt will play from the bedside speaker: "Hello, do you need help? Please wait for the nurse to come." Upon seeing this, the nurse will go to check on the patient, who has just sat up and is still in bed. The nurse will assist him to use the toilet, preventing a fall.

[0040] It should also be noted that the warning threshold is not a fixed value, but is dynamically adjusted based on the patient's historical behavioral patterns and circadian rhythms, such as... Figure 2 As shown. For example, if the frequency of patient turning over decreases at night, the system will appropriately lower the trigger threshold to improve sensitivity; during the day, it will return to the standard value to reduce false alarms caused by normal nursing operations.

[0041] If a patient is determined to exhibit abnormal behavior, the current video frame image, warning level, probability value, spatial feature vector, temporal feature vector, millimeter-wave radar data, and pressure-sensing mattress data are input into the target large language model to obtain a corresponding natural language warning report. This natural language warning report is then fed back to the staff. This solution also introduces a multimodal large language model as an auxiliary engine to perform further semantic analysis on the warning results, improving interpretability. ST-LACNN serves as the main engine, continuously performing real-time detection and providing the primary basis for system decisions. The large language model, as the auxiliary engine, runs asynchronously in the background, performing in-depth analysis on high-confidence warnings. Information such as warning stage, risk probability, feature vector, and keyframes is exchanged between the two engines. The auxiliary engine executes asynchronously and does not consume the main engine's resources. Auxiliary analysis is automatically triggered when the risk probability reaches a certain threshold. The results of the auxiliary engine do not directly override the main engine's decisions; they are only submitted to the management platform for manual review when there is a significant contradiction between the two.

[0042] The auxiliary engine uses a domestically developed lightweight multimodal model, which is quantized and deployed on edge devices. During inference, only video frames and structural information are loaded; patient identity information is not involved, and the video does not exit the edge nodes. The specific functions of the auxiliary engine include: First, predictive report generation. Upon entering Phase C or Phase E, the system inputs information such as the current frame image, warning level, and features into the large language model to generate a natural language report containing behavioral descriptions, risk assessments, and handling recommendations.

[0043] Second, historical video retrospective annotation. The system automatically extracts video clips before and after the warning, and uses a large language model to perform frame-by-frame anomaly localization and classification. The annotation results are manually reviewed and then fed back into the training dataset, forming a data closed loop.

[0044] Thirdly, there is multimodal conflict arbitration. When visual and radar / pressure sensor signals contradict each other (e.g., the visual system shows the patient getting up but the pressure sensor does not change), the large language model performs comprehensive analysis and provides an arbitration conclusion, explaining the possible reasons. This is more flexible than traditional threshold fusion methods.

[0045] In one specific implementation, the current video frame, the optical flow map of the past 10 seconds, the radar distance curve, the pressure center trajectory, the ST-LACNN output features, and the warning level are input into a large language model. The large language model generates a report: "At 02:00:30, the patient began to roll over to the right from a supine position, with the right arm extending beyond the edge of the bed. The pressure center shifted from the lower back to the right hip. The radar detected a forward tilting speed of 0.1 m / s. Based on the overall assessment, this indicates an intention to get out of bed at night, with a moderate to high risk of falling out of bed. Recommendation: Check if the bed rails are raised and ask the patient if they need to use the toilet." This segment is then marked and added to the training set after manual review.

[0046] As can be seen, this application utilizes multi-source asynchronous fusion of video, millimeter-wave radar, and pressure mattresses, combined with spatiotemporal features extracted by depthwise separable convolution, coordinate attention, and one-dimensional dilated temporal convolutional networks, to obtain highly discriminative behavioral representations even under parameter constraints. Furthermore, it dynamically estimates future risk probabilities using a hidden Markov model and triggers early warning responses in stages based on preset thresholds, thus providing an effective intervention window before abnormal events such as falls from beds or extubation actually occur. The overall solution balances real-time performance, accuracy, and deployment economy, significantly improving the advance warning, recall rate, and practicality of abnormal behavior warnings in medical settings compared to traditional post-event alarms or single-modal systems. Through the systematic design of multimodal sensor collaboration and lightweight deep learning models, it achieves low-cost, high-precision, and early warning-enabled real-time monitoring of abnormal patient behavior in an edge computing environment.

[0047] Medical data has very high security requirements; therefore, the specific encryption system in this application is as follows: Figure 3 As shown. It is equipped with a national cryptographic hardware security module to manage keys and encryption operations. Data encryption protection uses national cryptographic algorithms, covering all aspects of storage, transmission, verification, and auditing. (1) Storage encryption: Video clips and warning records are encrypted using the SM4 algorithm before storage, and the key is managed by the hardware security module; (2) Transmission encryption: The edge nodes and the management platform communicate using the national cryptographic protocol TLCP (Transport layer cryptography protocol), with SM2 for authentication and SM4 for data encryption. (3) Integrity verification: Calculate the SM3 hash digest for data such as early warning records to prevent tampering; (4) Log auditing: Operation logs are stored in SM3 hash chain to ensure that they cannot be tampered with.

[0048] Video data is processed only in the memory of the edge nodes. Reported alerts only include the device ID (identity number), alert level, and timestamp, without patient identification information. At the edge nodes, the video is anonymized: facial areas are blurred, the background is Gaussian blurred, and only the area of ​​patient movement is retained. Access control is hierarchical based on user roles (nurse, doctor, administrator, etc.), and abnormal data is only viewable by the responsible nurse and attending physician.

[0049] Edge computing nodes are network isolated to prevent lateral movement attacks. Each edge device is equipped with an SM2 certificate for two-way authentication with the management platform. Communication between internal system components also uses TLCP authentication.

[0050] In this way, regarding data security, this solution adopts a domestic technology approach from hardware to algorithms. On the hardware side, edge nodes use domestically produced processors and operating systems, along with a national cryptographic hardware security module for key management. Algorithmically, data storage uses SM4 encryption, communication uses the national cryptographic TLCP protocol and SM2 authentication, and integrity verification and log auditing use SM3 hashing. Furthermore, the system implements network isolation for edge nodes, two-way authentication between devices, and video data is anonymized at the edge before being reported, ensuring that only relevant personnel can access warning information.

[0051] As can be seen from the above, the system architecture of this application is divided into four layers: data acquisition layer, edge computing layer, cloud / nurse station management platform layer, and terminal display layer.

[0052] (1) Data acquisition layer: including high-definition camera (1 per bed, FOV (Field of View) ≥120°, 1080P / 25fps, supporting ONVIF (Open Network Video Interface Forum) protocol), millimeter-wave radar module (optional) and pressure-sensing mattress (optional).

[0053] (2) Edge computing layer: Based on the AI ​​(Artificial Intelligence) edge computing node, the ST-LACNN real-time inference engine, multimodal fusion module, progressive early warning engine and large language model auxiliary enhancement engine are run.

[0054] (3) Cloud / Nurse Station Management Platform Layer: Deploy an early warning information management platform to support functions such as Web monitoring screen, early warning information push, data management, access control and system management.

[0055] (4) Terminal display layer: including the large screen at the nurse station displaying early warning information, nurses receiving real-time early warning pushes and viewing on-site videos via mobile APP, and management personnel viewing statistical reports and historical records via the Web terminal.

[0056] In this way, the application can trigger an early warning before an abnormal event occurs, giving nursing staff time to take preventative measures.

[0057] See Figure 4 As shown, an embodiment of the present invention discloses an abnormal behavior early warning device, comprising: Data acquisition module 11 is used to acquire target data corresponding to the patient through target sensors; the target sensors include a video acquisition sensor, a millimeter-wave radar, and a pressure-sensing mattress; the target data includes video frames at the patient's bedside, millimeter-wave radar data, and pressure-sensing mattress data; The feature vector determination module 12 is used to determine the spatial feature vector based on the video frame through a depthwise separable convolution and a preset coordinate attention module, determine the optical flow feature sequence corresponding to the video frame, and determine the temporal feature vector based on the optical flow feature sequence and a one-dimensional dilated convolutional network. The multimodal fusion module 13 is used to determine a fusion feature vector based on the spatial feature vector and the temporal feature vector, and to perform multimodal fusion of the fusion feature vector, the radar feature vector corresponding to the millimeter-wave radar data and the pressure feature vector corresponding to the pressure sensor mattress data to obtain a multimodal fusion feature. The early warning operation determination module 14 is used to input the multimodal fusion features into the trained Hidden Markov Model to obtain the probability value of abnormal patient behavior within a future target time period, and determine the corresponding early warning operation based on the relationship between the probability value and a preset threshold.

[0058] As can be seen, this application utilizes multi-source asynchronous fusion of video, millimeter-wave radar, and pressure mattresses, combined with spatiotemporal features extracted by depthwise separable convolution, coordinate attention, and one-dimensional dilated temporal convolutional networks, to obtain highly discriminative behavioral representations even under parameter constraints. Furthermore, it dynamically estimates future risk probabilities using a hidden Markov model and triggers early warning responses in stages based on preset thresholds, thus providing an effective intervention window before abnormal events such as falls from beds or extubation actually occur. The overall solution balances real-time performance, accuracy, and deployment economy, significantly improving the advance warning, recall rate, and practicality of abnormal behavior warnings in medical settings compared to traditional post-event alarms or single-modal systems. Through the systematic design of multimodal sensor collaboration and lightweight deep learning models, it achieves low-cost, high-precision, and early warning-enabled real-time monitoring of abnormal patient behavior in an edge computing environment.

[0059] In some specific embodiments, the device is further configured to determine the time corresponding to the video frame as a time reference, and perform timestamp interpolation alignment operation on the millimeter-wave radar data and pressure-sensing mattress data based on the time reference to complete the time synchronization of the target data.

[0060] In some specific embodiments, the feature vector determination module 12 is used to extract the region of interest in the video frame and calculate the optical flow feature sequence of the region of interest based on the Lucas-Cannard method.

[0061] In some specific embodiments, the multimodal fusion module 13 is used to concatenate the spatial feature vector and the temporal feature vector to obtain a concatenated feature vector; and to perform weighted fusion on the concatenated feature vector to obtain the fused feature vector.

[0062] In some specific embodiments, the device is further configured to acquire historical target data, label whether there is abnormal behavior in the historical target data within a preset time period, and train the Hidden Markov Model based on the corresponding labeled data to obtain a trained Hidden Markov Model.

[0063] In some specific embodiments, the early warning operation determination module 14 is used to determine that the patient's behavior is normal and issue a level one early warning if the probability value is less than a first preset threshold; the level one early warning is a screen display warning; if the probability value is greater than or equal to the first preset threshold and less than a second preset threshold, it is determined that the patient will exhibit abnormal behavior and issue a level two early warning; the level two early warning is to push early warning information to staff and issue an audible and visual warning; if the probability value is greater than or equal to the second preset threshold, it is determined that the patient exhibits abnormal behavior and issue a level three early warning; the level three early warning is to issue an audible and visual alarm, notify the staff by telephone, and push on-site video to the staff.

[0064] In some specific embodiments, the device is further configured to, if it is determined that the patient will exhibit abnormal behavior or that the patient has exhibited abnormal behavior, input the current video frame image, warning level, probability value, spatial feature vector, temporal feature vector, millimeter-wave radar data, and pressure-sensing mattress data into the target large language model to obtain a corresponding natural language warning report; and to feed back the natural language warning report to the staff.

[0065] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0066] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the abnormal behavior warning method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0067] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0068] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0069] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the abnormal behavior warning method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0070] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned abnormal behavior warning method. The specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0072] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0073] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0074] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0075] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for early warning of abnormal behavior, characterized in that, include: The system collects target data corresponding to the patient through target sensors, including video acquisition sensors, millimeter-wave radar, and pressure-sensing mattresses; the target data includes video frames from the patient's bedside, millimeter-wave radar data, and pressure-sensing mattress data. The spatial feature vector is determined based on the video frame by using depthwise separable convolution and a preset coordinate attention module, the optical flow feature sequence corresponding to the video frame is determined, and the temporal feature vector is determined based on the optical flow feature sequence and a one-dimensional dilated convolutional network. Based on the spatial feature vector and the temporal feature vector, a fusion feature vector is determined. The fusion feature vector, the radar feature vector corresponding to the millimeter-wave radar data, and the pressure feature vector corresponding to the pressure sensor mattress data are then fused in a multimodal manner to obtain a multimodal fusion feature. The multimodal fusion features are input into a trained Hidden Markov Model to obtain the probability value of abnormal patient behavior within a future target time period. Based on the relationship between the probability value and a preset threshold, a corresponding early warning operation is determined.

2. The abnormal behavior early warning method according to claim 1, characterized in that, The millimeter-wave radar data includes the distance between the patient and the radar, the patient's movement speed, and changes in the patient's azimuth angle; the pressure-sensing mattress data includes pressure distribution data. Accordingly, after collecting the target data corresponding to the patient through the target sensor, the process further includes: The time corresponding to the video frame is determined as the time base. Based on the time base, the millimeter-wave radar data and the pressure-sensing mattress data are subjected to timestamp interpolation and alignment operations to complete the time synchronization of the target data.

3. The abnormal behavior early warning method according to claim 1, characterized in that, Determining the optical flow feature sequence corresponding to the video frame includes: Extract the region of interest from the video frame, and calculate the optical flow feature sequence of the region of interest based on the Lucas-Cannard method.

4. The abnormal behavior early warning method according to claim 1, characterized in that, The step of determining the fused feature vector based on the spatial feature vector and the temporal feature vector includes: The spatial feature vector and the temporal feature vector are concatenated to obtain the concatenated feature vector; The concatenated feature vectors are weighted and fused to obtain the fused feature vector.

5. The abnormal behavior early warning method according to claim 1, characterized in that, Before inputting the multimodal fused features into the trained Hidden Markov Model, the process also includes: Historical target data is acquired, and the historical target data within a preset time period is labeled to indicate whether there is any abnormal behavior. The Hidden Markov Model is then trained based on the labeled data to obtain a trained Hidden Markov Model.

6. The abnormal behavior early warning method according to any one of claims 1 to 5, characterized in that, The step of determining the corresponding early warning operation based on the relationship between the probability value and the preset threshold includes: If the probability value is less than the first preset threshold, the patient's behavior is determined to be normal, and a level one warning is issued; the level one warning is a screen display warning. If the probability value is greater than or equal to the first preset threshold and less than the second preset threshold, it is determined that the patient will exhibit abnormal behavior, and a secondary warning is issued; the secondary warning is to push warning information to staff and issue an audio-visual warning. If the probability value is greater than or equal to the second preset threshold, it is determined that the patient has abnormal behavior and a level three warning is issued. The level three warning is to issue an audible and visual alarm, notify the staff by telephone, and push the on-site video to the staff.

7. The abnormal behavior early warning method according to claim 6, characterized in that, Also includes: If it is determined that the patient will exhibit abnormal behavior or that the patient has abnormal behavior, the current video frame image, warning level, probability value, spatial feature vector, temporal feature vector, millimeter-wave radar data, and pressure sensor mattress data are input into the target large language model to obtain the corresponding natural language warning report. The natural language warning report will be fed back to the staff member.

8. An abnormal behavior early warning device, characterized in that, include: The data acquisition module is used to acquire target data corresponding to the patient through target sensors; the target sensors include a video acquisition sensor, a millimeter-wave radar, and a pressure-sensing mattress; the target data includes video frames at the patient's bedside, millimeter-wave radar data, and pressure-sensing mattress data. The feature vector determination module is used to determine the spatial feature vector based on the video frame through a depthwise separable convolution and a preset coordinate attention module, determine the optical flow feature sequence corresponding to the video frame, and determine the temporal feature vector based on the optical flow feature sequence and a one-dimensional dilated convolutional network. The multimodal fusion module is used to determine a fusion feature vector based on the spatial feature vector and the temporal feature vector, and to perform multimodal fusion of the fusion feature vector, the radar feature vector corresponding to the millimeter-wave radar data, and the pressure feature vector corresponding to the pressure sensor mattress data to obtain a multimodal fusion feature. The early warning operation determination module is used to input the multimodal fusion features into a trained hidden Markov model to obtain the probability value of abnormal patient behavior within a future target time period, and to determine the corresponding early warning operation based on the relationship between the probability value and a preset threshold.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the abnormal behavior warning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, and when executed by a processor, the computer program implements the abnormal behavior warning method as described in any one of claims 1 to 7.

Images