Method and system for recognizing abnormal state of critical patient based on dynamic video monitoring

Abstract

The application provides a critical patient abnormal state recognition method and system based on dynamic video monitoring, and belongs to the technical field of video recognition. The method comprises the following steps: obtaining a plurality of dynamic video streams and a plurality of source monitoring data sequences; obtaining a plurality of video frame posture estimation node framework sequences; performing adjacent misfit separation to obtain a plurality of homomeric and heteromeric video frame posture estimation node framework sub-sequence sets, thereby obtaining a plurality of homomeric abnormal state recognition result sets and a plurality of heteromeric abnormal state recognition result sets; and after interactive verification, obtaining a target abnormal state recognition result. The application solves the technical problems in the prior art that a critical patient's normal behavior and abnormal state cannot be accurately distinguished, and real abnormal events such as falls and apnea are easily misjudged as noise signals, and achieves the technical effects of accurately recognizing the abnormal state of a critical patient, effectively avoiding misjudgment of abnormal events as noise, and improving the accuracy and robustness of abnormal state detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video recognition, and particularly relates to a critical patient abnormal state recognition method and system based on dynamic video monitoring. BACKGROUND

[0002] In the intensive care unit, real-time monitoring of patient state is of great significance. Critical patients often face the risk of sudden abnormal states such as falling, apnea, and seizures due to their critical condition. If these abnormal states are not discovered and handled in time, they may lead to serious medical consequences.

[0003] Traditional critical patient monitoring methods mainly rely on contact sensors, such as electrocardiogram monitors, blood oxygen saturation monitors, and other equipment, but they have the disadvantages of causing patient discomfort, sensor falling off, and inability to monitor behavioral abnormalities. In recent years, non-contact patient monitoring methods based on video monitoring have gradually attracted attention, but existing video monitoring technologies face many challenges in practical applications. Existing technologies usually use motion detection, image recognition, and other methods to monitor patients, but these methods often filter the motion signals generated by the normal activities of critical patients (such as turning over, adjusting posture, etc.) as noise, resulting in the suppression of real abnormal events such as falls and apnea as noise signals, causing missed detection of abnormal states. Therefore, the existing technology has the technical problems of being unable to accurately distinguish between normal behavior and abnormal state of critical patients, and easily misjudging real abnormal events such as falls and apnea as noise signals. SUMMARY

[0004] The present application provides a critical patient abnormal state recognition method and system based on dynamic video monitoring to solve the technical problems of being unable to accurately distinguish between normal behavior and abnormal state of critical patients, and easily misjudging real abnormal events such as falls and apnea as noise signals in the prior art.

[0005] The technical solution of the present application to solve the above technical problems is as follows:

[0006] In a first aspect, the present application provides a method for identifying abnormal state of a critical patient based on dynamic video monitoring, comprising: deploying a plurality of video monitoring cameras, obtaining a monitoring starting instruction, synchronously starting the plurality of video monitoring cameras based on the monitoring starting instruction, and obtaining a plurality of dynamic video streams; obtaining a multi-source monitoring data sequence based on the monitoring starting instruction and using remote photoelectric capacitance tracing according to a reflection spectrum signal; obtaining an initial posture estimation node framework of a target critical patient, filling the initial posture estimation node framework for each video frame in the plurality of dynamic video streams, and obtaining a plurality of video frame posture estimation node framework sequences; performing adjacent misfit separation on the plurality of video frame posture estimation node framework sequences, and obtaining a plurality of homofit video frame posture estimation node framework subsequence sets and a plurality of misfit video frame posture estimation node framework subsequence sets; performing double-channel abnormal state identification on the plurality of homofit video frame posture estimation node framework subsequence sets and the plurality of misfit video frame posture estimation node framework subsequence sets respectively, and obtaining a plurality of homofit abnormal state identification result sets and a plurality of misfit abnormal state identification result sets; and performing interactive verification on the plurality of homofit abnormal state identification result sets and the plurality of misfit abnormal state identification result sets in combination with the multi-source monitoring data sequence, and obtaining a target abnormal state identification result.

[0007] In a second aspect, the present application provides a system for identifying abnormal state of a critical patient based on dynamic video monitoring, comprising: a video acquisition module, configured to deploy a plurality of video monitoring cameras, obtain a monitoring starting instruction, synchronously start the plurality of video monitoring cameras based on the monitoring starting instruction, and obtain a plurality of dynamic video streams; a photoelectric monitoring module, configured to obtain a multi-source monitoring data sequence based on the monitoring starting instruction and using remote photoelectric capacitance tracing according to a reflection spectrum signal; a posture estimation module, configured to obtain an initial posture estimation node framework of a target critical patient, fill the initial posture estimation node framework for each video frame in the plurality of dynamic video streams, and obtain a plurality of video frame posture estimation node framework sequences; a sequence separation module, configured to perform adjacent misfit separation on the plurality of video frame posture estimation node framework sequences, and obtain a plurality of homofit video frame posture estimation node framework subsequence sets and a plurality of misfit video frame posture estimation node framework subsequence sets; an abnormal identification module, configured to perform double-channel abnormal state identification on the plurality of homofit video frame posture estimation node framework subsequence sets and the plurality of misfit video frame posture estimation node framework subsequence sets respectively, and obtain a plurality of homofit abnormal state identification result sets and a plurality of misfit abnormal state identification result sets; and an interactive verification module, configured to perform interactive verification on the plurality of homofit abnormal state identification result sets and the plurality of misfit abnormal state identification result sets in combination with the multi-source monitoring data sequence, and obtain a target abnormal state identification result.

[0008] The present application has the following beneficial effects:

[0009] The multiple video monitoring cameras are deployed, a monitoring starting instruction is acquired, the multiple video monitoring cameras are synchronously started based on the monitoring starting instruction, and multiple dynamic video streams are obtained; through the synchronous starting of the multiple cameras, the dynamic behavior of the critical patient can be captured from different angles at the same time, providing comprehensive video data basis for subsequent multi-view posture analysis, and avoiding the problems of occlusion and blind area caused by a single view. Based on the monitoring starting instruction, a multiple-source monitoring data sequence is obtained from the reflectance spectrum signal by using remote photoplethysmography, physiological parameter acquisition synchronized with video monitoring is realized, and independent physiological index basis is provided for subsequent multi-modal verification of abnormal states. An initial posture estimation node framework of the target critical patient is acquired, each video frame in the multiple dynamic video streams is traversed to fill the initial posture estimation node framework, a multiple-video-frame posture estimation node framework sequence is obtained, and a reference frame of the normal state of the patient is established, providing a comparison reference for subsequent identification of abnormal behavior. The multiple-video-frame posture estimation node framework sequence is subjected to adjacent misfit separation, a multiple-same-fit-video-frame posture estimation node framework sub-sequence set and a multiple-misfit-video-frame posture estimation node framework sub-sequence set are obtained, preliminary screening and classification of potential abnormal states are realized, and it is ensured that abnormal signals will not be treated as noise. Double-channel abnormal state identification is respectively performed on the multiple-same-fit-video-frame posture estimation node framework sub-sequence set and the multiple-misfit-video-frame posture estimation node framework sub-sequence set, a multiple-same-fit-abnormal-state identification result set and a multiple-misfit-abnormal-state identification result set are obtained, the identification ability for different types of abnormal states is enhanced, and it is effectively avoided that a real abnormality is misjudged as noise. The multiple-same-fit-abnormal-state identification result set and the multiple-misfit-abnormal-state identification result set are interactively verified in combination with the multiple-source monitoring data sequence, a target abnormal state identification result is obtained, the accuracy and robustness of abnormal state detection are improved, the false positive rate and the false negative rate are effectively reduced, and it is ensured that the abnormal state can be reliably identified.

[0010] Through the above technical solution, the normal behavior and the abnormal state are accurately distinguished through the adjacent misfit separation mechanism, the gradual and sudden abnormalities are respectively captured by using the double-channel identification mechanism, and the interactive verification is performed in combination with the multiple-source physiological monitoring data, the technical problems in the prior art that the normal behavior and the abnormal state of the critical patient cannot be accurately distinguished, and the real abnormal events such as falling and apnea are easily misjudged as noise signals are effectively solved, the technical effects that the abnormal state of the critical patient is accurately identified, the abnormal events are effectively avoided to be misjudged as noise, and the accuracy and robustness of abnormal state detection are improved are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A flowchart of a critical patient abnormal state recognition method based on dynamic video monitoring provided by the present application is shown in the figure;

[0012] Figure 2A structural schematic diagram of the abnormal state recognition system for critical patients based on dynamic video monitoring is provided in the present application.

[0013] In the drawings, the components represented by various reference numerals are as follows:

[0014] The video acquisition module 11, the photoelectric monitoring module 12, the posture estimation module 13, the sequence separation module 14, the abnormality recognition module 15, and the interactive verification module 16. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0016] In the description of the present application, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0017] In the description of the present application, the term "for example" is used to indicate "as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily interpreted as more preferred or more advantageous than other embodiments. The following description is given in order to enable any person skilled in the art to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that those skilled in the art can recognize that the present application can be implemented without using these specific details. In other examples, well-known structures and processes will not be described in detail in order to avoid unnecessary details making the description of the present application obscure. Therefore, the present application is not intended to be limited to the shown embodiments, but is consistent with the broadest scope of principles and features disclosed.

[0018] In one embodiment, as shown in the accompanying drawings, the present application provides an abnormal state recognition method for critical patients based on dynamic video monitoring, comprising: Figure 1

[0019] S1, multiple video monitoring cameras are laid out, a monitoring start instruction is acquired, the multiple video monitoring cameras are synchronously started based on the monitoring start instruction, and multiple dynamic video streams are obtained.

[0020] ​Specifically, first, multiple video monitoring cameras are arranged in the intensive care unit, which monitor the target critical patient from different angles in real time. Specifically, the multiple video monitoring cameras can include but are not limited to a top camera installed directly above the bed, a side-view camera installed on the side of the bed, and a front-view camera installed at the foot of the bed. By arranging video monitoring cameras at multiple angles, the patient's body movements and posture changes can be captured in all directions, avoiding the problems of occlusion or blind spots caused by a single perspective.

[0021] When the monitoring start instruction is received, the multiple video monitoring cameras are synchronously started based on the monitoring start instruction. The monitoring start instruction is a control signal sent through the system control terminal, which is used to trigger the video monitoring system to start working. The monitoring start instruction can be an instruction signal generated by clicking the "start monitoring" button on the monitoring system interface when a medical staff is monitoring a critical patient; or it can be an automatically generated start instruction according to a preset monitoring process when the patient enters the intensive care unit; or it can be an emergency monitoring start instruction automatically triggered when the patient's vital signs are detected to fluctuate. Synchronous start means that each video monitoring camera starts recording at the same time, ensuring that video streams from different perspectives remain consistent on the time axis, facilitating subsequent time alignment and fusion processing of multi-view data. The synchronization start can be realized through the network time protocol or a special hardware trigger signal, so that the timestamp error of each video monitoring camera is controlled within the millisecond level.

[0022] Through the above synchronous start operation, multiple dynamic video streams are obtained through multiple video monitoring cameras, each dynamic video stream corresponding to real-time video data of a video monitoring camera. Each dynamic video stream contains a continuous video frame sequence, each video frame sequence recording the image information of the patient at a specific time, providing a basic data source for subsequent posture estimation and abnormal state recognition.

[0023] S2, based on the monitoring start instruction, obtaining a multi-source monitoring data sequence according to the reflectance spectrum signal by remote photoelectric plethysmography.

[0024] Specifically, remote photoelectric plethysmography is a non-contact physiological monitoring technology based on optical principles. The remote photoelectric plethysmography captures the reflectance spectrum signal of the patient's face or exposed skin area, analyzes the slight color changes on the skin surface to extract physiological parameter information. When the heart beats, the blood flowing through the skin microvessels will cause periodic changes in the skin color, which is difficult to detect with the naked eye, but can be captured by high-sensitivity optical sensors. The video monitoring camera is used to capture the patient's facial video image, and the RGB color channel information of a specific skin area in the video frame is extracted. By analyzing the change rule of the color channel over time, the reflectance spectrum signal is obtained.

[0025] Based on the reflection spectrum signals obtained by each video monitoring camera, the respiratory frequency and heart rate data of the patient are extracted through a signal processing algorithm. Specifically, the reflection spectrum signals are filtered, denoised and spectrum analyzed, and the frequency components of the respiratory signal and the heartbeat signal are separated from them. Because respiration and heartbeat have different physiological cycle characteristics, they exhibit different frequency ranges in the frequency domain: normal respiratory frequency is usually 6-30 times per minute, corresponding to a frequency range of 0.1-0.5 Hz; normal heart rate is usually 48-180 times per minute, corresponding to a frequency range of 0.8-3 Hz. Based on the above differences in frequency characteristics, by setting bandpass filters of different frequency ranges, the respiratory signal and the heartbeat signal can be effectively separated, and the respiratory frequency data and the heart rate data can be obtained respectively.

[0026] Through continuous monitoring and data acquisition, a multi-source monitoring data sequence is obtained. The multi-source monitoring data sequence includes a plurality of respiratory frequency data sequences and heart rate data sequences extracted by the video monitoring cameras respectively, each video monitoring camera corresponds to a set of monitoring data sequences, and each data point corresponds to a physiological parameter value at a specific time. These multi-source monitoring data sequences are synchronized with the dynamic video streams obtained by the corresponding video monitoring cameras in the time axis, providing data support at the physiological signal level for subsequent multi-modal data fusion and interactive verification.

[0027] S3, an initial pose estimation node framework of the target critical patient is obtained, and each video frame in the plurality of dynamic video streams is traversed to fill the initial pose estimation node framework, thereby obtaining a plurality of video frame pose estimation node framework sequences.

[0028] Specifically, the initial pose estimation node framework refers to a framework structure constructed by the joint nodes of the target patient in the most comfortable state. The initial pose estimation node framework is obtained by pose estimation of the initial state of the target patient, and includes the main joint nodes of the patient's body, such as the head node, the neck node, the shoulder node, the elbow node, the wrist node, the hip node, the knee node and the ankle node, etc. The joint nodes are connected by bone connection lines to form the topology structure of the human skeleton. The initial pose estimation node framework serves as a reference framework for subsequent tracking and abnormal identification of the patient's pose changes.

[0029] After obtaining the initial pose estimation node framework, each video frame in multiple dynamic video streams is traversed, and pose estimation processing is performed on each video frame. Specifically, for each video frame, the joint node positions of the patient's body are detected using a pose estimation algorithm, and the detected joint node coordinates are filled into the corresponding node positions in the initial pose estimation node framework, thereby obtaining the pose estimation node framework for that video frame. The pose estimation algorithm can employ a deep learning-based human pose estimation model, such as OpenPose or HRNet, which can accurately identify the joint node position coordinates of the patient in the video frame.

[0030] By sequentially performing the above pose estimation processing on all video frames in each dynamic video stream, multiple video frame pose estimation node framework sequences are obtained. Each dynamic video stream corresponds to one video frame pose estimation node framework sequence, which arranges the pose estimation node frameworks of each video frame in the video stream in chronological order. Each video frame pose estimation node framework records the joint node position information of the patient at the corresponding time, reflecting the patient's pose change trajectory and providing a skeleton sequence data foundation for subsequent abnormal state identification.

[0031] S4. Perform adjacency-discordation on the multiple video frame attitude estimation node framework sequences to obtain multiple sets of identical video frame attitude estimation node framework subsequences and multiple sets of discordant video frame attitude estimation node framework subsequences.

[0032] Specifically, adjacency-discordant separation refers to comparing and analyzing temporally adjacent video frame pose estimation node frames in a video frame pose estimation node frame sequence to determine whether there are any abrupt changes in pose between adjacent frames. A homologous video frame pose estimation node frame subsequence refers to a subsequence composed of continuous video frame pose estimation node frames with smooth pose changes conforming to normal motion patterns, reflecting the patient's normal continuous movements, such as slow turning over or subtle movements during steady breathing. A discordant video frame pose estimation node frame subsequence refers to a subsequence composed of video frame pose estimation node frames containing abrupt pose changes or abnormal motion patterns, reflecting the patient's abnormal movements or state changes, such as sudden falls, convulsions, or respiratory arrest.

[0033] For each video frame attitude estimation node frame sequence in a series of video frame attitude estimation node frames, the first and second video frame attitude estimation node frames, arranged chronologically, are extracted sequentially. Adjacency-mismatch separation verification is performed on these two temporally adjacent attitude frames. If the verification result shows normal attitude change between the two frames without significant abrupt changes, the verification fails, and the first and second video frame attitude estimation node frames are added to the matching video frame attitude estimation node frame subsequence. If the verification result shows abrupt attitude changes between the two frames, the verification succeeds, and the first video frame attitude estimation node frame is added to the mismatch video frame attitude estimation node frame subsequence. Subsequently, using the second video frame attitude estimation node frame as a benchmark, adjacency-mismatch separation verification is performed on the third video frame attitude estimation node frame, and so on, until all frames of the video frame attitude estimation node frame sequence have been traversed.

[0034] Through the above processing, a video frame pose estimation node framework sequence is divided into several matching video frame pose estimation node framework subsequences and several dismatching video frame pose estimation node framework subsequences. The above adjacency-dismatch separation operation is then performed on multiple video frame pose estimation node framework sequences, ultimately resulting in multiple sets of matching video frame pose estimation node framework subsequences and multiple sets of dismatching video frame pose estimation node framework subsequences. This separation process can distinguish between continuous normal action sequences and abnormal action sequences containing abrupt changes, providing a data foundation for subsequent targeted analysis using different identification strategies.

[0035] S5. Perform dual-channel abnormal state identification on multiple sets of pose estimation node framework subsequences of the same video frames and multiple sets of pose estimation node framework subsequences of different video frames respectively, to obtain multiple sets of abnormal state identification results of the same and multiple sets of abnormal state identification results of different.

[0036] Specifically, the pose estimation node framework subsequence of the same-match video frame reflects the patient's continuous and smooth normal movements, while the pose estimation node framework subsequence of the different-match video frame reflects abnormal movements containing abrupt pose changes. These two types of subsequences have different temporal characteristics and abnormal patterns, therefore, different recognition strategies are required for processing them separately. A dual-channel abnormal state recognition mechanism is constructed for each of these two types of subsequences. The dual-channel refers to constructing two parallel recognition channels, each using a different temporal receptive field to capture abnormal features at different time scales.

[0037] For the set of pose estimation node framework subsequences in matched video frames, a dual-channel system for identifying matched abnormal states is constructed. First, the recognition bandwidths of the first and second matched channels are obtained and used as the temporal receptive fields for the first and second matched abnormal state identification channels, respectively. The first matched abnormal state identification channel uses a larger temporal receptive field, enabling analysis of pose change trends over a longer time span, and is used to identify progressive abnormal states, such as a patient's gradually slowing respiratory rate or decreasing limb movement. The second matched abnormal state identification channel uses a smaller temporal receptive field, focusing on short-term pose detail changes, and is used to identify potential local abnormal fluctuations in the matched sequence. The two channels process the input matched video frame pose estimation node framework subsequences in parallel, extracting features at different time scales. Their respective outputs are simultaneously connected to the first fully connected layer for feature fusion, and the matched abnormal state identification result is output through a classifier.

[0038] For the set of pose estimation node framework subsequences in mismatched video frames, a dual-channel system for mismatched abnormal state recognition is constructed. First, the recognition bandwidths of the first and second mismatched channels are obtained and used as the temporal receptive fields for the first and second mismatched abnormal state recognition channels, respectively. The first mismatched abnormal state recognition channel uses a larger temporal receptive field to analyze the contextual information before and after the abnormal action, determining whether the pose change constitutes a true abnormal state. The second mismatched abnormal state recognition channel uses a smaller temporal receptive field to accurately capture the instantaneous features of the pose change, used to identify sudden abnormal states, such as a patient suddenly falling off the bed or sudden convulsions. The two channels process the input mismatched video frame pose estimation node framework subsequences in parallel, extracting features at different time scales. Their respective outputs are simultaneously connected to a second fully connected layer for feature fusion, and the mismatched abnormal state recognition result is output through a classifier.

[0039] Through the aforementioned dual-channel anomaly state recognition processing, multiple sets of pose estimation node framework sub-sequences from matching video frames are identified to obtain multiple sets of matching anomaly state recognition results; similarly, multiple sets of pose estimation node framework sub-sequences from dismatched video frames are identified to obtain multiple sets of dismatched anomaly state recognition results. The dual-channel architecture can simultaneously capture anomaly features at different time scales, avoiding omissions or misjudgments of anomaly events due to a single time scale, thus improving the accuracy and completeness of anomaly state recognition.

[0040] S6. Combine the multi-source monitoring data sequence to perform interactive verification on the multiple sets of same-matching abnormal state identification results and the multiple sets of dismatching abnormal state identification results to obtain the target abnormal state identification result.

[0041] Specifically, cross-validation refers to using physiological signal data obtained through remote photoplethysmography to verify and correct the abnormal state identification results obtained from video posture analysis. Since video posture analysis may be affected by factors such as occlusion and changes in lighting, leading to misjudgments, while physiological signal data can reflect the patient's true state from another dimension, cross-validating the data from these two modalities can effectively improve the accuracy of abnormal state identification.

[0042] First, based on the time intervals corresponding to the sets of same-match and different-match abnormal state identification results, the multi-source monitoring data sequences are time-aligned and extracted. Specifically, for each same-match abnormal state identification result, respiratory rate and heart rate data within the corresponding time period are extracted to obtain an extracted same-match multi-source monitoring data subsequence; for each different-match abnormal state identification result, respiratory rate and heart rate data within the corresponding time period are extracted to obtain an extracted different-match multi-source monitoring data subsequence. Through this extraction operation, multiple sets of extracted same-match multi-source monitoring data subsequences and multiple sets of extracted different-match multi-source monitoring data subsequences are obtained, ensuring that the physiological signal data and video recognition results correspond precisely in time.

[0043] Secondly, an anomaly detector is invoked to identify anomalies in the extracted multi-source monitoring data sub-sequences. This anomaly detector is a classification model trained on physiological signal features, capable of determining whether a patient is in an abnormal state based on changes in respiratory rate and heart rate. For same-match scenarios, the anomaly detector analyzes multiple extracted same-match multi-source monitoring data sub-sequences to identify any abnormal physiological features such as abnormally slowed respiratory rate or abnormal heart rate fluctuations, obtaining multiple sets of anomaly identification results. For different-match scenarios, the anomaly detector analyzes multiple extracted different-match multi-source monitoring data sub-sequences to identify any physiological signal mutations corresponding to postural changes, obtaining multiple sets of anomaly identification results.

[0044] Subsequently, the results of multi-source monitoring anomaly identification and video pose anomaly identification are matched and verified. For same-match scenarios, the set of same-match multi-source monitoring anomaly identification results and the set of same-match anomaly identification results are matched one by one. If the physiological signal data of a certain same-match anomaly identification result also shows anomalies in the corresponding time period, the matching and identification are successful, and the same-match anomaly identification result is added to the first anomaly identification result set; if the physiological signal data is normal, it is determined to be a video recognition misjudgment and is not adopted. For different-match scenarios, the same matching and verification logic is used, and the verified different-match anomaly identification results are added to the second anomaly identification result set.

[0045] Subsequently, the first and second sets of abnormal state identification results are summarized to obtain the target abnormal state identification result. This target abnormal state identification result integrates information from two dimensions: video posture analysis and physiological signal monitoring. Through interactive verification of multimodal data, it effectively filters out misjudgments that may arise from single-modal data, improves the accuracy and robustness of abnormal state identification, and provides reliable technical support for the real-time monitoring of critically ill patients.

[0046] Furthermore, adjacency-matching separation is performed on the multiple video frame pose estimation node framework sequences to obtain multiple sets of identically matched video frame pose estimation node framework subsequences and multiple sets of dissimilarly matched video frame pose estimation node framework subsequences, including:

[0047] S41. Extract the first video frame pose estimation node frame sequence from the plurality of video frame pose estimation node frame sequences;

[0048] S42. Perform adjacency and heterogeneous separation verification on the first video frame attitude estimation node framework and the second video frame attitude estimation node framework in the first video frame attitude estimation node framework sequence. If the verification fails, add the first video frame attitude estimation node framework and the second video frame attitude estimation node framework into the first homogeneous video frame attitude estimation node framework subsequence.

[0049] S43. If the verification is successful, add the first video frame pose estimation node framework to the first mismatched video frame pose estimation node framework subsequence.

[0050] S44. Based on the second video frame attitude estimation node framework, the third video frame attitude estimation node framework in the first video frame attitude estimation node framework sequence is separated into adjacent and heterogeneous pairs until the end of the first video frame attitude estimation node framework sequence is reached, thereby obtaining the first homogeneous video frame attitude estimation node framework subsequence set and the first heterogeneous video frame attitude estimation node framework subsequence set.

[0051] In a preferred embodiment, firstly, a first video frame pose estimation node frame sequence is extracted from multiple video frame pose estimation node frame sequences. Here, "first" in "first video frame pose estimation node frame sequence" does not refer to the first sequence in order, but rather to any sequence among the multiple video frame pose estimation node frame sequences. Since there are multiple video surveillance cameras, each corresponding to one video frame pose estimation node frame sequence, it is necessary to perform adjacency-discordant separation processing on each video frame pose estimation node frame sequence sequentially. This section uses any one of these video frame pose estimation node frame sequences as an example for detailed explanation; this selected sequence is referred to as the first video frame pose estimation node frame sequence.

[0052] Then, adjacency-discordant separation verification is performed on the first and second video frame attitude estimation node frames in the first video frame attitude estimation node frame sequence. The first and second video frame attitude estimation node frames refer to the first two video frame attitude estimation node frames arranged chronologically in the first video frame attitude estimation node frame sequence. The purpose of adjacency-discordant separation verification is to determine whether there are abnormal attitude abrupt changes between these two temporally adjacent video frame attitude estimation node frames. This verification process calculates characteristic parameters such as the change amplitude of joint node positions, motion speed, and similarity of attitude structures between the two video frame attitude estimation node frames to determine whether the attitude change between adjacent frames exceeds the normal range. If the adjacency-discordant separation verification fails, it indicates that the attitude change between the first and second video frame attitude estimation node frames is within the normal range, without significant abrupt changes, and the two frames exhibit a continuous and smooth attitude transition, belonging to a homologous relationship. At this time, the first and second video frame attitude estimation node frames are added to the first homologous video frame attitude estimation node frame subsequence. The first homologous video frame pose estimation node framework subsequence is the first homologous subsequence divided from the first video frame pose estimation node framework sequence, used to store the pose estimation node framework of consecutive normal video frames in the sequence.

[0053] If the adjacency-disjoint pair separation verification is successful, it indicates that there is an abnormal attitude change between the first video frame attitude estimation node framework and the second video frame attitude estimation node framework. The attitude change between the two frames exceeds the normal range, which is a disjoint pair relationship. At this time, the first video frame attitude estimation node framework is added to the first disjoint video frame attitude estimation node framework subsequence. The first disjoint video frame attitude estimation node framework subsequence is the first disjoint subsequence divided from the first video frame attitude estimation node framework sequence, used to store the attitude estimation node frameworks of the video frames in the sequence where attitude changes occur. It should be noted that only the first video frame attitude estimation node framework is added to the disjoint subsequence at this time, because the first video frame attitude estimation node framework is the last frame before the change occurs, while the second video frame attitude estimation node framework represents the new state after the change and will serve as the starting benchmark for subsequent analysis.

[0054] Subsequently, adjacency-discordant separation verification is performed on the pose estimation node frames of subsequent video frames in the first video frame pose estimation node frame sequence. Specifically, regardless of whether the adjacency-discordant separation verification result in step S42 or step S43 is successful or unsuccessful, the second video frame pose estimation node frame is always used as the new reference frame, and adjacency-discordant separation verification is performed on the third video frame pose estimation node frame in the first video frame pose estimation node frame sequence, that is, to analyze whether there is a change in pose between the second and third video frame pose estimation node frames.

[0055] If the adjacency separation verification between the second and third video frame pose estimation node frames fails, it indicates that the pose change between these two frames is normal. In this case, the third video frame pose estimation node frame is added to the currently constructed matching video frame pose estimation node frame subsequence. If, at this point, the verification between the first and second video frame pose estimation node frames also fails, then the first, second, and third video frame pose estimation node frames all belong to the same matching video frame pose estimation node frame subsequence, forming a continuous normal pose sequence segment.

[0056] If the adjacency separation verification between the second and third video frame pose estimation node frames is successful, it indicates that there is a pose change between the two frames. Then, the second video frame pose estimation node frame is added to the heterogeneous video frame pose estimation node frame subsequence. At this point, we need to handle different cases: If the verification between the first video frame attitude estimation node framework and the second video frame attitude estimation node framework is successful (i.e., the case in step S43), then the first video frame attitude estimation node framework has been added to the heterogamete sequence, and the second video frame attitude estimation node framework constitutes a new homogamete video frame attitude estimation node framework subsequence (although this subsequence contains only a single frame, and such continuous mutation is relatively rare in practical applications); if the verification between the first video frame attitude estimation node framework and the second video frame attitude estimation node framework fails (i.e., the case in step S42), then the first video frame attitude estimation node framework and the second video frame attitude estimation node framework have formed a homogamete video frame attitude estimation node framework subsequence, and the second video frame attitude estimation node framework, as the last frame of this homogamete subsequence and also the starting frame of the mutation, is added to the heterogamete video frame attitude estimation node framework subsequence.

[0057] The pose estimation node frames in the first video frame pose estimation node frame sequence are processed sequentially according to the above rules. That is, after each verification, the pose estimation node frame of the video frame that is later in the current verification is always used as the reference frame for the next verification, and the adjacent mismatch separation verification is performed on the next video frame pose estimation node frame that follows. This process continues until the end of the first video frame pose estimation node frame sequence is reached, that is, the last video frame pose estimation node frame in the sequence is processed.

[0058] Through the aforementioned frame-by-frame progressive adjacency-dismatch separation and verification process, the first video frame attitude estimation node framework sequence is divided into several consecutive homologous video frame attitude estimation node framework subsequences and several dismatched video frame attitude estimation node framework subsequences containing attitude abrupt change points. Summarizing these division results yields a set of first homologous video frame attitude estimation node framework subsequences and a set of first dismatched video frame attitude estimation node framework subsequences. The set of first homologous video frame attitude estimation node framework subsequences contains all homologous video frame attitude estimation node framework subsequences extracted from the first video frame attitude estimation node framework sequence, and the set of first dismatched video frame attitude estimation node framework subsequences contains all dismatched video frame attitude estimation node framework subsequences extracted from the first video frame attitude estimation node framework sequence.

[0059] The above-described adjacency-dismatch separation process is performed on each video frame pose estimation node framework sequence in multiple video frame pose estimation node framework sequences, ultimately obtaining multiple sets of homo-matched video frame pose estimation node framework sub-sequences and multiple sets of dismatched video frame pose estimation node framework sub-sequences. This frame-by-frame progressive verification method can accurately capture the location and characteristics of abrupt change points in the pose sequence, achieving precise division between homo-matched and dismatched sequences, laying a data foundation for subsequent targeted abnormal state identification using different recognition strategies.

[0060] Furthermore, adjacency-discordant separation verification is performed on the first video frame pose estimation node framework and the second video frame pose estimation node framework in the first video frame pose estimation node framework sequence, including:

[0061] S421. Perform mapping node similarity analysis on the first video frame pose estimation node framework and the second video frame pose estimation node framework to obtain a mapping node similarity set.

[0062] S422. Normalize the similarity set of the mapped nodes and construct an adjacency-discordant separation matrix based on the normalization result;

[0063] S423. Call the adjacency-discordant separation verifier to perform separation verification on the adjacency-discordant separation matrix and output the separation verification result, wherein the separation verification result includes verification success and verification failure.

[0064] In a preferred embodiment, firstly, a mapping node similarity analysis is performed on the pose estimation node frames of the first and second video frames to obtain a mapping node similarity set. Specifically, the mapping node similarity analysis refers to evaluating the magnitude of pose change between two temporally adjacent pose frames by calculating the similarity between corresponding joint nodes in the first and second video frame pose estimation node frames. Both the first and second video frame pose estimation node frames contain multiple joint nodes of the patient's body, such as head nodes, neck nodes, shoulder nodes, elbow nodes, wrist nodes, hip nodes, knee nodes, and ankle nodes. For each pair of corresponding joint nodes in the two pose frames, the similarity value between the pair of nodes is calculated. Node similarity can be calculated in various ways. One approach is to calculate the Euclidean distance between corresponding joint nodes in space; a smaller distance indicates higher similarity. Another approach is to calculate the difference in motion velocity between corresponding joint nodes; a smaller velocity difference indicates higher similarity. Alternatively, multiple factors can be considered, such as changes in node position, changes in motion direction, and relative positional relationships with adjacent nodes, and a weighted fusion can be used to obtain a comprehensive similarity value. By calculating the similarity of all corresponding joint node pairs, a mapping node similarity set is obtained. This mapping node similarity set contains the similarity values ​​of all corresponding joint node pairs between the pose estimation node frame of the first video frame and the pose estimation node frame of the second video frame.

[0065] Then, normalization is performed on the similarity set of mapped nodes, and an adjacency-dissimilarity separation matrix is ​​constructed based on the normalization results. Specifically, since the similarity values ​​of different joint node pairs may fall within different numerical ranges, to facilitate subsequent unified processing and analysis, all similarity values ​​in the mapped node similarity set are first normalized. Normalization maps each similarity value to a unified numerical interval, such as [0,1], where 0 represents complete dissimilarity and 1 represents complete similarity. Common normalization techniques such as max-min normalization or standardization can be used. After normalization, an adjacency-dissimilarity separation matrix is ​​constructed based on the normalization results. The adjacency-dissimilarity separation matrix is ​​an initially empty matrix structure used to represent the joint node mapping relationship between the first video frame pose estimation node framework and the second video frame pose estimation node framework. The rows and columns of this adjacency-dissimilarity separation matrix correspond to each joint node in the first and second video frame pose estimation node frameworks, respectively, and the element values ​​in the matrix are the normalized similarity values ​​of the corresponding joint node pairs. The normalized similarity values ​​are then sequentially filled into the corresponding positions of the adjacency-disjoint pair separation matrix to construct the complete adjacency-disjoint pair separation matrix.

[0066] The adjacency-discordant separation matrix is ​​constructed based on the principle of joint node mapping correspondence. In human pose analysis, there is a correspondence between joint nodes in pose frames at different times. This correspondence can be quantitatively represented by constructing an adjacency matrix. Specifically, the rows and columns of the adjacency-discordant separation matrix correspond to the joint nodes in the pose estimation node frames of the first and second video frames, respectively. The element values ​​in the matrix are the normalized similarity values ​​of the corresponding joint node pairs. This matrix establishes the mapping relationship between the joint nodes in the pose estimation node frames of the first and second video frames, and the matrix element values ​​reflect the degree of similarity between the corresponding nodes. When the joint node positions of two adjacent pose frames change little, the corresponding matrix element value is larger, indicating a smooth pose change; when the joint node positions change significantly, the corresponding matrix element value is smaller, indicating a possible abrupt pose change. Therefore, the adjacency-discordant separation matrix provides a quantitative basis for subsequent judgment of whether abnormal pose changes have occurred.

[0067] Subsequently, the adjacency-disjoint match separation verifier is invoked to perform separation verification on the adjacency-disjoint match separation matrix, and the separation verification result is output, which includes verification success and verification failure. Specifically, the adjacency-disjoint match separation verifier is a classification model based on a neural network, used to determine whether the pose change represented by the adjacency-disjoint match separation matrix belongs to an anomalous mutation.

[0068] The construction process of the adjacency-disjoint pair separation validator is as follows: First, a large amount of historical video monitoring data of critically ill patients is collected and manually labeled to mark samples with normal pose changes and abnormal pose mutations. For each sample, the pose estimation node framework of the video frames of the two adjacent frames is extracted, and the corresponding adjacency-disjoint pair separation matrix is ​​constructed according to the methods in steps S421 and S422. Samples with normal pose changes are labeled as the "verification failed" category, indicating that there is no disjoint pair; samples with abnormal pose mutations are labeled as the "verification successful" category, indicating that there is a disjoint pair. Based on the labeled training samples, a neural network model is constructed as the adjacency-disjoint pair separation validator. This neural network model can adopt a convolutional neural network architecture because the adjacency-disjoint pair separation matrix is ​​essentially a two-dimensional matrix structure, and convolutional neural networks can effectively extract spatial feature patterns in the matrix. Specifically, the neural network model includes multiple convolutional layers for extracting local features of the adjacency-disjoint pair separation matrix, multiple pooling layers for dimensionality reduction and enhancing the robustness of features, and fully connected layers for mapping the extracted features to the classification results. The output layer of the network uses the softmax activation function to output the probability values ​​of the two categories, corresponding to "verification failed" and "verification successful," respectively. The neural network model is trained using labeled training samples. During training, the adjacency-disjoint pair separation matrix is ​​used as input, and the labeled class labels are used as supervision signals. The network parameters are optimized using the backpropagation algorithm, enabling the network to learn the feature differences between normal pose changes and abrupt pose changes in the adjacency-disjoint pair separation matrix. After training, the model is validated and tested to ensure that it can accurately determine the nature of pose changes even on unseen data.

[0069] In practical applications, the adjacency-discordant separation matrix constructed in step S422 is input into the trained adjacency-discordant separation verifier. The adjacency-discordant separation verifier performs feature extraction and classification judgment on the adjacency-discordant separation matrix and outputs the separation verification result. If the adjacency-discordant separation verifier determines that the pose change corresponding to the adjacency-discordant separation matrix is ​​an abrupt change, it outputs "verification failed," indicating that there is a discordant relationship between the pose estimation node framework of the first video frame and the pose estimation node framework of the second video frame; if it determines that the pose change is normal, it outputs "verification successful," indicating that there is a homologous relationship between the pose estimation node framework of the first video frame and the pose estimation node framework of the second video frame.

[0070] Through the above process of mapping node similarity analysis, adjacency-discordant separation matrix construction, and adjacency-discordant separation verifier judgment, it is possible to accurately identify whether there are abnormal attitude changes between adjacent video frame attitude estimation node frames, thereby realizing the function of adjacency-discordant separation verification. This provides a reliable basis for subsequent division of the video frame attitude estimation node frame sequence into homologous video frame attitude estimation node frame subsequences and heterologous video frame attitude estimation node frame subsequences.

[0071] Furthermore, dual-channel abnormal state identification is performed on multiple sets of pose estimation node framework subsequences for matching video frames and multiple sets of pose estimation node framework subsequences for dismatched video frames, respectively, to obtain multiple sets of identical abnormal state identification results and multiple sets of dismatched abnormal state identification results, including:

[0072] S51. Obtain a set of channel identification bandwidths, wherein the set of channel identification bandwidths includes a first matching channel identification bandwidth, a second matching channel identification bandwidth, a first dissimilar channel identification bandwidth, and a second dissimilar channel identification bandwidth.

[0073] S52. Using the first matching channel recognition bandwidth and the second matching channel recognition bandwidth as receptive fields respectively, construct the first matching abnormal state recognition channel and the second matching abnormal state recognition channel, connect the first matching abnormal state recognition channel and the second matching abnormal state recognition channel in parallel, and connect the output to the first fully connected layer at the same time to obtain the matching abnormal state recognition dual channel.

[0074] S53. Using the first mismatch channel recognition bandwidth and the second mismatch channel recognition bandwidth as receptive fields respectively, construct the first mismatch abnormal state recognition channel and the second mismatch abnormal state recognition channel, connect the first mismatch abnormal state recognition channel and the second mismatch abnormal state recognition channel in parallel, and connect the output to the second fully connected layer at the same time to obtain the mismatch abnormal state recognition dual channel.

[0075] S54. Using the dual-channel identification of the same-matching abnormal state, perform dual-channel abnormal state identification on the set of multiple same-matching video frame pose estimation node framework sub-sequences to obtain a set of multiple same-matching abnormal state identification results.

[0076] S55. Using the dual-channel identification of heterogeneous abnormal states, perform dual-channel abnormal state identification on the multiple heterogeneous video frame pose estimation node frame sub-sequence sets respectively to obtain the multiple heterogeneous abnormal state identification result sets.

[0077] In a preferred embodiment, firstly, a channel identification bandwidth set is obtained, wherein the channel identification bandwidth set includes a first homologous channel identification bandwidth, a second homologous channel identification bandwidth, a first heterologous channel identification bandwidth, and a second heterologous channel identification bandwidth. Channel identification bandwidth refers to the size of the temporal receptive field of the identification channel when processing pose sequences, i.e., the number of video frames or the time span analyzed by the identification channel at one time. Different channel identification bandwidths correspond to different time scales; larger channel identification bandwidths can capture pose change patterns with longer durations, while smaller channel identification bandwidths can capture rapidly occurring instantaneous pose changes. Since homologous video frame pose estimation node framework subsequences and heterologous video frame pose estimation node framework subsequences have different temporal characteristics, different channel identification bandwidths need to be set for these two types of subsequences.

[0078] The first and second matching channel recognition bandwidths are receptive field parameters with two different time scales set for the pose estimation node framework subsequence of matching video frames. The first matching channel recognition bandwidth is set to a larger time span to analyze the pose evolution trend over a longer period in the matching subsequence, such as progressive abnormal states like a patient's gradually slowing respiratory rate and decreasing limb movement. The second matching channel recognition bandwidth is set to a smaller time span to capture short-term pose detail changes in the matching sequence, such as local abnormal fluctuations that appear in a patient during continuous normal movements.

[0079] The first and second heterogametic channel recognition bandwidths are receptive field parameters with two different time scales set for the pose estimation node framework subsequence of heterogametic video frames. The first heterogametic channel recognition bandwidth is set to a larger time span to analyze the contextual information before and after the pose change in the heterogametic subsequence, and to determine whether the pose change constitutes a true abnormal state, such as distinguishing between a patient's active turning over and an abnormal event of suddenly falling off the bed. The second heterogametic channel recognition bandwidth is set to a smaller time span to accurately capture the instantaneous features of the pose change in the heterogametic subsequence, and to quickly identify sudden abnormal states, such as acute abnormal events such as sudden convulsions, convulsions, or respiratory arrest in patients.

[0080] The identification bandwidths of the first matching channel, the second matching channel, the first dissimilar channel, and the second dissimilar channel are aggregated to obtain the channel identification bandwidth set, which provides a parameter basis for the subsequent construction of a dual-channel abnormal state identification architecture.

[0081] Then, the first and second matching channel recognition bandwidths are used as receptive fields to construct a first matching abnormal state recognition channel and a second matching abnormal state recognition channel, respectively. These two channels are then connected in parallel, and their outputs are simultaneously connected to the first fully connected layer to obtain a dual-channel matching abnormal state recognition system. Specifically, the first matching abnormal state recognition channel is a recognition channel constructed based on the first matching channel recognition bandwidth. This first matching abnormal state recognition channel uses the first matching channel recognition bandwidth as its temporal receptive field. When processing the matching video frame pose estimation node framework subsequence, it extracts the pose estimation node framework of consecutive video frames within the time span corresponding to the first matching channel recognition bandwidth as input each time. The first matching abnormal state recognition channel can adopt a graph convolutional network-based architecture, representing the input video frame pose estimation node framework sequence as a spatiotemporal graph structure, where each joint node is a graph node, the connection between adjacent joints is a graph edge, and nodes between adjacent frames in the temporal dimension are also connected by edges. Through multi-layer graph convolution operations, the first isogamete abnormal state recognition channel can extract the spatiotemporal features of isogamete sequences over a longer time scale and capture progressive posture change patterns.

[0082] The second matching anomaly state recognition channel is a recognition channel built based on the recognition bandwidth of the second matching channel. This recognition channel uses the recognition bandwidth of the second matching channel as its temporal receptive field. When processing the pose estimation node framework subsequence of matching video frames, it extracts the pose estimation node framework of consecutive video frames within the time span corresponding to the recognition bandwidth of the second matching channel as input each time. The second matching anomaly state recognition channel also adopts a graph convolutional network-based architecture, but due to its smaller temporal receptive field, it can pay more attention to short-term pose detail changes and capture local anomaly features in the matching subsequence.

[0083] The first and second matching anomaly state recognition channels are connected in parallel, meaning the two recognition channels process the input matching video frame pose estimation node framework subsequence in parallel. The first matching anomaly state recognition channel extracts features from a larger time scale and outputs a feature vector at a larger time scale; the second matching anomaly state recognition channel extracts features from a smaller time scale and outputs a feature vector at a smaller time scale. The outputs of both recognition channels are simultaneously connected to a first fully connected layer. The first fully connected layer fuses the feature vectors from the two recognition channels, integrating features from different time scales into a unified feature representation. After the first fully connected layer, a classifier, such as a softmax classifier, is connected to classify the fused features and output the matching anomaly state recognition result, determining whether the input matching video frame pose estimation node framework subsequence contains anomalies. Through the above structure, a dual-channel matching anomaly state recognition is obtained.

[0084] Simultaneously, the first and second heterogeneous match channel recognition bandwidths are used as receptive fields to construct a first heterogeneous match anomaly state recognition channel and a second heterogeneous match anomaly state recognition channel. These two channels are then connected in parallel, and their outputs are simultaneously connected to a second fully connected layer, resulting in a dual-channel heterogeneous match anomaly state recognition system. Specifically, the first heterogeneous match anomaly state recognition channel is constructed based on the first heterogeneous match channel recognition bandwidth. This first heterogeneous match anomaly state recognition channel uses the first heterogeneous match channel recognition bandwidth as its temporal receptive field. When processing heterogeneous video frame pose estimation node framework subsequences, it extracts the pose estimation node frameworks of consecutive video frames within the time span corresponding to the first heterogeneous match channel recognition bandwidth as input each time. The first heterogeneous match anomaly state recognition channel also adopts a graph convolutional network-based architecture. Through its large temporal receptive field, it can analyze the contextual information over a longer time range before and after pose abrupt changes in the heterogeneous match subsequence, thereby determining the nature and severity of the pose abrupt changes.

[0085] The second heterogamous anomaly identification channel is built upon the identification bandwidth of the second heterogamous channel. This channel uses the identification bandwidth of the second heterogamous channel as its temporal receptive field. When processing heterogamous video frame pose estimation node framework subsequences, it extracts the pose estimation node frameworks of consecutive video frames within the time span corresponding to the identification bandwidth of the second heterogamous channel as input each time. Through its smaller temporal receptive field, the second heterogamous anomaly identification channel can accurately capture the instantaneous features of pose abrupt changes in heterogamous subsequences, enabling rapid response to sudden anomaly events.

[0086] The first and second heterogeneous anomaly state recognition channels are connected in parallel, meaning that the two recognition channels process the input heterogeneous video frame pose estimation node framework subsequence in parallel. The first heterogeneous anomaly state recognition channel extracts features from a larger time scale and outputs a feature vector at a larger time scale; the second heterogeneous anomaly state recognition channel extracts features from a smaller time scale and outputs a feature vector at a smaller time scale. The outputs of both recognition channels are simultaneously connected to a second fully connected layer. The second fully connected layer fuses the feature vectors from the two recognition channels, integrating features from different time scales into a unified feature representation. After the second fully connected layer, a classifier, such as a softmax classifier, is connected to classify the fused features and outputs the heterogeneous anomaly state recognition result, determining whether the input heterogeneous video frame pose estimation node framework subsequence contains anomalies and the specific type of the anomaly. Through the above structure, a dual-channel heterogeneous anomaly state recognition is obtained.

[0087] The design principle of the dual-channel architecture lies in the fact that a single time-scale recognition channel may not be able to fully capture the diverse characteristics of abnormal states. Some abnormal states manifest as rapid, instantaneous changes, such as sudden falls or spasms; these require recognition channels with small time receptive fields to capture them promptly. Other abnormal states manifest as slow, gradual changes, such as a gradual decline in activity or a gradual decrease in respiratory rate; these require recognition channels with large time receptive fields for accurate identification. By constructing a dual-channel parallel processing architecture with different time receptive fields, posture changes can be analyzed simultaneously from both rapid and slow time scales, avoiding the omission or misjudgment of abnormal events caused by a single time scale.

[0088] Subsequently, dual-channel anomaly state recognition is used to perform dual-channel anomaly state recognition on multiple sets of pose estimation node framework sub-sequences of the same video frames, obtaining multiple sets of anomaly state recognition results. Specifically, each pose estimation node framework sub-sequence of the same video frame in the multiple sets of pose estimation node framework sub-sequences of the same video frames is sequentially input into the dual-channel anomaly state recognition. For each input pose estimation node framework sub-sequence of the same video frame, the first and second anomaly state recognition channels are processed in parallel, extracting features from the large and small time scales respectively. The first fully connected layer fuses the feature outputs of the two channels, and the classifier determines whether the pose estimation node framework sub-sequence of the same video frame contains an anomaly state based on the fused features, outputting the corresponding anomaly state recognition result. The anomaly state recognition result can be a binary classification result, i.e., "normal" or "abnormal"; or it can be a multi-class result, such as specific anomaly types like "normal", "abnormal breathing", or "reduced activity". By performing the above identification process on all the same-match video frame pose estimation node framework subsequences in multiple sets of matching video frame pose estimation node framework subsequences, multiple sets of matching abnormal state identification results are obtained. Each set of matching abnormal state identification results corresponds to a set of matching video frame pose estimation node framework subsequences for a video surveillance camera, containing the abnormal state identification results of all matching subsequences under the monitoring of that camera.

[0089] Simultaneously, dual-channel anomaly state recognition is used to perform dual-channel anomaly state recognition on multiple sets of pose estimation node framework sub-sequences of heterogeneous video frames, obtaining multiple sets of heterogeneous anomaly state recognition results. Specifically, each heterogeneous video frame pose estimation node framework sub-sequence in the multiple sets of heterogeneous video frame pose estimation node framework sub-sequences is sequentially input into the dual-channel heterogeneous anomaly state recognition. For each input heterogeneous video frame pose estimation node framework sub-sequence, the first and second heterogeneous anomaly state recognition channels are processed in parallel, extracting features from large and small time scales respectively. The second fully connected layer fuses the feature outputs of the two channels. The classifier determines whether the heterogeneous video frame pose estimation node framework sub-sequence contains an anomaly state and the specific type of the anomaly state based on the fused features, and outputs the corresponding heterogeneous anomaly state recognition result. The heterogeneous anomaly state recognition result is usually a multi-class result, such as specific anomaly types like "normal movement," "sudden fall," "convulsions," and "respiratory arrest." By performing the above recognition processing on all heterogeneous video frame pose estimation node framework sub-sequences in the multiple sets of heterogeneous video frame pose estimation node framework sub-sequences, multiple sets of heterogeneous anomaly state recognition results are obtained. Each set of heterogeneous abnormal state identification results corresponds to a set of heterogeneous video frame pose estimation node subsequences of a video surveillance camera, containing the abnormal state identification results of all heterogeneous subsequences under the monitoring of that camera.

[0090] Through the aforementioned dual-channel anomaly state recognition process, targeted anomaly state recognition was performed on both the same-match video frame pose estimation node framework subsequence set and the different-match video frame pose estimation node framework subsequence set, resulting in multiple sets of same-match and different-match anomaly state recognition results. The dual-channel architecture can simultaneously capture anomaly features at different time scales, effectively avoiding omissions or misjudgments of anomaly events caused by a single time scale, thus improving the accuracy, completeness, and robustness of anomaly state recognition.

[0091] Furthermore, a channel identification bandwidth set is obtained, wherein the channel identification bandwidth set includes a first matching channel identification bandwidth, a second matching channel identification bandwidth, a first dissimilar channel identification bandwidth, and a second dissimilar channel identification bandwidth, including:

[0092] S511. Obtain the historical abnormal state identification log set, and divide the historical abnormal state identification log set according to the same-matching abnormal type and the different-matching abnormal type to obtain the historical same-matching abnormal state identification log set and the historical different-matching abnormal state identification log set.

[0093] S512. Extract the maximum interval duration from the historical matching anomaly state identification log set and the historical mismatch anomaly state identification log set respectively to obtain the first matching channel identification bandwidth and the first mismatch channel identification bandwidth.

[0094] S513. Extract the minimum interval duration from the historical matching anomaly state identification log set and the historical mismatch anomaly state identification log set respectively to obtain the second matching channel identification bandwidth and the second mismatch channel identification bandwidth.

[0095] S514. The first matching channel identification bandwidth, the first dissimilar channel identification bandwidth, the second matching channel identification bandwidth, and the second dissimilar channel identification bandwidth are summed to obtain the channel identification bandwidth set.

[0096] In a preferred embodiment, firstly, a set of historical abnormal state identification logs is obtained, and then this set is divided into a set of historical matching abnormal state identification logs and a set of historical dismatching abnormal state identification logs, based on matching and dismatching abnormal state types. The historical abnormal state identification log set is a collection of abnormal state identification data recorded during past monitoring of critically ill patients. Each abnormal state identification log records information such as the occurrence time, duration, abnormal type, and posture change characteristics of an abnormal state. This historical log data reflects the actual occurrence patterns and temporal characteristics of abnormal states in critically ill patients, providing a data basis for determining a reasonable channel identification bandwidth.

[0097] The historical abnormal state identification log set was classified according to the nature of the abnormal state. Specifically, based on the postural change characteristics corresponding to the abnormal state, the abnormal states were divided into homologous abnormalities and heterologous abnormalities. Homologous abnormalities refer to abnormal states that occur during continuous and gradual postural changes, such as a gradual decrease in the patient's respiratory rate, a gradual reduction in limb movement, or a prolonged period of unchanged body position. Heterologous abnormalities refer to abnormal states that occur with abrupt changes in posture, such as a sudden fall from the bed, sudden convulsions, or respiratory arrest.

[0098] Each log record in the historical anomaly identification log set is traversed and classified according to its anomaly type. Log records belonging to the same-match anomaly type are assigned to the historical same-match anomaly identification log set, and log records belonging to the different-match anomaly type are assigned to the historical different-match anomaly identification log set. This partitioning operation yields the historical same-match anomaly identification log set and the historical different-match anomaly identification log set, providing a classification data foundation for subsequent analysis of the time-series characteristics of different types of anomalies.

[0099] Then, the maximum interval duration is extracted from the historical matching anomaly state identification log set and the historical mismatch anomaly state identification log set, respectively, to obtain the identification bandwidth of the first matching channel and the identification bandwidth of the first mismatch channel. Specifically, the interval duration refers to the time span from the occurrence of an anomaly state to its identification by the system, reflecting the duration characteristic of the anomaly state. The maximum interval duration represents the time span required for the longest-lasting anomaly event of this type, characterizing the maximum time scale that this type of anomaly state may have.

[0100] For the historical matching anomaly state identification log set, all log records in the set are traversed, and the anomaly state interval duration recorded in each log record is extracted. By comparing the interval durations of all matching anomaly states, the maximum value is found, and this maximum interval duration is used as the identification bandwidth of the first matching channel. The identification bandwidth of the first matching channel represents the time span of the longest-lasting anomaly event among the matching anomaly states. Using this as the temporal receptive field of the first matching anomaly state identification channel ensures that the identification channel has a sufficiently large time observation window to capture all known progressive matching anomaly states.

[0101] For the historical heterogamous anomaly state identification log set, all log records in the set are traversed, and the interval duration of the anomaly state recorded in each log record is extracted. By comparing the interval durations of all heterogamous anomaly states, the maximum value is found, and this maximum interval duration is used as the identification bandwidth of the first heterogamous channel. The identification bandwidth of the first heterogamous channel represents the time span of the longest-lasting anomaly event in the heterogamous anomaly state. Using this as the temporal receptive field of the first heterogamous anomaly state identification channel ensures that the identification channel can analyze a sufficiently long period of contextual information before and after the attitude change in the heterogamous subsequence, and accurately determine the nature of the attitude change.

[0102] Through the above extraction operations, the recognition bandwidth of the first matching channel and the recognition bandwidth of the first dismatch channel are obtained, providing parameter basis for constructing recognition channels on a large time scale.

[0103] Subsequently, the minimum interval duration is extracted from the historical matching anomaly state identification log sets and the historical mismatch anomaly state identification log sets to obtain the identification bandwidth of the second matching channel and the identification bandwidth of the second mismatch channel. The minimum interval duration represents the time span required for the shortest-duration anomaly event in this type of anomaly state, characterizing the smallest time scale that this type of anomaly state may have.

[0104] For the historical homogamous anomaly identification log set, all log records in the set are traversed, and the anomaly interval duration recorded in each log record is extracted. By comparing the interval durations of all homogamous anomalies, the minimum value is found and used as the second homogamous channel identification bandwidth. The second homogamous channel identification bandwidth represents the time span of the shortest-duration anomaly event in the homogamous anomaly state. Using this as the temporal receptive field of the second homogamous anomaly identification channel ensures that the identification channel has a sufficiently fine temporal resolution to capture rapidly occurring local anomaly fluctuations in the homogamous subsequence, avoiding the omission of short-term anomaly events due to an excessively large temporal receptive field.

[0105] For the historical heteromigration anomaly state identification log set, all log records in the set are traversed, and the anomaly state interval duration recorded in each log record is extracted. By comparing the interval durations of all heteromigration anomalies, the minimum value is found and used as the identification bandwidth of the second heteromigration channel. The identification bandwidth of the second heteromigration channel represents the time span of the shortest-duration anomaly event in the heteromigration anomaly state. Using this as the temporal receptive field of the second heteromigration anomaly state identification channel ensures that the identification channel can quickly respond to instantaneous attitude changes in heteromigration subsequences and accurately capture the instantaneous characteristics of sudden anomalies.

[0106] Through the above extraction operations, the recognition bandwidth of the second matching channel and the recognition bandwidth of the second mismatched channel are obtained, providing a parameter basis for constructing recognition channels on a small time scale.

[0107] Subsequently, the identification bandwidths of the first matching channel, the first dismatch channel, the second matching channel, and the second dismatch channel are aggregated to obtain a channel identification bandwidth set. Specifically, the identification bandwidths of the first matching channel and the first dismatch channel obtained in step S512, and the identification bandwidths of the second matching channel and the second dismatch channel obtained in step S513 are aggregated and integrated to form a channel identification bandwidth set. The channel identification bandwidth set includes receptive field parameters of two time scales, set for matching abnormal states and dismatch abnormal states respectively, providing a complete parameter configuration for the subsequent construction of dual-channel identification for matching abnormal states and dual-channel identification for dismatch abnormal states.

[0108] By using the channel identification bandwidth determination method based on historical abnormal state identification log data, the temporal receptive field of the identification channel can be adaptively set according to the real temporal characteristics of the actual abnormal state. This ensures that the dual-channel architecture can cover the complete time scale range of abnormal states, capturing both long-lasting, gradual abnormal states and quickly responding to short-lasting, sudden abnormal events, thereby improving the accuracy and completeness of abnormal state identification.

[0109] Furthermore, by combining the multi-source monitoring data sequences, the multiple sets of same-matching abnormal state identification results and the multiple sets of dismatching abnormal state identification results are interactively verified to obtain the target abnormal state identification result, including:

[0110] S61. Based on the time intervals corresponding to the multiple sets of same-matching abnormal state identification results and the multiple sets of different-matching abnormal state identification results, the multi-source monitoring data sequence is extracted to obtain multiple sets of extracted same-matching multi-source monitoring data sub-sequences and multiple sets of extracted different-matching multi-source monitoring data sub-sequences.

[0111] S62. Based on the multiple sets of extracted homologous multi-source monitoring data sub-sequences, interactive verification is performed on the multiple sets of homologous abnormal state identification results to obtain a first set of abnormal state identification results.

[0112] S63. Based on the multiple sets of extracted heterogeneous multi-source monitoring data sub-sequences, interactive verification is performed on the multiple sets of heterogeneous abnormal state identification results to obtain a second set of abnormal state identification results.

[0113] S64. Summarize the first abnormal state identification result set and the second abnormal state identification result set to obtain the target abnormal state identification result.

[0114] In a preferred embodiment, firstly, based on the time intervals corresponding to multiple sets of same-match anomaly state identification results and multiple sets of different-match anomaly state identification results, the multi-source monitoring data sequences are extracted to obtain multiple sets of extracted same-match multi-source monitoring data sub-sequences and multiple sets of extracted different-match multi-source monitoring data sub-sequences. Each anomaly state identification result in each set of same-match and different-match anomaly state identification results corresponds to a specific time interval, which records the start and end times of the corresponding video frame pose estimation node framework sub-sequence. To interactively verify the anomaly state identification results obtained from video pose analysis with physiological signal data, it is necessary to extract the multi-source monitoring data that corresponds to these identification results in time.

[0115] For multiple sets of matching abnormal state identification results, each matching abnormal state identification result in the set is traversed to obtain the time interval of the corresponding matching video frame pose estimation node framework subsequence. Based on the start and end times of this time interval, respiratory rate data and heart rate data within this time interval are extracted from the multi-source monitoring data sequence. Specifically, since the multi-source monitoring data sequence and video surveillance data are synchronized on the time axis, the data segment corresponding to this time interval in the multi-source monitoring data sequence can be accurately located by timestamp matching. The extracted data segment constitutes an extracted matching multi-source monitoring data subsequence, which contains respiratory rate data sequences and heart rate data sequences that are perfectly aligned in time with the corresponding matching abnormal state identification result. By performing the above extraction operation on all identification results in multiple sets of matching abnormal state identification results, multiple sets of extracted matching multi-source monitoring data subsequences are obtained.

[0116] For multiple sets of mismatched anomaly identification results, the same extraction method is used. Each mismatched anomaly identification result in the set is traversed, and the time interval of the corresponding mismatched video frame pose estimation node framework subsequence is obtained. Based on this time interval, respiratory rate data and heart rate data within the corresponding time period are extracted from the multi-source monitoring data sequence to form the extracted mismatched multi-source monitoring data subsequence. By performing the above extraction operation on all identification results in multiple sets of mismatched anomaly identification results, multiple sets of extracted mismatched multi-source monitoring data subsequences are obtained.

[0117] The above extraction operations ensure that the abnormal state identification results obtained from video pose analysis correspond precisely in time with the physiological signal data, providing a paired multimodal data foundation for subsequent interactive verification.

[0118] Subsequently, multiple sets of abnormal state identification results were interactively verified using multiple sets of extracted, matched, multi-source monitoring data sub-sequences to obtain the first abnormal state identification result set. Interactive verification refers to using physiological signal data to verify and filter the abnormal state identification results obtained from video posture analysis. The reliability of the video posture analysis results is confirmed by judging whether the physiological signal data also shows abnormalities. Since video posture analysis may be affected by factors such as occlusion, lighting changes, and shooting angles, leading to misjudgments, while physiological signal data can directly reflect the patient's true state from a physiological perspective, interactive verification using multimodal data can effectively improve the accuracy of abnormal state identification. Specifically, the monitoring anomaly identifyer is first invoked to identify anomalies in multiple extracted, matched, multi-source monitoring data sub-sequences. The monitoring anomaly identifyer is a classification model trained based on physiological signal features, capable of determining whether the patient is in an abnormal state based on the change patterns of respiratory rate and heart rate. The monitoring anomaly identifyer can employ machine learning-based classification algorithms, such as support vector machines, random forests, or deep neural networks, to identify the presence of abnormal fluctuations by analyzing the statistical characteristics, frequency domain characteristics, and temporal change patterns of the respiratory rate and heart rate data sequences. For example, abnormally slowed respiratory rate (below the normal range), abnormally fasted or slowed heart rate, and abnormally increased or decreased variability in respiratory rate and heart rate can all be identified as abnormal physiological signals. Then, each extracted homologous multi-source monitoring data subsequence is input into the monitoring anomaly identifier to obtain the corresponding extracted homologous multi-source monitoring anomaly identification result. This homologous multi-source monitoring anomaly identification result indicates whether the physiological signal data within that time period shows abnormalities. By performing anomaly identification on all subsequences in multiple extracted homologous multi-source monitoring data subsequence sets, multiple sets of extracted homologous multi-source monitoring anomaly identification results are obtained.

[0119] Subsequently, multiple sets of extracted and matched multi-source monitoring anomaly identification results and multiple sets of matched anomaly state identification results are matched and identified. Specifically, for each matched anomaly state identification result, its corresponding extracted and matched multi-source monitoring anomaly identification result is searched, and it is determined whether both show anomalies. If a matched anomaly state identification result indicates an anomaly state in that time period, and the corresponding extracted and matched multi-source monitoring anomaly identification result also indicates anomalies in the physiological signal data in that time period, then the matching and identification is considered successful, indicating that the anomaly state identification result has been verified and supported by physiological signal data, and has high reliability. At this time, the verified matched anomaly state identification result is added to the first anomaly state identification result set. If a matched anomaly state identification result indicates an anomaly, but the corresponding extracted and matched multi-source monitoring anomaly identification result indicates that the physiological signal data is normal, then the matching and identification is considered unsuccessful, indicating that the anomaly state identification result may be a misjudgment by video pose analysis. This identification result is not added to the first anomaly state identification result set, thereby filtering out the suspected misjudgment identification result.

[0120] By performing the aforementioned interactive verification and matching recognition processing on all recognition results in multiple sets of mating abnormal state recognition results, a first set of abnormal state recognition results is obtained. This first set of abnormal state recognition results contains mating abnormal state recognition results verified and confirmed by physiological signal data. These recognition results are supported by abnormal evidence from both video pose analysis and physiological signal monitoring dimensions, demonstrating high accuracy and reliability.

[0121] Simultaneously, multiple sets of abnormal state identification results for heterogamous pairing are interactively verified based on multiple sets of extracted heterogamous multi-source monitoring data sub-sequences to obtain a second abnormal state identification result set. Specifically, the same interactive verification method as in step S62 is used to verify the abnormal state identification results for heterogamous pairing. First, the monitoring anomaly identifier is invoked to identify anomalies in multiple extracted heterogamous multi-source monitoring data sub-sequences, obtaining multiple sets of extracted heterogamous multi-source monitoring anomaly identification results. The monitoring anomaly identifier analyzes the respiratory rate and heart rate data in the extracted heterogamous multi-source monitoring data sub-sequences to identify whether there are physiological signal mutations corresponding to postural changes. For example, when a patient suddenly falls or convulses, it is usually accompanied by a sharp increase in heart rate or a sudden change in breathing pattern. These physiological signal changes can physiologically confirm the occurrence of postural changes. Subsequently, the multiple sets of extracted heterogamous multi-source monitoring anomaly identification results and the multiple sets of abnormal state identification results for heterogamous pairing are matched and identified. For each abnormal state identification result for heterogamous pairing, the corresponding extracted heterogamous multi-source monitoring anomaly identification result is searched to determine whether both show abnormalities. If the match is successful, it indicates that the identification result of the mismatch abnormal state has been verified and supported by physiological signal data, and the identification result is added to the second abnormal state identification result set. If the match fails, it indicates that the identification result may be a misjudgment and is not adopted.

[0122] By performing the aforementioned interactive verification and matching recognition processing on all the recognition results in multiple sets of mismatched abnormal state recognition results, a second set of abnormal state recognition results is obtained. This second set of abnormal state recognition results contains mismatched abnormal state recognition results verified and confirmed by physiological signal data. These recognition results are supported by abnormal evidence from both video pose analysis and physiological signal monitoring dimensions.

[0123] Next, the first and second abnormal state identification result sets are summarized to obtain the target abnormal state identification result. Specifically, the first abnormal state identification result set obtained in step S62 and the second abnormal state identification result set obtained in step S63 are summarized and integrated. The first abnormal state identification result set contains verified matching abnormal state identification results, and the second abnormal state identification result set contains verified dismatch abnormal state identification results. All abnormal state identification results in these two sets are merged to form a complete target abnormal state identification result. The target abnormal state identification result integrates information from two dimensions: video posture analysis and physiological signal monitoring. It has undergone multimodal data interaction verification, effectively filtering out possible misjudgments from single-modal data. This target abnormal state identification result includes not only progressive matching abnormal states but also sudden dismatch abnormal states, comprehensively reflecting the abnormal state of critically ill patients. Based on this target abnormal state identification result, a corresponding alarm mechanism is triggered to notify medical staff to intervene in a timely manner, thereby providing reliable technical support for real-time monitoring of critically ill patients and improving patient safety and the quality of medical services.

[0124] By combining multi-source monitoring data sequences for interactive verification, multimodal data fusion was achieved. This fully leverages the respective advantages of video surveillance and physiological signal monitoring, allowing them to complement and verify each other, and significantly improving the accuracy, reliability, and robustness of abnormal state identification.

[0125] Furthermore, based on the multiple sets of extracted homologous multi-source monitoring data sub-sequences, the multiple sets of homologous abnormal state identification results are interactively verified to obtain a first abnormal state identification result set, including:

[0126] S621. Call the monitoring anomaly identifier to perform anomaly identification on the multiple extracted homogeneous multi-source monitoring data subsequences, and obtain a set of multiple extracted homogeneous multi-source monitoring anomaly identification results;

[0127] S622. Match and identify the multiple sets of extracted same-match multi-source monitoring anomaly identification results and the multiple sets of same-match anomaly state identification results. If the matching and identification is successful, add the corresponding same-match anomaly state identification result to the first anomaly state identification result set.

[0128] In a preferred embodiment, firstly, a monitoring anomaly identifier is invoked to identify anomalies in multiple extracted multi-source monitoring data sub-sequences, obtaining multiple sets of anomaly identification results. The monitoring anomaly identifier is a classification model trained based on physiological signal features, used to determine whether a patient is in an abnormal state based on the changing patterns of respiratory rate and heart rate data. The construction process of this monitoring anomaly identifier is as follows: First, a large amount of historical physiological monitoring data from critically ill patients is collected, including respiratory rate and heart rate data sequences under normal and abnormal states. These data are manually labeled to mark samples of normal and abnormal physiological states. Normal physiological state samples include data segments where respiratory rate and heart rate are within the normal range and change steadily, while abnormal physiological state samples include abnormally slowed or accelerated respiratory rate, abnormal heart rate fluctuations, and respiratory arrest. Based on the labeled training samples, a machine learning model is constructed as the monitoring anomaly identifier, which can employ various classification algorithms, such as support vector machines, random forests, gradient boosting decision trees, or deep neural networks. For the input respiratory rate and heart rate data sequences, the anomaly detector first extracts features, including statistical features, frequency domain features, and temporal features. Statistical features include mean, standard deviation, maximum, and minimum values; frequency domain features are obtained through Fourier transform or wavelet transform and can reflect the frequency distribution characteristics of physiological signals; temporal features include trends, variability, and abrupt change points. The extracted features are input into a classification model, which outputs the physiological state classification result corresponding to the data sequence, i.e., normal or abnormal. The anomaly detector is trained using labeled training samples, and the model parameters are adjusted through optimization algorithms to enable the model to accurately distinguish between normal and abnormal physiological states. After training, the model is validated and tested to ensure that it can accurately determine physiological states even on unseen data.

[0129] In practical applications, each extracted multi-source monitoring data sub-sequence from multiple sets of identically matched sub-sequences is sequentially input into a trained monitoring anomaly identifier. For each input sub-sequence, the anomaly identifier analyzes the respiratory rate and heart rate data sequences it contains, extracts features, performs classification, and outputs the physiological state identification result corresponding to that sub-sequence. If the anomaly identifier determines that the physiological signals in the sub-sequence are abnormal, such as a respiratory rate below the normal range, abnormal heart rate fluctuations, or abnormal variability in respiratory rate and heart rate, it outputs an anomaly identification result; if it determines that the physiological signals are normal, it outputs a normal identification result.

[0130] By performing the aforementioned anomaly identification processing on all extracted multi-source monitoring data subsequences in multiple sets of extracted matching multi-source monitoring data, multiple sets of extracted matching multi-source monitoring anomaly identification results are obtained. Each extracted matching multi-source monitoring anomaly identification result corresponds to an extracted matching multi-source monitoring data subsequence, recording whether the patient's physiological state was abnormal within the time period represented by that data subsequence. This set of identification results provides a physiological signal-level judgment basis for subsequent matching and verification with video pose analysis results.

[0131] Subsequently, multiple sets of extracted and matched multi-source monitoring anomaly identification results and multiple sets of matched anomaly state identification results are matched and identified. If the matching and identification is successful, the corresponding matched anomaly state identification result is added to the first anomaly state identification result set. Matching and identification refers to comparing the matched anomaly state identification results obtained from video pose analysis with the extracted and matched multi-source monitoring anomaly identification results obtained from physiological signal monitoring, determining whether both show anomalies within the same time period, thereby verifying the reliability of the video pose analysis results. Specifically, each matched anomaly state identification result in the multiple sets of matched anomaly state identification results is traversed. For each matched anomaly state identification result, the time interval identifier information of the matched video frame pose estimation node frame subsequence corresponding to the identification result is first obtained. Based on this time interval identifier information, the extracted and matched multi-source monitoring anomaly identification result corresponding to this time interval is searched in the multiple sets of extracted and matched multi-source monitoring anomaly identification results. Since step S61 has already ensured the precise temporal correspondence between the extracted and matched multi-source monitoring data subsequences and the matched anomaly state identification results, the paired extracted and matched multi-source monitoring anomaly identification results can be accurately found.

[0132] After finding the paired anomaly identification results from the extracted multi-source monitoring, the two identification results are compared and judged. Specifically, it is checked whether the paired anomaly identification result indicates an abnormal state during that time period, and whether the corresponding extracted multi-source monitoring anomaly identification result also indicates anomalies in the physiological signal data during that time period. If both identification results show anomalies, that is, both video pose analysis and physiological signal monitoring determine that there is anomalies, then the matching identification is considered successful. Successful matching identification indicates that the anomaly identification result has been validated by the consistency of multimodal data, and evidence of anomalies has been observed from two independent dimensions: video pose and physiological signals. The reliability of the identification result has been significantly improved.

[0133] When a match is successfully identified, the corresponding anomaly identification result is added to the first anomaly identification result set. The first anomaly identification result set stores anomaly identification results that have been verified by physiological signal data. These identification results are not only determined to be anomalies based on video pose analysis, but are also supported and verified by physiological signal data, thus possessing high accuracy and reliability.

[0134] If the two identification results are inconsistent—that is, the matching anomaly identification result indicates an anomaly but the extracted matching multi-source monitoring anomaly identification result indicates a normal anomaly, or vice versa—then the matching identification is deemed a failure. Matching identification failure indicates a conflict between the video pose analysis result and the physiological signal monitoring result. This could be due to misjudgment caused by factors such as occlusion or lighting in the video pose analysis, or misjudgment caused by noise interference in the physiological signal monitoring, or perhaps some anomalies are only apparent in a single modality. In cases of matching identification failure, the matching anomaly identification result is not added to the first anomaly identification result set, thereby filtering out potential misjudgments and improving the accuracy of the final anomaly identification result.

[0135] By performing the aforementioned matching and identification process on all matching abnormal state identification results in multiple sets of matching abnormal state identification results, interactive verification of all matching abnormal state identification results was completed. The verified identification results were then aggregated into the first abnormal state identification result set. The first abnormal state identification result set contains matching abnormal state identification results that have been confirmed through multimodal data interactive verification. These identification results are supported by abnormal evidence from both video pose analysis and physiological signal monitoring dimensions, effectively reducing the impact of misjudgment from single-modal data and significantly improving the accuracy, reliability, and robustness of abnormal state identification.

[0136] Example 2, as Figure 2 As shown, based on the same inventive concept as the method for identifying abnormal states of critically ill patients based on dynamic video monitoring provided in Embodiment 1, this embodiment of the invention also provides a system for identifying abnormal states of critically ill patients based on dynamic video monitoring, including:

[0137] The video acquisition module 11 is used to deploy multiple video surveillance cameras, obtain monitoring start commands, and synchronously start the multiple video surveillance cameras based on the monitoring start commands to obtain multiple dynamic video streams.

[0138] The photoelectric monitoring module 12 is used to obtain multi-source monitoring data sequences based on the reflection spectrum signal using remote photoplethysmography based on the monitoring start command.

[0139] The attitude estimation module 13 is used to obtain the initial attitude estimation node framework of the target critically ill patient, and to fill the initial attitude estimation node framework by traversing each video frame in the multiple dynamic video streams to obtain a sequence of attitude estimation node frameworks for multiple video frames.

[0140] Sequence separation module 14 is used to perform adjacency and heterogeneous separation on the multiple video frame pose estimation node frame sequences to obtain multiple sets of homogeneous video frame pose estimation node frame subsequences and multiple sets of heterogeneous video frame pose estimation node frame subsequences.

[0141] Anomaly identification module 15 is used to perform dual-channel anomaly identification on multiple sets of pose estimation node framework subsequences of the same video frames and multiple sets of pose estimation node framework subsequences of different video frames respectively, and obtain multiple sets of anomaly identification results of the same and multiple sets of anomaly identification results of different.

[0142] The interactive verification module 16 is used to perform interactive verification on the multiple sets of same-matching abnormal state identification results and the multiple sets of dismatching abnormal state identification results by combining the multi-source monitoring data sequence, so as to obtain the target abnormal state identification result.

[0143] Furthermore, the execution steps of the sequence separation module 14 include:

[0144] Extract the first video frame pose estimation node frame sequence from the plurality of video frame pose estimation node frame sequences;

[0145] The first and second video frame attitude estimation node frames in the first video frame attitude estimation node frame sequence are subjected to adjacency and heterogeneous separation verification. If the verification fails, the first and second video frame attitude estimation node frames are added to the first homogeneous video frame attitude estimation node frame subsequence.

[0146] If the verification is successful, the first video frame pose estimation node framework will be added to the first heterogeneous video frame pose estimation node framework subsequence.

[0147] Based on the second video frame attitude estimation node framework, the third video frame attitude estimation node framework in the first video frame attitude estimation node framework sequence is separated into adjacent and heterogeneous pairs until the end of the first video frame attitude estimation node framework sequence is reached, thereby obtaining the first homogeneous video frame attitude estimation node framework subsequence set and the first heterogeneous video frame attitude estimation node framework subsequence set.

[0148] Furthermore, the execution steps of the sequence separation module 14 also include:

[0149] A mapping node similarity analysis is performed on the first video frame pose estimation node framework and the second video frame pose estimation node framework to obtain a mapping node similarity set.

[0150] The similarity set of the mapped nodes is normalized, and an adjacency-discordant separation matrix is ​​constructed based on the normalization result.

[0151] The adjacency-discordant separation verifier is invoked to perform separation verification on the adjacency-discordant separation matrix, and the separation verification result is output, wherein the separation verification result includes verification success and verification failure.

[0152] Furthermore, the execution steps of the anomaly detection module 15 include:

[0153] Obtain a set of channel identification bandwidths, wherein the set of channel identification bandwidths includes a first matching channel identification bandwidth, a second matching channel identification bandwidth, a first dissimilar channel identification bandwidth, and a second dissimilar channel identification bandwidth;

[0154] The first matching channel recognition bandwidth and the second matching channel recognition bandwidth are used as receptive fields to construct the first matching abnormal state recognition channel and the second matching abnormal state recognition channel. The first matching abnormal state recognition channel and the second matching abnormal state recognition channel are connected in parallel, and the output is simultaneously connected to the first fully connected layer to obtain the matching abnormal state recognition dual channel.

[0155] The first mismatch channel recognition bandwidth and the second mismatch channel recognition bandwidth are used as receptive fields to construct the first mismatch abnormal state recognition channel and the second mismatch abnormal state recognition channel. The first mismatch abnormal state recognition channel and the second mismatch abnormal state recognition channel are connected in parallel, and the output is simultaneously connected to the second fully connected layer to obtain the dual channel for mismatch abnormal state recognition.

[0156] The dual-channel identification of the same-matching abnormal state is used to perform dual-channel abnormal state identification on the multiple sets of pose estimation node sub-sequences of the same-matching video frames to obtain multiple sets of same-matching abnormal state identification results.

[0157] The dual-channel identification of heterogeneous abnormal states is used to perform dual-channel abnormal state identification on the multiple heterogeneous video frame pose estimation node frame sub-sequence sets respectively, so as to obtain the multiple heterogeneous abnormal state identification result sets.

[0158] Furthermore, the execution steps of the anomaly detection module 15 also include:

[0159] Obtain a set of historical abnormal state identification logs, and divide the set of historical abnormal state identification logs into the same-matching abnormal type and the different-matching abnormal type to obtain a set of historical same-matching abnormal state identification logs and a set of historical different-matching abnormal state identification logs.

[0160] Extract the maximum interval duration from the historical matching anomaly state identification log set and the historical mismatch anomaly state identification log set respectively to obtain the first matching channel identification bandwidth and the first mismatch channel identification bandwidth.

[0161] Extract the minimum interval duration from the historical matching anomaly state identification log set and the historical mismatch anomaly state identification log set respectively to obtain the second matching channel identification bandwidth and the second mismatch channel identification bandwidth.

[0162] The first matching channel identification bandwidth, the first mismatched channel identification bandwidth, the second matching channel identification bandwidth, and the second mismatched channel identification bandwidth are summed to obtain the channel identification bandwidth set.

[0163] Furthermore, the execution steps of the interactive verification module 16 include:

[0164] Based on the time intervals corresponding to the multiple sets of same-matching abnormal state identification results and the multiple sets of different-matching abnormal state identification results, the multi-source monitoring data sequence is extracted to obtain multiple sets of extracted same-matching multi-source monitoring data sub-sequences and multiple sets of extracted different-matching multi-source monitoring data sub-sequences.

[0165] Based on the multiple sets of extracted homologous multi-source monitoring data sub-sequences, the multiple sets of homologous abnormal state identification results are interactively verified to obtain the first abnormal state identification result set.

[0166] Based on the multiple sets of extracted heterogeneous multi-source monitoring data sub-sequences, the multiple sets of heterogeneous abnormal state identification results are interactively verified to obtain a second abnormal state identification result set.

[0167] The first abnormal state identification result set and the second abnormal state identification result set are summarized to obtain the target abnormal state identification result.

[0168] Furthermore, the execution steps of the interactive verification module 16 also include:

[0169] The monitoring anomaly detector is invoked to perform anomaly identification on the multiple extracted homogeneous multi-source monitoring data subsequences, and multiple sets of extracted homogeneous multi-source monitoring anomaly identification results are obtained.

[0170] The multiple sets of extracted same-match multi-source monitoring anomaly identification results and multiple sets of same-match anomaly state identification results are matched and identified. If the matching and identification are successful, the corresponding same-match anomaly state identification result is added to the first anomaly state identification result set.

[0171] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0172] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Although preferred embodiments of the invention have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0177] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for identifying abnormal states of critically ill patients based on dynamic video monitoring, characterized in that, The method includes: Multiple video surveillance cameras are deployed, a monitoring start command is obtained, and the multiple video surveillance cameras are started synchronously based on the monitoring start command to obtain multiple dynamic video streams; Based on the monitoring start command, remote photoplethysmography is used to obtain multi-source monitoring data sequences according to the reflectance spectral signals; The initial pose estimation node framework of the target critically ill patient is obtained, and the initial pose estimation node framework is filled in each video frame in the multiple dynamic video streams to obtain a sequence of multiple video frame pose estimation node frameworks. The multiple video frame pose estimation node framework sequences are subjected to adjacency-discordant separation to obtain multiple sets of identical video frame pose estimation node framework subsequences and multiple sets of discordant video frame pose estimation node framework subsequences. The adjacency-discordant separation refers to comparing and analyzing the temporally adjacent video frame pose estimation node frameworks in the video frame pose estimation node framework sequence to determine whether there are abnormal abrupt changes in the pose changes between adjacent frames. Dual-channel abnormal state identification is performed on multiple sets of pose estimation node framework subsequences of the same video frames and multiple sets of pose estimation node framework subsequences of different video frames to obtain multiple sets of abnormal state identification results of the same and multiple sets of abnormal state identification results of different. By combining the multi-source monitoring data sequence, the multiple sets of same-matching abnormal state identification results and the multiple sets of dismatching abnormal state identification results are interactively verified to obtain the target abnormal state identification result. Dual-channel anomaly state identification is performed on multiple sets of pose estimation node framework subsequences for matching video frames and multiple sets of pose estimation node framework subsequences for dismatched video frames, respectively, to obtain multiple sets of anomaly state identification results for matching and multiple sets of anomaly state identification results for dismatched video frames, including: Obtain a set of channel identification bandwidths, wherein the set of channel identification bandwidths includes a first matching channel identification bandwidth, a second matching channel identification bandwidth, a first dissimilar channel identification bandwidth, and a second dissimilar channel identification bandwidth; The first matching channel recognition bandwidth and the second matching channel recognition bandwidth are used as receptive fields to construct the first matching abnormal state recognition channel and the second matching abnormal state recognition channel. The first matching abnormal state recognition channel and the second matching abnormal state recognition channel are connected in parallel, and the output is simultaneously connected to the first fully connected layer to obtain the matching abnormal state recognition dual channel. The first mismatch channel recognition bandwidth and the second mismatch channel recognition bandwidth are used as receptive fields to construct the first mismatch abnormal state recognition channel and the second mismatch abnormal state recognition channel. The first mismatch abnormal state recognition channel and the second mismatch abnormal state recognition channel are connected in parallel, and the output is simultaneously connected to the second fully connected layer to obtain the dual channel for mismatch abnormal state recognition. The dual-channel identification of the same-matching abnormal state is used to perform dual-channel abnormal state identification on the multiple sets of pose estimation node sub-sequences of the same-matching video frames to obtain multiple sets of same-matching abnormal state identification results. The dual-channel identification of heterogeneous abnormal states is used to perform dual-channel abnormal state identification on the multiple heterogeneous video frame pose estimation node frame sub-sequence sets respectively, so as to obtain the multiple heterogeneous abnormal state identification result sets.

2. The method for identifying abnormal states of critically ill patients based on dynamic video monitoring as described in claim 1, characterized in that, The multiple video frame pose estimation node framework sequences are subjected to adjacency-discordant separation to obtain multiple sets of identically matched video frame pose estimation node framework subsequences and multiple sets of discordantly matched video frame pose estimation node framework subsequences, including: Extract the first video frame pose estimation node frame sequence from the plurality of video frame pose estimation node frame sequences; The first and second video frame attitude estimation node frames in the first video frame attitude estimation node frame sequence are subjected to adjacency and heterogeneous separation verification. If the verification fails, the first and second video frame attitude estimation node frames are added to the first homogeneous video frame attitude estimation node frame subsequence. If the verification is successful, the first video frame pose estimation node framework will be added to the first heterogeneous video frame pose estimation node framework subsequence. Based on the second video frame attitude estimation node framework, the third video frame attitude estimation node framework in the first video frame attitude estimation node framework sequence is separated into adjacent and heterogeneous pairs until the end of the first video frame attitude estimation node framework sequence is reached, thereby obtaining the first homogeneous video frame attitude estimation node framework subsequence set and the first heterogeneous video frame attitude estimation node framework subsequence set.

3. The method for identifying abnormal states of critically ill patients based on dynamic video monitoring as described in claim 2, characterized in that, The adjacency-discordance separation verification is performed on the first and second video frame pose estimation node frames in the first video frame pose estimation node frame sequence, including: A mapping node similarity analysis is performed on the first video frame pose estimation node framework and the second video frame pose estimation node framework to obtain a mapping node similarity set. The similarity set of the mapped nodes is normalized, and an adjacency-discordant separation matrix is ​​constructed based on the normalization result. The adjacency-discordant separation verifier is invoked to perform separation verification on the adjacency-discordant separation matrix, and the separation verification result is output, wherein the separation verification result includes verification success and verification failure.

4. The method for identifying abnormal states of critically ill patients based on dynamic video monitoring as described in claim 1, characterized in that, Obtain a channel identification bandwidth set, wherein the channel identification bandwidth set includes a first matching channel identification bandwidth, a second matching channel identification bandwidth, a first dissimilar channel identification bandwidth, and a second dissimilar channel identification bandwidth, including: Obtain a set of historical abnormal state identification logs, and divide the set of historical abnormal state identification logs into the same-matching abnormal type and the different-matching abnormal type to obtain a set of historical same-matching abnormal state identification logs and a set of historical different-matching abnormal state identification logs. Extract the maximum interval duration from the historical matching anomaly state identification log set and the historical mismatch anomaly state identification log set respectively to obtain the first matching channel identification bandwidth and the first mismatch channel identification bandwidth. Extract the minimum interval duration from the historical matching anomaly state identification log set and the historical mismatch anomaly state identification log set respectively to obtain the second matching channel identification bandwidth and the second mismatch channel identification bandwidth. The first matching channel identification bandwidth, the first mismatched channel identification bandwidth, the second matching channel identification bandwidth, and the second mismatched channel identification bandwidth are summed to obtain the channel identification bandwidth set.

5. The method for identifying abnormal states of critically ill patients based on dynamic video monitoring as described in claim 1, characterized in that, By combining the multi-source monitoring data sequences, the multiple sets of same-matching abnormal state identification results and the multiple sets of dismatching abnormal state identification results are interactively verified to obtain the target abnormal state identification result, including: Based on the time intervals corresponding to the multiple sets of same-matching abnormal state identification results and the multiple sets of different-matching abnormal state identification results, the multi-source monitoring data sequence is extracted to obtain multiple sets of extracted same-matching multi-source monitoring data sub-sequences and multiple sets of extracted different-matching multi-source monitoring data sub-sequences. Based on the multiple sets of extracted homologous multi-source monitoring data sub-sequences, the multiple sets of homologous abnormal state identification results are interactively verified to obtain the first abnormal state identification result set. Based on the multiple sets of extracted heterogeneous multi-source monitoring data sub-sequences, the multiple sets of heterogeneous abnormal state identification results are interactively verified to obtain a second abnormal state identification result set. The first abnormal state identification result set and the second abnormal state identification result set are summarized to obtain the target abnormal state identification result.

6. The method for identifying abnormal states of critically ill patients based on dynamic video monitoring as described in claim 5, characterized in that, Based on the multiple sets of extracted homologous multi-source monitoring data sub-sequences, the multiple sets of homologous abnormal state identification results are interactively verified to obtain a first set of abnormal state identification results, including: The monitoring anomaly detector is invoked to perform anomaly identification on the multiple extracted homogeneous multi-source monitoring data subsequences, and multiple sets of extracted homogeneous multi-source monitoring anomaly identification results are obtained. The multiple sets of extracted same-match multi-source monitoring anomaly identification results and multiple sets of same-match anomaly state identification results are matched and identified. If the matching and identification are successful, the corresponding same-match anomaly state identification result is added to the first anomaly state identification result set.

7. A system for identifying abnormal states of critically ill patients based on dynamic video surveillance, characterized in that, For implementing the method for identifying abnormal states of critically ill patients based on dynamic video monitoring as described in any one of claims 1 to 6, the system comprises: The video acquisition module is used to deploy multiple video surveillance cameras, obtain monitoring start commands, and synchronously start the multiple video surveillance cameras based on the monitoring start commands to obtain multiple dynamic video streams. The photoelectric monitoring module is used to obtain multi-source monitoring data sequences based on the reflectance spectral signals using remote photoplethysmography based on the monitoring start command. The attitude estimation module is used to obtain the initial attitude estimation node framework of the target critically ill patient, and to fill the initial attitude estimation node framework by traversing each video frame in the multiple dynamic video streams to obtain a sequence of attitude estimation node frameworks for multiple video frames. The sequence separation module is used to perform adjacency and heterogeneous separation on the multiple video frame attitude estimation node framework sequences to obtain multiple sets of homogeneous video frame attitude estimation node framework sub-sequences and multiple sets of heterogeneous video frame attitude estimation node framework sub-sequences. The anomaly identification module is used to perform dual-channel anomaly identification on multiple sets of pose estimation node framework subsequences of the same video frames and multiple sets of pose estimation node framework subsequences of different video frames, respectively, to obtain multiple sets of anomaly identification results for the same and multiple sets of anomaly identification results for different. The interactive verification module is used to perform interactive verification on the multiple sets of same-matching abnormal state identification results and the multiple sets of dismatching abnormal state identification results by combining the multi-source monitoring data sequence, so as to obtain the target abnormal state identification result.

Images