Plastic team clinical treatment cooperation abnormal behavior detection method and system based on spatial-temporal feature fusion

Abstract

The invention relates to a plastic team clinical treatment cooperation abnormal behavior detection method and system based on spatio-temporal feature fusion. The method comprises the steps of obtaining clinical multi-modal data and performing spatio-temporal synchronization alignment; performing feature extraction on the clinical multi-modal data to obtain a time-space interaction feature set and a prediction feature set; acquiring operation specification information, and inputting the operation specification information, the space-time interaction feature set and the prediction feature set into a collaborative detection model; when the abnormal behavior information is received, identifying an abnormal type and matching an abnormal response strategy; according to the method, the multi-modal data is processed through time-space synchronization alignment, and the operation specification information is fused to carry out multi-level anomaly detection and dynamic risk assessment, so that the problems that a traditional method cannot quantify a time-space interaction relationship and lacks a multi-source data fusion capability are solved; the method has the effects of improving the abnormal behavior detection accuracy, enhancing the multi-modal data fusion capability and realizing dynamic risk prediction.

Description

Technical Field

[0001] This application relates to the technical field of abnormal behavior prediction, and in particular to a method and system for detecting abnormal behaviors in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion. Background Technology

[0002] In the clinical treatment of plastic surgery, the quality of collaboration within the surgical team directly affects the patient's treatment outcome and surgical safety. With the increasing prevalence of minimally invasive surgery and complex reconstructive plastic surgery, the complexity of multi-member collaborative operations in the operating room has significantly increased, placing higher demands on the precision and timeliness of teamwork.

[0003] Currently, operating room behavior monitoring mainly relies on video surveillance systems and manual observation. This method has obvious limitations: traditional video analysis technology is difficult to quantify and assess the spatiotemporal interaction relationships among team members, and cannot capture abnormal patterns in micro-collaborative behaviors such as instrument transfer and personnel movement in real time.

[0004] Meanwhile, while existing monitoring systems based on single-sensor data can identify some overt anomalies, they lack the ability to fuse and process multi-source heterogeneous data, resulting in insufficient accuracy in predicting potential risks. More importantly, existing technical solutions fail to establish a dynamic correlation mechanism between surgical guidelines and real-time behavioral data, leading to discrepancies between anomaly detection results and actual clinical needs. These shortcomings directly affect the timeliness of abnormal behavior warnings and the effectiveness of intervention measures, increasing potential risks during surgery. Especially in fields like plastic surgery, where high precision is required, traditional methods cannot effectively identify critical abnormal behaviors such as improper instrument use or team member coordination errors. These problems can lead to serious consequences such as prolonged surgery time and increased risk of complications. Furthermore, existing systems lack the ability to synchronously process multimodal data in time and space, making it difficult to accurately reconstruct the three-dimensional spatial relationships and temporal logic in the surgical scene, further limiting the accuracy of anomaly detection.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] To address the aforementioned shortcomings, this application provides a method and system for detecting abnormal behaviors in clinical treatment collaboration within plastic surgery teams based on spatiotemporal feature fusion.

[0007] The above-mentioned objective of this application is achieved through the following technical solution:

[0008] A method and system for detecting abnormal collaborative behaviors in the clinical treatment of plastic surgery teams based on spatiotemporal feature fusion, comprising the following steps:

[0009] Acquire clinical multimodal data, determine benchmark data, and perform spatiotemporal synchronization alignment based on the benchmark data;

[0010] Feature extraction is performed on the spatiotemporally synchronized clinical multimodal data to obtain the spatiotemporal interaction feature set and the prediction feature set;

[0011] Obtain surgical standard information, and input the surgical standard information, spatiotemporal interaction feature set and prediction feature set into a pre-built collaborative detection model to detect and predict abnormal behavior.

[0012] When abnormal behavior information is received from the collaborative detection model, the abnormal type of the abnormal behavior information is identified and the corresponding abnormal response strategy is matched.

[0013] By adopting the above technical solutions, clinical multimodal data is acquired and precisely spatiotemporally synchronized, achieving a unified spatiotemporal benchmark for multi-dimensional behavioral data such as instrument transfer and personnel movement in surgical scenarios. This solves the problem that traditional video surveillance struggles to quantify and assess spatiotemporal interaction relationships. Deep feature extraction of multimodal data constructs a complete feature set including spatiotemporal interaction features and predictive features, overcoming the limitations of single-sensor system data fusion capabilities and significantly improving the accuracy of potential risk identification. By inputting surgical protocol information and real-time feature data into a collaborative detection model, a dynamic correlation mechanism between clinical standards and real-time behavioral data is established, ensuring a high degree of consistency between abnormal detection results and actual clinical needs. Finally, intelligent abnormality type identification and response strategy matching achieve closed-loop management from abnormality detection to clinical intervention. This application, through the above solutions, effectively addresses the problems of monitoring blind spots, high false alarm rates, and delayed response in traditional methods in plastic surgery, providing reliable technical support for high-precision surgical operations. It significantly reduces surgical risks caused by abnormal team collaboration, improving the timeliness of abnormal behavior warnings, enhancing the accuracy of risk prediction, and optimizing clinical decision support.

[0014] In a preferred embodiment, this application can be further configured as follows: the steps of acquiring clinical multimodal data, determining benchmark data, and performing spatiotemporal synchronization alignment based on the benchmark data include the following steps:

[0015] Acquire static and dynamic clinical data of the target patients;

[0016] A surgical area coordinate system was constructed based on static clinical data, and spatial alignment was performed.

[0017] Time synchronization is performed based on dynamic clinical data as a time benchmark.

[0018] By adopting the above technical solutions, static and dynamic clinical data of the target patient are acquired respectively, enabling independent optimization of spatial and temporal references in the surgical scenario. Specifically, a surgical area coordinate system is constructed and spatially aligned based on static clinical data to ensure spatial consistency of surgical instrument positioning and personnel movement trajectories, solving the registration problem of multi-source sensor data in three-dimensional space. Simultaneously, dynamic clinical data is used as a temporal reference for synchronous calibration, overcoming time drift between different acquisition devices. This application, through the above solutions, achieves both meeting the high-precision spatial positioning requirements of plastic surgery and adapting to the dynamic temporal characteristics of clinical treatment, enabling the entire monitoring system to more accurately reflect the actual collaborative status of the surgical team. It also improves the spatiotemporal alignment accuracy of multimodal data, enhances the spatial consistency of anomaly detection, and improves the accuracy of temporal analysis.

[0019] In a preferred embodiment, this application can be further configured such that: the clinical multimodal data includes device information data, medical staff information data, vital sign data, and environmental perception data; the step of extracting features from the spatiotemporally synchronized clinical multimodal data to obtain a spatiotemporal interaction feature set and a prediction feature set includes the following steps:

[0020] Extracting device frequency characteristics and device movement characteristics based on device information data;

[0021] Extracting surgeon's action and movement features based on medical member information data;

[0022] Extract vital sign features based on vital sign data;

[0023] Constructing micro-environment feature vectors based on environmental perception data;

[0024] Based on the extracted instrument frequency features, instrument movement features, surgeon action features, surgeon movement features, vital sign features, and microenvironment feature vectors, a spatiotemporal interaction feature set and a prediction feature set are generated.

[0025] By adopting the above technical solution, the following key clinical data are collected and processed: instrument information data, medical staff information data, vital sign data, and environmental perception data, to achieve a full-dimensional characterization of the surgical scene. Specifically, frequency and movement features extracted from instrument information data accurately reflect the usage patterns and movement trajectories of surgical instruments; surgeon action and movement features obtained from medical staff information data comprehensively record the operational behavior and spatial distribution of the surgical team; feature parameters extracted from vital sign data monitor the physiological state of medical staff in real time; and micro-environment feature vectors constructed from environmental perception data quantify the impact of operating room environmental factors on the operation. This application, through the above solution, can not only accurately describe the current surgical state but also effectively predict potential collaborative risks, providing feature input for subsequent intelligent analysis and decision-making. It has the effects of constructing a comprehensive feature system, achieving multi-dimensional data fusion, and improving feature representation capabilities.

[0026] In a preferred embodiment, this application can be further configured as follows: the collaborative detection model includes a feature fusion layer, an anomaly detection layer, a risk assessment layer, and an information output layer. The step of acquiring surgical specification information and inputting the surgical specification information, spatiotemporal interaction feature set, and prediction feature set into the pre-constructed collaborative detection model for abnormal behavior detection and abnormal behavior prediction includes the following steps:

[0027] The feature fusion layer generates a joint feature vector based on the spatiotemporal interaction feature set and the predicted feature set;

[0028] The anomaly detection layer generates multi-level anomaly detection information based on surgical protocol information and joint feature vectors;

[0029] The risk assessment layer generates anomaly prediction information based on surgical protocol information and predictive feature sets.

[0030] The information output layer identifies multi-level anomaly detection information and anomaly prediction information, and generates abnormal behavior information.

[0031] By adopting the above technical solution, a collaborative detection model is constructed, comprising a feature fusion layer, an anomaly detection layer, a risk assessment layer, and an information output layer, to achieve intelligent monitoring and early warning of the collaborative behavior of the surgical team. The model first deeply integrates the spatiotemporal interaction feature set and the predicted feature set through the feature fusion layer, generating a joint feature vector with higher representational capabilities. Subsequently, the anomaly detection layer, combined with surgical protocol information, performs multi-level analysis of the joint feature vector to accurately identify existing abnormal behaviors. Simultaneously, the risk assessment layer, based on the predicted feature set, proactively assesses potential surgical risks. Finally, the information output layer integrates various detection and assessment results to generate structured abnormal behavior information. This application, through the above solution, not only improves the accuracy of anomaly identification but also enhances the system's early warning capabilities, providing the surgical team with more timely and effective decision support. It achieves the dual functions of real-time anomaly detection and risk prediction, improves the comprehensiveness and reliability of system decision-making, and optimizes the operability of information output.

[0032] In a preferred embodiment, this application can be further configured as follows: the step of generating multi-level anomaly detection information based on surgical specification information and joint feature vectors in the anomaly detection layer includes the following steps:

[0033] Transform surgical protocol information into a spatial constraint matrix and a time series template;

[0034] A pre-set attention mechanism is used to calculate the deviation scores between the joint feature vector and the spatial constraint matrix and the time series template, respectively.

[0035] Abnormal levels are classified based on deviation scores, and abnormal information is identified, including abnormal information of medical devices and abnormal information of medical personnel.

[0036] Kinematic verification and baseline comparison were performed on the identified abnormal information of medical devices and abnormal information of medical personnel, respectively.

[0037] Multi-level anomaly detection information is generated based on kinematic verification results and baseline comparison results.

[0038] By adopting the above technical solution, surgical standard information is transformed into a quantifiable spatial constraint matrix and time series template, enabling accurate comparison between clinical standards and real-time monitoring data. A pre-set attention mechanism is used to calculate deviation scores, dynamically assessing the degree of difference between surgical procedures and standard requirements, and intelligently classifying abnormalities based on the degree of deviation. Kinematic verification and baseline comparison are performed on instrument abnormalities and medical personnel abnormalities respectively, ensuring the credibility and accuracy of abnormality detection results. This application, through the above solution, can not only accurately identify various abnormal behaviors but also provide differentiated early warning strategies based on the severity of the abnormalities, offering surgical teams more accurate risk assessment and decision support. It achieves graded early warning of abnormalities, improves the credibility of detection results, and enhances the system's fault tolerance.

[0039] In a preferred embodiment, this application can be further configured as follows: the step of the risk assessment layer generating abnormal prediction information based on surgical protocol information and a prediction feature set includes the following steps:

[0040] Construct a temporal probability model for the division of associated surgical stages, and map the predicted feature set to the time axis of each surgical stage;

[0041] Calculate the matching degree between the predicted feature set and the corresponding surgical stage, and generate a stage risk score;

[0042] Predicting the evolution trend of risk in successive surgical phases based on hidden Markov models;

[0043] Preliminary abnormal information is generated based on stage risk scores and risk evolution trends, and risk prediction weights are adjusted by combining clinical multimodal data to obtain abnormal prediction information.

[0044] By adopting the above technical solution, a temporal probability model for the division of associated surgical stages is constructed to achieve intelligent segmented monitoring and risk assessment of the surgical process: First, the predicted feature set is accurately mapped to the time axis of each surgical stage, establishing a dynamic correlation between feature data and surgical stages; by calculating the matching degree between the predicted feature set and the standard requirements of the surgical stage, a quantitative stage risk score is generated; then, a hidden Markov model is used to analyze the risk evolution trend of consecutive surgical stages and predict the development trajectory of potential abnormalities; finally, the prediction results are dynamically corrected by combining real-time clinical multimodal data to generate highly reliable abnormality prediction information; through the above solution, this application can not only identify potential risks in advance, but also dynamically optimize the prediction model according to the surgical process, providing the surgical team with more accurate early warning and intervention suggestions, achieving the effects of forward-looking prediction of surgical risks, improving the accuracy of early warning information, and optimizing the timeliness of clinical decision-making.

[0045] The second objective of this invention is achieved through the following technical solution:

[0046] A system for detecting abnormal collaborative behaviors in the clinical treatment of plastic surgery teams based on spatiotemporal feature fusion, comprising:

[0047] The data processing module is used to acquire clinical multimodal data, determine benchmark data, and perform spatiotemporal synchronization alignment based on the benchmark data.

[0048] The feature extraction module is used to extract features from the spatiotemporally synchronized clinical multimodal data to obtain a spatiotemporal interaction feature set and a prediction feature set.

[0049] The input module is used to acquire surgical standard information and input the surgical standard information, spatiotemporal interaction feature set and prediction feature set into the pre-built collaborative detection model to detect and predict abnormal behavior.

[0050] The strategy matching module is used to identify the anomaly type of the abnormal behavior information when it receives abnormal behavior information output by the collaborative detection model, and to match the corresponding anomaly response strategy.

[0051] By adopting the above technical solution, the data processing module is used to acquire clinical multimodal data, determine benchmark data, and perform spatiotemporal synchronization alignment based on the benchmark data; the feature extraction module is used to extract features from the spatiotemporally synchronized clinical multimodal data to obtain a spatiotemporal interaction feature set and a prediction feature set; the input module is used to acquire surgical specification information and input the surgical specification information, spatiotemporal interaction feature set, and prediction feature set into a pre-built collaborative detection model for abnormal behavior detection and prediction; the strategy matching module is used to identify the abnormal type of abnormal behavior information when receiving abnormal behavior information output by the collaborative detection model and match the corresponding abnormal response strategy.

[0052] In a preferred embodiment, this application can be further configured such that the data processing module includes:

[0053] The data acquisition submodule is used to acquire static and dynamic clinical data of the target patient.

[0054] The spatial alignment submodule is used to construct a surgical area coordinate system based on static clinical data and perform spatial alignment.

[0055] The time synchronization submodule is used to perform time synchronization based on dynamic clinical data as a time reference.

[0056] By adopting the above technical solution, the data acquisition submodule is used to acquire the static and dynamic clinical data of the target patient; the spatial alignment submodule is used to construct the surgical area coordinate system based on the static clinical data and perform spatial alignment; and the time synchronization submodule is used to perform time synchronization based on the dynamic clinical data as the time reference.

[0057] In summary, this application provides a method and system for detecting abnormal behaviors in clinical treatment collaboration within plastic surgery teams based on spatiotemporal feature fusion. By processing multimodal data through spatiotemporal synchronization and alignment, and fusing surgical protocol information, it performs multi-level abnormality detection and dynamic risk assessment. This solves the problems of traditional methods being unable to quantify spatiotemporal interaction relationships and lacking multi-source data fusion capabilities. It has the effects of improving the accuracy of abnormal behavior detection, enhancing multimodal data fusion capabilities, and achieving dynamic risk prediction. Attached Figure Description

[0058] Figure 1 This is a flowchart of an embodiment of a method for detecting abnormal behaviors in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion, as described in this application.

[0059] Figure 2 This is a flowchart of step S10 in an embodiment of a method for detecting abnormal behavior in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion in this application.

[0060] Figure 3 This is a flowchart of step S20 in an embodiment of a method for detecting abnormal behavior in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion in this application.

[0061] Figure 4 This is a flowchart of step S30 in an embodiment of a method for detecting abnormal behavior in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion in this application.

[0062] Figure 5 This is a flowchart of step S32 in an embodiment of a method for detecting abnormal behavior in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion in this application.

[0063] Figure 6 This is a flowchart of step S33 in an embodiment of the abnormal behavior detection method for clinical treatment collaboration in plastic surgery teams based on spatiotemporal feature fusion in this application. Detailed Implementation

[0064] The following is in conjunction with the appendix Figure 1-6 This application will be described in further detail.

[0065] In one embodiment, such as Figure 1 As shown, this application discloses a method for detecting abnormal behaviors in clinical treatment collaboration within a plastic surgery team based on spatiotemporal feature fusion, specifically including the following steps:

[0066] S10: Acquire clinical multimodal data, determine benchmark data, and perform spatiotemporal synchronization alignment based on the benchmark data;

[0067] In this embodiment, clinical multimodal data refers to multi-source heterogeneous data including instrument movement trajectories, medical personnel actions, patient vital signs, and environmental parameters. It can be acquired using radio frequency identification tags, inertial sensors, and bioelectrical signal acquisition devices to comprehensively reflect the dynamic interactive information during the surgical process. Spatiotemporal synchronization alignment refers to establishing a unified spatiotemporal coordinate system to eliminate the time delay and spatial offset of multi-source data. This can be achieved through dynamic time warping algorithms and three-dimensional coordinate transformation matrices to ensure the consistency of different modal data in the spatiotemporal dimensions.

[0068] Specifically, multi-source sensor systems deployed in the operating room can be used to acquire real-time clinical multimodal data, including the movement trajectory of surgical instruments, the operation actions of medical staff, vital signs parameters, and environmental monitoring data. Based on a preset spatiotemporal benchmark, these heterogeneous data are precisely synchronized in time and unified in spatial coordinates to ensure that all data have a consistent spatiotemporal reference system.

[0069] S20: Extract features from the spatiotemporally synchronized clinical multimodal data to obtain a spatiotemporal interaction feature set and a prediction feature set;

[0070] In this embodiment, the spatiotemporal interaction feature set refers to the feature set that characterizes the spatiotemporal association between the movement of the device and the operator. The coupling features of the device's movement speed and the operator's hand trajectory can be extracted by a convolutional neural network to identify abnormal collaboration patterns. The prediction feature set refers to the feature set that reflects the evolution trend of potential risks. The fluctuation features of vital sign sequences can be extracted by a long short-term memory network to predict possible abnormal events.

[0071] Specifically, convolutional neural networks and other methods can be used to extract the spatiotemporal correlation feature set between the movement trajectory of the related instruments and the surgeon's hand movements. Long short-term memory networks and other methods can be combined to analyze the trend of vital signs changes and generate a predictive feature set.

[0072] S30: Obtain surgical standard information, and input the surgical standard information, spatiotemporal interaction feature set and prediction feature set into the pre-built collaborative detection model to detect and predict abnormal behavior;

[0073] In this embodiment, the surgical protocol information refers to standardized surgical procedures and requirements, including: instrument usage guidelines, personnel division of labor requirements, operation time thresholds, and sterile area restrictions; the collaborative detection model is a pre-trained multi-layer neural network model for anomaly detection, which is mainly used for real-time abnormal behavior detection and prediction based on surgical protocol information, spatiotemporal interaction feature sets, and prediction feature sets.

[0074] Specifically, the pre-stored surgical guidelines are retrieved and input together with the real-time extracted feature set into a pre-trained collaborative detection model. This collaborative detection model uses a multi-layered neural network architecture to comprehensively analyze the current surgical state, detecting both real-time abnormal behaviors and predicting potential collaborative risks.

[0075] S40: When receiving abnormal behavior information output by the collaborative detection model, identify the abnormal type of the abnormal behavior information and match the corresponding abnormal response strategy.

[0076] In this embodiment, the abnormal behavior information is the comprehensive information output by the collaborative detection model, including abnormal behavior detection information and abnormal behavior prediction information; the abnormal type is the type of abnormality corresponding to the abnormal behavior information; and the abnormal response strategy is the corresponding measures preset based on the abnormal type.

[0077] Specifically, when the model outputs abnormal behavior information, it identifies the specific type of abnormality, such as improper use of equipment, violation of personnel positioning, or abnormal vital signs, and automatically matches the most appropriate treatment plan based on the preset response strategy library, including graded early warning prompts, operational restriction suggestions, or emergency intervention measures, thereby realizing intelligent monitoring and risk management of the clinical treatment collaboration process of the plastic surgery team.

[0078] In one embodiment, such as Figure 2 As shown, step S10 includes the following steps:

[0079] S11: Obtain static and dynamic clinical data of the target patients;

[0080] S12: Construct a surgical area coordinate system based on static clinical data and perform spatial alignment;

[0081] S13: Perform time synchronization based on dynamic clinical data as a time benchmark.

[0082] In this embodiment, static clinical data refers to fixed anatomical structure information within the surgical area, such as bone markers or a 3D model reconstructed from preoperative images. Specifically, it can be implemented using 3D mesh models derived from CT scan data or MRI image segmentation, used to establish a spatial reference system related to the surgical procedure. Dynamic clinical data refers to continuously changing physiological or operational signals during the surgical process, such as electrocardiogram waveforms or respiratory rate curves. Specifically, it can be implemented using time-series data output from vital sign monitoring devices, serving as a benchmark for the unified time dimension of multimodal data. The surgical area coordinate system refers to a local 3D coordinate system established based on the patient's anatomical structure. Specifically, it can be defined by selecting at least three non-coplanar anatomical landmarks to define the origin and axis, used to map spatial data from different sources to a unified reference system. Spatial alignment refers to the process of converting spatial data acquired by multimodal acquisition devices to the surgical area coordinate system. Specifically, it can be implemented using rigid body transformation matrices or affine transformation algorithms to eliminate spatial positioning deviations between devices. Time synchronization refers to calibrating the time axis of other data sources based on the timestamps of dynamic clinical data. Specifically, it can be implemented using dynamic time warping algorithms or sliding window matching mechanisms to align data streams with different sampling frequencies.

[0083] Specifically, for example, in a plastic surgery scenario, a static three-dimensional model of the patient's surgical area is first acquired using medical imaging equipment. Anatomical landmarks such as the anterior superior iliac spine and pubic symphysis are selected to construct a coordinate system. The spatial coordinate data of the surgical field camera is then mapped into this coordinate system using a coordinate transformation matrix. Simultaneously, the end-tidal carbon dioxide concentration curve output by the anesthesia machine is used as a time reference. A dynamic time warping algorithm is used to phase-align the time series of the surgeon's motion capture device and the instrument tracking system, ensuring that the timestamps of all data are unified onto the same reference axis. For instance, when there is a time offset between the captured data of the surgeon's forceps holding action and the electrosurgical activation signal, a sliding window is used to match the waveform feature points of both, eliminating the timing misalignment caused by the device sampling delay.

[0084] In one embodiment, clinical multimodal data includes device information data, medical staff information data, vital sign data, and environmental perception data, such as... Figure 3 As shown, step S20 includes the following steps:

[0085] S21: Extract instrument frequency features and instrument movement features based on instrument information data;

[0086] S22: Extracting surgeon's action features and surgeon's movement features based on medical member information data;

[0087] S23: Extract vital sign features based on vital sign data;

[0088] S24: Constructing micro-environment feature vectors based on environmental perception data;

[0089] S25: Generate a spatiotemporal interaction feature set and a prediction feature set based on the extracted instrument frequency features, instrument movement features, surgeon action features, surgeon movement features, vital sign features, and microenvironment feature vectors.

[0090] In this embodiment, the instrument frequency characteristic refers to the number of times a surgical instrument is used per unit time and the interval period, which can be achieved using an accelerometer combined with a sliding window statistical method to identify abnormal instrument operation rhythm; the instrument movement characteristic refers to the trajectory and speed changes of the instrument in three-dimensional space, which can be achieved using an inertial measurement unit combined with a Kalman filter algorithm to detect instrument delivery path deviation; the surgeon's action characteristic refers to the changes in the joint angles of the medical staff's hands and their operating posture, which can be achieved using an optical motion capture system combined with skeletal joint point analysis to identify non-standard operating movements; the surgeon's movement characteristic refers to the displacement distance and dwell time of the medical staff within the surgical area, which can be achieved using a UWB positioning system combined with a path integral algorithm to detect positioning deviation problems; the vital signs characteristic refers to the dynamic changes of the patient's heart rate, blood pressure, and other physiological indicators, which can be achieved using biosensors combined with time series analysis to capture sudden physiological abnormalities; the microenvironment feature vector refers to the quantitative representation of the temperature, humidity, and airflow status of the surgical area, which can be achieved using a multi-parameter environmental sensor combined with principal component analysis to assess the impact of environmental factors on the operation.

[0091] Specifically, during the surgery, instrument information data is collected in real time by sensors attached to the surgical instruments. For example, the number of times a high-frequency electrosurgical unit is used can be recorded by a contact sensor, and the movement trajectory is tracked by a built-in IMU module. Medical staff information data is acquired through wearable motion capture devices. For example, the surgeon's finger bending angle is measured by a flexible strain sensor, and the movement path is calculated by a positioning base station. Vital sign data is continuously acquired through a monitor connected to the patient. For example, blood pressure fluctuations are monitored by a cuff sensor. Environmental perception data is collected by sensor nodes distributed around the operating table. For example, an infrared thermal imager is used to monitor the temperature distribution of the surgical field. After spatiotemporal alignment of the above multi-source data, instrument frequency characteristics and movement characteristics are jointly extracted through time-frequency analysis. Surgeon action characteristics and movement characteristics are separated through a motion decomposition algorithm. Vital sign characteristics are extracted for trend information through moving average filtering. Microenvironment feature vectors are processed to eliminate redundant data through dimensionality reduction. The final spatiotemporal interaction feature set includes an instrument-surgeon spatial relationship matrix and action synchronization indicators, while the prediction feature set includes the rate of change of physiological indicators and environmental interference factors.

[0092] In one embodiment, the collaborative detection model includes a feature fusion layer, an anomaly detection layer, a risk assessment layer, and an information output layer, such as... Figure 4 As shown, step S30 includes the following steps:

[0093] S31: The feature fusion layer generates a joint feature vector based on the spatiotemporal interaction feature set and the predicted feature set;

[0094] S32: The anomaly detection layer generates multi-level anomaly detection information based on surgical protocol information and joint feature vectors;

[0095] S33: The risk assessment layer generates anomaly prediction information based on surgical protocol information and prediction feature sets;

[0096] S34: The information output layer identifies multi-level anomaly detection information and anomaly prediction information, and generates abnormal behavior information.

[0097] In this embodiment, the feature fusion layer refers to the module that associates and maps spatiotemporal interaction features with predicted features. Specifically, it can be implemented using graph neural networks or cross-attention mechanisms to eliminate semantic gaps between features of different modalities. The anomaly detection layer refers to the module that determines anomalies based on normative constraints. Specifically, it can be implemented using dynamic threshold segmentation or pattern matching algorithms to establish a mapping relationship between real-time behavior and standard operating procedures. The risk assessment layer refers to the module that predicts the evolution trend of potential anomalies. Specifically, it can be implemented using time-series prediction models or probabilistic graphical models to quantify the cumulative risk effect at different surgical stages. The information output layer refers to the module that integrates detection results and prediction results. Specifically, it can be implemented using decision tree classifiers or rule engines to generate interpretable conclusions regarding anomaly behavior.

[0098] Specifically, during the surgical procedure, the feature fusion layer first integrates spatiotemporal interaction features such as instrument movement trajectories and changes in the positions of medical staff with predictive features such as the fluctuation trends of vital signs, forming a joint feature vector that includes spatial correlation and temporal evolution. The anomaly detection layer decomposes the surgical protocol into a spatial constraint matrix and a time series template, and calculates the degree of deviation between the joint feature vector and the protocol requirements. For example, when the surgeon's movement path deviates from the standard stationary area by more than a threshold, an instrument operation anomaly alarm is triggered. The risk assessment layer predicts the potential risk superposition effect in subsequent operations based on the current surgical stage division results. For example, if an abnormality in the use of the high-frequency electrosurgical unit is found during the tissue separation stage, the probability of massive bleeding during the subsequent hemostasis stage is predicted. Finally, the information output layer performs correlation analysis between the real-time detected abnormal events and the predicted potential risks, generating a complete report that includes the abnormality type, risk level, and treatment recommendations.

[0099] In one embodiment, such as Figure 5 As shown, step S32 includes the following steps:

[0100] S321: Transform surgical protocol information into a spatial constraint matrix and a time series template;

[0101] S322: The deviation scores between the joint feature vector and the spatial constraint matrix and the time series template are calculated using a pre-set attention mechanism.

[0102] S323: Classify the abnormality level based on the deviation score and identify abnormal information, including abnormal information of medical devices and abnormal information of medical personnel;

[0103] S324: Perform kinematic verification and baseline comparison on the identified abnormal information of medical devices and abnormal information of medical personnel respectively;

[0104] S235: Generate multi-level anomaly detection information based on kinematic verification results and baseline comparison results.

[0105] In this embodiment, the spatial constraint matrix refers to the mathematical expression of the spatial operation requirements in the surgical specifications as multi-dimensional constraints. Specifically, it can be implemented using an instrument position threshold matrix in a three-dimensional coordinate system, for example, dividing the instrument movement range in the surgical area into permitted areas, warning areas, and prohibited areas. The time series template refers to the standard operation time window divided according to the surgical stage. Specifically, it can be generated using a dynamic time warping algorithm, for example, mapping the standard time consumption of operations such as suturing and hemostasis to segmented constraints on the time axis. The deviation score refers to the quantitative index of the difference between real-time operation features and the specification template. Specifically, it can be calculated by cosine similarity or Euclidean distance to determine the degree of matching between the joint feature vector and the template. Kinematic verification refers to the rationality analysis of the instrument movement trajectory. Specifically, it can be used to determine whether the instrument operation conforms to the ergonomic principles by using thresholds for acceleration and angular velocity. Baseline comparison refers to comparing the statistical characteristics of the current operation data with the historical normal surgical dataset. Specifically, it can be used by the sliding window mean-variance analysis method to identify abnormal fluctuations.

[0106] Specifically, surgical protocol information is transformed into calculable spatial and temporal constraints in the form of matrices and templates. The deviation of the joint feature vector from these constraints is dynamically weighted using an attention mechanism. For example, the spatial deviation weight can be set to 0.6, and the temporal deviation weight can be set to 0.4. When the instrument movement path exceeds the warning area of ​​the spatial constraint matrix, kinematic verification is automatically triggered. If the instrument acceleration exceeds a preset threshold, it is determined to be an instrument abnormality. For the positional deviation of medical personnel, by comparing the positional distribution heatmap in the historical baseline data, a personnel abnormality warning is generated when the deviation exceeds two standard deviations. The abnormality level can be preset into three levels: mild, moderate, and severe based on the deviation score. For example, a deviation of less than 30% is mild, 30%-60% is moderate, and more than 60% is severe.

[0107] In one embodiment, such as Figure 6 As shown, step S33 includes the following steps:

[0108] S331: Construct a temporal probability model for the division of associated surgical stages and map the predicted feature set to the time axis of each surgical stage;

[0109] S332: Calculate the matching degree between the predicted feature set and the corresponding surgical stage, and generate a stage risk score;

[0110] S333: Predicting the evolution trend of risk in successive surgical phases based on hidden Markov models;

[0111] S334: Generate preliminary abnormal information based on stage risk scores and risk evolution trends, and adjust risk prediction weights by combining clinical multimodal data to obtain abnormal prediction information.

[0112] In this embodiment, the temporal probability model for the division of surgical stages refers to a mathematical model that decomposes the surgical procedure into multiple standard stages and establishes probabilistic relationships. Specifically, it can be implemented using Bayesian networks or decision tree algorithms to quantify the state transition probabilities between different surgical stages. The stage risk score refers to a numerical risk assessment index calculated through feature matching degree. Specifically, it can be implemented using cosine similarity or Euclidean distance algorithms to reflect the degree of deviation between the current operation and the standard surgical stage. The hidden Markov model refers to a statistical learning model used to model the temporal state transition rules. Specifically, it can be trained using a forward-backward algorithm to capture the dynamic evolution of potential risks during the surgical process. The risk prediction weight refers to the adjustment coefficient that adjusts the influence of different data sources on the final prediction result. Specifically, it can be dynamically optimized using gradient descent or genetic algorithms to improve the accuracy of multimodal data fusion.

[0113] Specifically, during the surgical procedure, the surgical process is first divided into several time stages with clearly defined operational standards according to standard operating procedures. A temporal probability model is used to map real-time collected instrument movement trajectories, vital sign fluctuations, and other predictive feature sets onto the time axis of the corresponding stages, forming a spatiotemporally correlated feature distribution map. A similarity calculation method is used to compare the feature matching degree between real-time data and the standard stages, generating a quantitative score reflecting the risk level of the current stage. Based on this, a Hidden Markov Model is used to analyze the risk score change trends across multiple consecutive stages, predicting possible future abnormal behavior patterns. Finally, the generation logic of abnormal prediction information is optimized by dynamically adjusting the weight ratio of environmental perception data and personnel behavior data; for example, the weight coefficient of environmental data is increased when the microenvironment temperature is abnormal.

[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0115] In one embodiment, a system for detecting abnormal collaborative behaviors in clinical treatment of plastic surgery teams based on spatiotemporal feature fusion is provided. This system corresponds one-to-one with the method for detecting abnormal collaborative behaviors in clinical treatment of plastic surgery teams based on spatiotemporal feature fusion described in the previous embodiment. The system includes:

[0116] The data processing module is used to acquire clinical multimodal data, determine benchmark data, and perform spatiotemporal synchronization alignment based on the benchmark data.

[0117] The feature extraction module is used to extract features from the spatiotemporally synchronized clinical multimodal data to obtain a spatiotemporal interaction feature set and a prediction feature set.

[0118] The input module is used to acquire surgical standard information and input the surgical standard information, spatiotemporal interaction feature set and prediction feature set into the pre-built collaborative detection model to detect and predict abnormal behavior.

[0119] The strategy matching module is used to identify the anomaly type of the abnormal behavior information when it receives abnormal behavior information output by the collaborative detection model, and to match the corresponding anomaly response strategy.

[0120] Optionally, the data processing module includes:

[0121] The data acquisition submodule is used to acquire static and dynamic clinical data of the target patient.

[0122] The spatial alignment submodule is used to construct a surgical area coordinate system based on static clinical data and perform spatial alignment.

[0123] The time synchronization submodule is used to perform time synchronization based on dynamic clinical data as a time reference.

[0124] Specific limitations regarding the system for detecting abnormal behaviors in clinical treatment collaboration within a plastic surgery team based on spatiotemporal feature fusion can be found in the above description of the method for detecting abnormal behaviors in clinical treatment collaboration within a plastic surgery team based on spatiotemporal feature fusion, and will not be repeated here. Each module in the aforementioned system for detecting abnormal behaviors in clinical treatment collaboration within a plastic surgery team based on spatiotemporal feature fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0125] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting abnormal collaborative behaviors in clinical treatment within a plastic surgery team based on spatiotemporal feature fusion, characterized in that, Including the following steps: Acquire clinical multimodal data, determine benchmark data, and perform spatiotemporal synchronization alignment based on the benchmark data; Feature extraction is performed on the spatiotemporally synchronized clinical multimodal data to obtain the spatiotemporal interaction feature set and the prediction feature set; Obtain surgical standard information, and input the surgical standard information, spatiotemporal interaction feature set and prediction feature set into a pre-built collaborative detection model to detect and predict abnormal behavior. When abnormal behavior information is received from the collaborative detection model, the abnormal type of the abnormal behavior information is identified and the corresponding abnormal response strategy is matched.

2. The method for detecting abnormal behaviors in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion according to claim 1, characterized in that: The steps of acquiring clinical multimodal data, determining benchmark data, and performing spatiotemporal synchronization alignment based on the benchmark data include the following steps: Acquire static and dynamic clinical data of the target patients; A surgical area coordinate system was constructed based on static clinical data, and spatial alignment was performed. Time synchronization is performed based on dynamic clinical data as a time benchmark.

3. The method for detecting abnormal behaviors in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion according to claim 1, characterized in that: The clinical multimodal data includes device information data, medical staff information data, vital sign data, and environmental perception data. The step of extracting features from the spatiotemporally synchronized clinical multimodal data to obtain a spatiotemporal interaction feature set and a prediction feature set includes the following steps: Extracting device frequency characteristics and device movement characteristics based on device information data; Extracting surgeon's action and movement features based on medical member information data; Extract vital sign features based on vital sign data; Constructing micro-environment feature vectors based on environmental perception data; Based on the extracted instrument frequency features, instrument movement features, surgeon action features, surgeon movement features, vital sign features, and microenvironment feature vectors, a spatiotemporal interaction feature set and a prediction feature set are generated.

4. The method for detecting abnormal behaviors in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion according to claim 1, characterized in that: The collaborative detection model includes a feature fusion layer, an anomaly detection layer, a risk assessment layer, and an information output layer. The step of acquiring surgical specification information and inputting the surgical specification information, spatiotemporal interaction feature set, and prediction feature set into the pre-constructed collaborative detection model for abnormal behavior detection and prediction includes the following steps: The feature fusion layer generates a joint feature vector based on the spatiotemporal interaction feature set and the predicted feature set; The anomaly detection layer generates multi-level anomaly detection information based on surgical protocol information and joint feature vectors; The risk assessment layer generates anomaly prediction information based on surgical protocol information and predictive feature sets. The information output layer identifies multi-level anomaly detection information and anomaly prediction information, and generates abnormal behavior information.

5. The method for detecting abnormal behaviors in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion according to claim 4, characterized in that: The steps for generating multi-level anomaly detection information based on surgical protocol information and joint feature vectors in the anomaly detection layer include: Transform surgical protocol information into a spatial constraint matrix and a time series template; A pre-set attention mechanism is used to calculate the deviation scores between the joint feature vector and the spatial constraint matrix and the time series template, respectively. Abnormal levels are classified based on deviation scores, and abnormal information is identified, including abnormal information of medical devices and abnormal information of medical personnel. Kinematic verification and baseline comparison were performed on the identified abnormal information of medical devices and abnormal information of medical personnel, respectively. Multi-level anomaly detection information is generated based on kinematic verification results and baseline comparison results.

6. The method for detecting abnormal behaviors in clinical treatment collaboration of plastic surgery teams based on spatiotemporal feature fusion according to claim 4, characterized in that: The step of generating anomaly prediction information based on surgical protocol information and a predictive feature set in the risk assessment layer includes the following steps: Construct a temporal probability model for the division of associated surgical stages, and map the predicted feature set to the time axis of each surgical stage; Calculate the matching degree between the predicted feature set and the corresponding surgical stage, and generate a stage risk score; Predicting the evolution trend of risk in successive surgical phases based on hidden Markov models; Preliminary abnormal information is generated based on stage risk scores and risk evolution trends, and risk prediction weights are adjusted by combining clinical multimodal data to obtain abnormal prediction information.

7. A system for detecting abnormal behaviors in clinical treatment collaboration within a plastic surgery team based on spatiotemporal feature fusion, characterized in that: include: The data processing module is used to acquire clinical multimodal data, determine benchmark data, and perform spatiotemporal synchronization alignment based on the benchmark data. The feature extraction module is used to extract features from the spatiotemporally synchronized clinical multimodal data to obtain a spatiotemporal interaction feature set and a prediction feature set. The input module is used to acquire surgical specification information and input the surgical specification information, spatiotemporal interaction feature set and prediction feature set into the pre-built collaborative detection model to detect and predict abnormal behavior. The strategy matching module is used to identify the anomaly type of the abnormal behavior information when it receives abnormal behavior information output by the collaborative detection model, and to match the corresponding anomaly response strategy.

8. The abnormal behavior detection system for clinical treatment collaboration in plastic surgery teams based on spatiotemporal feature fusion according to claim 7, characterized in that: The data processing module includes: The data acquisition submodule is used to acquire static and dynamic clinical data of the target patient. The spatial alignment submodule is used to construct a surgical area coordinate system based on static clinical data and perform spatial alignment. The time synchronization submodule is used to perform time synchronization based on dynamic clinical data as a time reference.

Images