Intraoperative nursing behavior recognition system based on computer vision
By deploying multi-angle cameras and deep learning models in the operating room, combined with dynamic region extraction and augmented reality technologies, a behavior prediction model was constructed, which solved the problems of real-time monitoring and data security of intraoperative nursing behavior recognition, and improved the efficiency and safety of the operating room.
Patent Information
- Application Number
- CN202410974016.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing intraoperative nursing management systems have limited capabilities in identifying and monitoring intraoperative nursing behaviors in real time, making them unable to effectively handle emergencies. Furthermore, they suffer from inadequate data security and data integration, and lack in-depth analysis and optimization suggestions regarding nursing staff behavior patterns.
The surgical procedure video data was collected using multi-angle, high-resolution cameras. Behavior recognition and classification were performed by combining deep learning models and long short-term memory networks. The region of interest was adjusted by dynamic region extraction algorithms, and augmented reality technology was used to assist in annotation. A behavior prediction model was constructed to identify and predict nursing behaviors, and a multi-task learning framework was used to handle various nursing tasks.
It enables real-time identification of nursing behaviors and potential risks, reduces reliance on manual recording, optimizes information flow, improves operating room efficiency and safety, enhances data sharing capabilities, and provides immediate decision support.
Smart Images

Figure CN118968613A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an intraoperative nursing behavior recognition system, in particular to an intraoperative nursing behavior recognition system based on computer vision. Background Art
[0002] When analyzing the perioperative surgical care management system described in Chinese patent CN2022117405245, it can be found that the system integrates multiple mobile nursing terminals and a system server to process and manage patient identification information, instrument package barcode information, etc., thereby improving the efficiency of nursing work and surgical safety. However, although the system has shown significant advantages in some aspects, it still has several deficiencies and potential drawbacks in practical applications, especially when it comes to intraoperative nursing behavior identification and real-time monitoring. First, the core function of the system focuses on the collection and verification of patient information and surgical instrument information, which ensures the correctness and completeness of preoperative preparations. The system relies on barcode scanning to obtain information and compares this information with a pre-stored database to verify the accuracy of patient identity and instrument packages. This approach effectively reduces the possibility of preoperative errors, such as incorrect patient identification or incorrect use of surgical instruments. However, the system has limited ability to identify and monitor dynamic and complex nursing behaviors that occur during surgery.
[0003] Secondly, the patent description does not mention the system's real-time response and processing capabilities for intraoperative events. During surgery, a high degree of real-time monitoring and intelligent processing capabilities are required in terms of monitoring the amount of bleeding, evaluating the progress of the surgery, and rapid response in emergency situations. The system mainly displays pre-set nursing prompts through mobile terminals and records the input of nursing staff, which is not enough to handle intraoperative situations that require immediate decision-making and action. In addition, the system's reliance on barcode technology has also become a bottleneck for its efficiency. Although barcode scanning performs well in ensuring the accuracy of information, relying on barcode scanning in an environment that requires rapid response such as an operating room will cause operational delays. For example, in an emergency, nursing staff need to quickly obtain instruments or materials. At this time, any delay caused by scanning problems will affect the surgical effect and even patient safety. Another potential problem with the system is the security of data integration and communication. Since the system involves sensitive medical information and patient personal information, the security of data transmission and storage is crucial. However, the patent document lacks a specific description of how to ensure data security and prevent data leakage or unauthorized access. In the current network security environment, without sufficient security protection measures, the data security of the system is difficult to ensure. Finally, the system also fails to include in-depth analysis and optimization suggestions for nursing staff's behavior patterns. During intraoperative care, nursing staff's behavior patterns and operating habits directly affect the safety and efficiency of the operation. An efficient intraoperative nursing behavior recognition system should be able to identify best nursing practices, discover risky behaviors, and provide improvement suggestions by analyzing historical data. Summary of the invention
[0004] The purpose of the present invention is to provide an intraoperative nursing behavior recognition system based on computer vision, so as to solve some of the drawbacks and shortcomings pointed out in the background technology.
[0005] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: first, deploy multi-angle high-resolution cameras in the operating room to collect video data including the behaviors of surgeons and nursing staff, and have professional medical staff label and classify the data;
[0006] Secondly, deep learning models, including convolutional neural networks (CNN) or recurrent neural networks (RNN), are used in combination with multi-task learning techniques to identify and classify nursing behaviors, and long short-term memory networks (LSTM) are used to analyze time series data in the video to identify the temporal patterns of behaviors.
[0007] We further adopt a dynamic region extraction algorithm to adjust the focus area according to the dynamic changes in the scene, combining local behavior characteristics with global scene information;
[0008] Finally, through behavioral intention analysis and abnormal behavior detection, a behavior prediction model is established to predict the next nursing behavior and timely detect abnormal nursing behaviors that are inconsistent with the routine surgical process.
[0009] Furthermore, a multi-angle high-resolution camera is deployed in the operating room, and a focus adjustment function is used:
[0010]
[0011] Where x, y, z represent the relative position coordinates of the camera and the target, t is the time, and a, σ, ω are adjustment parameters to control the key surgical actions to always be captured at the best focus and viewing angle;
[0012] Then a deep learning model is used to annotate the video data. The annotation optimization model uses:
[0013]
[0014] Where p(t) represents the predicted behavior probability distribution, q(t) represents the actual behavior label, and λ is the learning rate;
[0015] The annotation process is assisted by augmented reality technology, controlling the real-time correction equations:
[0016]
[0017] Where u represents the virtual information level in augmented reality, v represents the interaction data between actual and virtual information, N is the number of items, and α n ,β n is a coefficient used to control the interactive accuracy of the optimization annotation process.
[0018] Furthermore, the deep learning model construction includes:
[0019] A customized convolutional neural network (CNN) model is deployed to identify specific visual features in the operating room, including medical tools, surgical clothing, and hand movements. The recurrent neural network (RNN) with a long short-term memory (LSTM) network is combined to analyze the time series of behaviors during surgery to capture the development process and continuity of behaviors, including tracking the delivery, use, and recovery of surgical tools.
[0020] Furthermore, by constructing a multi-task learning framework, shared hidden layers are used to learn common features between multiple tasks, while retaining specific network branches for each specific nursing behavior task to control the recognition and classification of different nursing tasks commonly used in surgery, including disinfection, instrument delivery, and monitoring of patient vital signs;
[0021] Finally, the LSTM network is used to control the recognition of current behavior and the prediction of future behavior trends to enhance the early warning function.
[0022] Furthermore, the identifying specific visual features in the operating room includes the following steps:
[0023] First, the feature extraction formula is used:
[0024]
[0025] To optimize the recognition ability of each pixel in the image, where x, y, z represent the pixel coordinates in the image, α i and β i is the adjustment coefficient, N represents the depth of multi-level feature extraction;
[0026] Then, the long short-term memory network (LSTM) enhanced recurrent neural network (RNN) is used to perform time series analysis on the identified features to track the behavioral dynamics during the operation. The time series analysis is performed through the function:
[0027]
[0028] , where t represents the time step, h(t) is the input feature at time t, and γ j is the weight coefficient, λ(t) is the modulation function, and T is the time window considered;
[0029] Finally, the MTL strategy is learned through multi-task learning to simultaneously handle the recognition and classification of multiple nursing behaviors. Each behavior is learned through the learning function:
[0030]
[0031] Optimize, where k is the number of tasks, l j is a task-specific feature, θ j are learning parameters, δ and ∈ regulate the complexity and regularization of learning.
[0032] Furthermore, the process of constructing a multi-task learning framework includes:
[0033] S1. First, deploy multiple shared hidden layers and use the advanced feature extraction formula:
[0034]
[0035] Processing input data, where X represents the input vector obtained from surgical videos and vital signs monitoring equipment, ω i and β i is the adjustment parameter, M represents the number of input features;
[0036] S2. Then set up independent network branches for each nursing behavior task, and each branch uses a task-specific processing formula:
[0037]
[0038] Perform recognition and classification for specific tasks, where Y represents the features passed from the shared layer, θ kj is a task-specific learning parameter, N k is the number of features of the kth task, α k is the normalization constant;
[0039] S3. Finally, use the comprehensive multi-task loss function:
[0040]
[0041] To train the network, where λ k and γ k are the weight and adjustment parameters of the kth task respectively.
[0042] Furthermore, the deep learning architecture for identifying the current behavior using the LSTM network control is:
[0043] First use the composite function:
[0044]
[0045] To process the continuous time series data in surgery, where X t represents the input data at time t, W hi and W xi is the learning weight, h t-1 is the hidden state at the previous moment, σ is the activation function used to capture long-term dependencies and short-term changes in surgical behavior;
[0046] Then use the real-time behavior recognition formula:
[0047]
[0048] Where h(s) is the LSTM output at time s, λ is the decay coefficient, and W c and b c are the weights and biases of the classification layer, which are used to identify the current surgical behavior in real time;
[0049] Finally, the function of predicting future trends is:
[0050]
[0051] Where Δt is the prediction window length, W p and b p are the weights and biases of the prediction layer, which are used to predict future events during surgery and enhance the early warning function.
[0052] Furthermore, the step of implementing the dynamic region extraction algorithm comprises:
[0053] S1. First, through adaptive focus technology, use the real-time analysis function:
[0054]
[0055] Where x, y represent the spatial coordinates on the image plane, t represents the time, and α, β, γ, ω are adjustment parameters used to dynamically adjust the camera focus to track the key activities in the surgical area;
[0056] S2, then use multi-scale feature fusion technology, through the multi-level integral equation:
[0057]
[0058] Where z represents the feature input at different levels, c n and d n is a layer-specific coefficient, N represents the number of layers, and controls the accuracy of extracting key events from multi-scale perspectives;
[0059] S3, finally use the nonlinear processing function that integrates local and global information:
[0060]
[0061] Where u represents the local behavior feature set and v represents the global scene feature set, which are used to control aspects of the surgical environment to ensure a comprehensive understanding of the surgical process.
[0062] Furthermore, the process of establishing the behavior prediction model includes:
[0063] First, we use the behavior intention analysis model to build a model through a deep learning network, especially the long short-term memory network LSTM, using the function:
[0064] f(s t )=σ(W f ·[h t-1 ,x t ]+b f )
[0065] Calculate the forget gate, where s t is the state at time t, h t-1 is the output of the previous time step, x t is the current input, W f ,b f is the learning parameter, σ is the sigmoid activation function to analyze and predict the next nursing behavior;
[0066] Then, the abnormal behavior detection module is introduced, and the real-time anomaly detection function improved by the isolation forest algorithm is used to map the high-dimensional space:
[0067]
[0068] Where x represents the feature vector extracted from real-time data, θ i ,μ i ,σ i They are the feature weight, mean, and standard deviation, respectively, which are used to instantly identify and warn of abnormal nursing behaviors that are inconsistent with routine surgical procedures.
[0069] Beneficial effects of the present invention:
[0070] 1. The system continuously monitors visual data during surgery to identify nursing behaviors and potential surgical risks in real time, including incorrect use of surgical tools and non-standard surgical operations. This real-time monitoring capability enables the surgical team to respond quickly to any abnormal situation, thereby preventing surgical errors and ensuring patient safety.
[0071] 2. Through visual recognition technology, the system reduces the reliance on manual recording of surgical procedures and records detailed surgical data, including nursing behaviors, tool use, patient reactions, etc. These data not only improve the accuracy of records, but can also be used for subsequent data analysis and medical quality improvement.
[0072] 3. The system improves the overall work efficiency of the operating room by optimizing information flow and reducing redundant operations. Nursing staff can focus more on the surgical operation itself rather than auxiliary activities such as recording and information verification.
[0073] 4. The system's data sharing capabilities can strengthen cooperation among different professions, including information flow and decision-making coordination between surgical, anesthesiological, and nursing teams, thereby optimizing the entire surgical process and improving surgical safety and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 This is a flow chart of the intraoperative nursing behavior recognition system based on computer vision of the present invention.
[0075] Figure 2 This is a flow chart of the deep learning model construction method of the present invention.
[0076] Figure 3 This is a module diagram of the dynamic region extraction algorithm used in the present invention.
[0077] Figure 4 A flow chart of the process of establishing a behavior prediction model for the present invention. DETAILED DESCRIPTION
[0078] The specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings.
[0079] Combined with Figure 1 The computer vision-based intraoperative nursing behavior recognition system first captures various behaviors during surgery by deploying multi-angle, high-resolution cameras at different locations in the operating room. These cameras can record the details of the actions of surgeons and nurses from different perspectives to ensure that no key activities are missed. Next, the video data is transmitted to the analysis system, where trained professional medical staff will annotate and classify it, marking each frame in the video with different nursing activities, including hand-delivery of tools, disinfection, and operation of equipment, to create a data set for subsequent processing;
[0080] Then, a deep learning model is used for processing, including using convolutional neural network (CNN) to extract spatial features in the image, recurrent neural network (RNN) and long short-term memory (LSTM) to process and analyze time series data. This combination enables the model to not only identify static image features, but also capture dynamic behavior patterns and temporal dependencies of sequences.
[0081] In addition, the system uses a dynamic region extraction algorithm to adjust the focus area in real time. When high-risk tools are used during surgery, the system automatically zooms in on the area to ensure detailed monitoring. It optimizes the focus point by analyzing subtle changes in local behavior and dynamic changes in the overall surgical scene.
[0082] Finally, by combining behavioral intent analysis and abnormal behavior detection, it is not only possible to predict future behaviors, including predicting the tools or equipment needed next based on past and current data, but also to promptly identify and warn of any abnormal behaviors that are inconsistent with the established surgical process, including incorrect tool use or unexpected medical operations, thereby providing critical safety assurance and process optimization recommendations in real time.
[0083] Embodiment 1:
[0084] The embodiment implements a computer vision-based intraoperative nursing behavior recognition system to ensure that all key nursing behaviors can be captured at the optimal focus and viewing angle. Four high-resolution cameras are deployed in the operating room, located above, on both sides, and at the end of the operating table to fully cover the surgical area. Each camera is equipped with a dynamic focus adjustment function to adapt to the changing scene requirements during the operation.
[0085] For the focus adjustment function, the following mathematical model is used to control the focus of the camera:
[0086]
[0087] x, y, z represent the relative position coordinates of the camera and the target (surgical area or specific surgical tools), and t is time. The adjustment parameters a, σ, and ω are used to adjust the focus intensity, spatial dispersion, and response frequency of time changes, respectively. Set a = 1 to maintain the baseline intensity of focus adjustment, σ = 50 cm to provide sufficient spatial coverage, and ω = 2πrad / s to match the dynamic changes of the surgical area caused by heart rate.
[0088] During surgery, the surgical area is assumed to move slightly due to the patient's breathing and the operation of the medical staff, and the camera adjusts the focus in real time according to the above formula. For example, if the surgical tool moves from the original coordinates (0,0,0) to (10cm,0,0), the output of the camera focus adjustment function at time t=0 is:
[0089]
[0090] This value is close to 1, indicating that the surgical tool remains in sharp focus of the camera even after being moved.
[0091] In addition, if there is a large movement during the operation, and the surgical tool suddenly moves from one side to the other, the system will quickly adjust the focus depth and angle according to the changes in (x, y, z, t) to ensure that the surgical scene is always clearly visible. This dynamic focus adjustment technology not only improves the visual quality of the surgical process, but also enhances the medical staff's ability to monitor the progress of the operation, ensuring that any emergency can be responded to in a timely manner.
[0092] Next, the intraoperative nursing behavior recognition system of the embodiment enters the detailed annotation stage of the video data, and uses a deep learning model to further analyze and annotate the video data to ensure that each nursing behavior is accurately identified and recorded. The automatic annotation process of video data is improved by using an annotation optimization model, which is defined as:
[0093]
[0094] Where p(t) represents the model's predicted probability distribution for a specific nursing behavior, including handing over tools or performing disinfection; q(t) represents the actual behavior annotation provided by professional medical staff. The learning rate λ is a tuning parameter, which is set in the range of 0.1 to 5.0 in practical applications to adjust the model's sensitivity to prediction errors. When λ is set to 1.0, when p(t) = 0.8 and q(t) = 1, the model's output is calculated as:
[0095]
[0096] This result helps the system evaluate the consistency between the predicted behavior probability and the actual behavior and optimize the model's annotation accuracy.
[0097] In order to further improve the accuracy and efficiency of annotation, the system uses augmented reality (AR) technology to assist medical staff in video annotation. By controlling the real-time correction equation:
[0098]
[0099] Where u and v represent the virtual information level and the interaction strength between real and virtual information in the augmented reality environment, respectively. The parameter N represents the number of terms in the equation, which is set to 1 to 5, and the coefficient α n and β n Control the weight of each item and the decay rate of the interaction strength respectively. Set α1 = 0.5, β1 = 0.3, at u = 0.8 and v = 0.4, the output of the correction equation is:
[0100]
[0101] Such technical implementation ensures high accuracy and real-time nature of annotation, improving the quality of surgical monitoring and the responsiveness of the surgical team.
[0102] Embodiment 2:
[0103] Combined with Figure 2 The embodiment uses a customized convolutional neural network (CNN) and a long short-term memory (LSTM) recurrent neural network (RNN) to identify and analyze key visual features and behavior patterns during surgery. The designed feature extraction formula is as follows:
[0104]
[0105] It is used to optimize the recognition of individual pixels in image data collected from the operating room, specifically identifying medical tools, surgical clothing, and hand movements. Here, x, y, z represent the pixel coordinates in the image, α i and β i is the adjustment coefficient. Adjusting these parameters can help the model better adapt to the visual complexity of different surgical scenes. Set N = 3 to extract features layer by layer from simple to complex. Adjust the coefficient α i Varies from 0.1 to 1.0, β i Varies from 0.5 to 2π to match different visual frequencies and depths.
[0106] During surgery, as surgical tools move from one side to the other in the field of view, from x = 10, y = 10, z = 0 to x = 50, y = 50, z = 10 in the image, the formula can help the system dynamically adjust the focus on these tools. When the tool is at x = 50, y = 50, z = 10, using the first set of coefficients α1 = 0.1, β1 = π, the calculation is:
[0107]
[0108] The system is able to precisely track and optimize the visual processing of surgical tools using these parameters, ensuring that critical actions and objects remain in optimal visual focus at all times in a dynamically changing surgical environment.
[0109] The embodiment further analyzes the identified features through a recurrent neural network (RNN) enhanced by a long short-term memory network (LSTM) and performs time series analysis. The time series analysis model used is as follows:
[0110]
[0111] t represents the time step, h(t) is the input feature at time t, including the feature matrix extracted from CNN, and γ j is the weight coefficient used to adjust the influence of input features at different time steps; λ(y) is the modulation function used to adjust the focus of the analysis, which varies with the different stages of the surgery; T is the time window used to define the duration of the analysis, which is set to 10 minutes (600 seconds) to cover the duration of the critical stage of the surgery.
[0112] Set the weight coefficient γ j The range of is 0.1 to 1.0, indicating a decreasing influence from the current time point to each time step in the past. The modulation function λ(t) is designed to be synchronized with the surgical process. At critical moments of the surgery, including the surgical incision or suturing stage, λ(t) increases to emphasize the characteristics of these stages.
[0113] The behavior dynamics of surgical tool use are analyzed during the surgical period of T = 600 seconds. λ(t) = 1.0 is set as a constant value, indicating uniform attention in the entire analysis window, and the weight coefficients γ0 = 1.0 and γ1 = 0.5 are modulated on the input features of the current and previous time steps. h(t) and h(t-1) are the tool use intensities of the current and previous time steps (set to 1 and 0.8, respectively), and the output of the model can be calculated as:
[0114]
[0115] It means the system can accurately track and analyze the development process of key surgical behaviors, including the delivery, use and retrieval of surgical tools.
[0116] The final example further uses a multi-task learning MTL strategy to simultaneously process and optimize the recognition and classification of multiple nursing behaviors, setting multiple nursing behavior recognition tasks, including tool delivery, disinfection procedure execution, and monitoring patient vital signs. Each behavior corresponds to a learning task and is optimized through a specific learning function:
[0117]
[0118] In this formula, k represents the total number of tasks, l j is the specific feature of the jth task, θ j is the learning parameter of the task, δ and ∈ are parameters for adjusting learning complexity and regularization. Suppose there are 3 tasks, set parameters δ = 0.5, ∈ = 0.1, and set the learning parameter θ of each task j is 0.2, 0.3, 0.4, and the eigenvalue l of each task j If they are 10, 20, and 30 respectively, the learning function is calculated as follows:
[0119]
[0120] By adjusting the specific parameters and features of each task, the multi-task learning model is able to accurately handle and optimize the recognition and classification of multiple nursing behaviors by sharing information while maintaining the independence between each task.
[0121] Embodiment 3:
[0122] The embodiment constructs a multi-task learning (MTL) framework, which enables the system to learn common features between multiple nursing tasks through shared hidden layers, while also accurately controlling the recognition and classification of different nursing activities such as disinfection, instrument delivery, and monitoring of patient vital signs by retaining specific network branches for each nursing task.
[0123] First, the system deploys multiple shared hidden layers to process input data obtained from surgical videos and vital signs monitoring devices. The feature extraction formula used is:
[0124]
[0125] X represents the input vector, X i is the i-th feature in the vector, ω i and β i is a tuning parameter used to adjust the model's response strength and transformation to different features, and M is the total number of input features.
[0126] There are 10 different features (M=10) extracted from video and vital signs monitoring, and each feature corresponds to ω i and β i are set in the range of [0.1, 0.5] and [1, 5] respectively. Such settings allow the system to balance the linear and nonlinear processing of features to adapt to different data characteristics. For a specific set of parameters ω i =0.3,β i =3, input X i =0.6, the feature extraction formula calculation result is:
[0127] f(0.6)=log(1+exp(0.3·tanh(3×0.6)))
[0128] =log(1+exp(0.3·tanh(1.8)))
[0129] ≈log(1+exp(0.3·0.9487))
[0130] ≈log(1+exp(0.2846))
[0131] ≈log(1+1.329)≈log(2.329)≈0.845
[0132] This calculation result shows that the input features can effectively enhance the expressiveness of the model after conversion, and better prepare for subsequent task-specific network branches.
[0133] The embodiment further sets up independent network branches for specific nursing behavior tasks. These branches use task-specific processing formulas to ensure that each nursing behavior, including disinfection, instrument delivery, monitoring of patient vital signs, etc., can be accurately identified and classified.
[0134] Each nursing behavior task is configured with an independent network branch, and the branch uses the formula:
[0135]
[0136] to process and classify behaviors. Y represents the features passed from the shared layer, θ kj is the learning parameter of the kth task, N k is the number of features of the task, α k is the normalization constant.
[0137] Set three specific nursing behavior tasks: disinfection (task 1), instrument delivery (task 2), and monitoring patient vital signs (task 3). Set each task to have 5 characteristics (N k =5), parameter θ kj ranges from 0.05 to 0.5, α k is set to 10. These parameters ensure that the network branches can react specifically to their own behavior.
[0138] Execute Task 1 (disinfection), the feature Y transferred from the shared layer = [1, 0.8, 0.6, 0.4, 0.2], the corresponding θ kj =[0.1,0.15,0.2,0.25,0.3], then the task-specific processing formula is calculated as:
[0139]
[0140] The results show that the system is able to effectively compute and identify disinfection behaviors based on input features, providing output values to indicate the presence of the behavior.
[0141] Next, the embodiment uses a comprehensive multi-task loss function to optimize and train the network to ensure that the system can comprehensively handle multiple nursing tasks while adjusting the importance and responsiveness of each task. The comprehensive multi-task loss function is used:
[0142]
[0143] K is the total number of tasks, λ k is the weight of the kth task, indicating the importance of different tasks; γ k is a tuning parameter used to adjust the loss sensitivity of task k; g k (Y(t)) is the network output for the kth task, and Y(t) is the input feature at time t.
[0144] There are three tasks: disinfection (task 1), instrument delivery (task 2), and monitoring patient vital signs (task 3). Set task weight λ k They are 0.3, 0.4, and 0.3 respectively, indicating that the importance of instrument delivery in surgery is slightly higher than the other two. k is set to 2,2,2 to keep the loss response consistent for all tasks.
[0145] Set the network output g of the three tasks k (Y(t)) are 0.8, 0.9, and 0.7 respectively, and the loss function is calculated as follows:
[0146]
[0147] This loss value shows that the system's comprehensive performance on the three tasks is gradually being optimized, indicating the system's ability to effectively identify and classify different nursing behaviors.
[0148] Embodiment 4:
[0149] The embodiment integrates the long short-term memory network LSTM to optimize the recognition of behaviors occurring during surgery and predict future behavior trends. This function is implemented through a deep learning architecture, using the composite function:
[0150]
[0151] X t represents the input data at time t, including image features extracted from surgical videos, vital signs monitoring data, etc.; W hi and W xiis the learning weight, used to adjust the hidden state h at the previous moment t-1 and the influence of the current input data; σ is the activation function, using the sigmoid function. These functions help the model capture long-term dependencies and short-term changes in surgical behavior.
[0152] Setting n = 5 means that five input features are represented simultaneously. Weight W hi and W xi The value range of is set to [-0.5, 0.5] to allow the model to flexibly adjust the response strength to historical data and new input data. i The value range is set to [-0.1, 0.1].
[0153] At a specific time t, input data X t is [0.2, 0.3, 0.4, 0.5, 0.1], and the hidden state h of the previous moment t-1 is [0.1,0.1,0.2,0.3,0.4]. If you choose W hi =[0.1,0.2,0.1,-0.1,0.2],W xi =[0.2,-0.2,0.3,0.1,-0.3], and b i =[0.05,0.05,-0.05,0.05,0.05], then the function is calculated as follows:
[0154] f(X t )=σ(0.1·0.1+0.2·0.2+0.1·0.4-0.1·0.3+0.2·0.4+0.2·0.2-0.2·0.3+0.3·0.4
[0155] +0.1 0.5-0.3 0.1+0.05+0.05-0.05+0.05+0.05)
[0156] =σ(0.01+0.04+0.04-0.03+0.08+0.04-0.06+0.12+0.05-0.03+0.15)
[0157] =σ(0.41)
[0158] Since σ is the sigmoid function:
[0159]
[0160] The results show that the LSTM network can effectively combine historical and current data and output a comprehensive value to predict the current surgical behavior status.
[0161] The embodiment further introduces a real-time behavior recognition function, which is implemented through a real-time behavior recognition formula, using the output of the long short-term memory network LSTM and combining the time decay factor to provide instant behavior analysis:
[0162]
[0163] In this expression, h(s) is the output of the LSTM network at time s, which represents the behavioral features extracted from the video or surveillance data; λ is the decay coefficient, which is used to adjust the influence of the behavioral features at earlier times on the current analysis; W c and b c are the weights and biases of the classification layer, which are used for the final behavior classification.
[0164] We set λ = 0.1 to ensure that the influence of past behavior characteristics on the current analysis gradually decreases, which is suitable for rapidly changing surgical scenarios. c The value is 1.5, bias b c The value was taken as 0.2. These values were based on previous experiments and adjustments to optimize the model's response to behavioral changes during surgery.
[0165] Set the current behavior state to t = 60 seconds. Set the LSTM output h(s) at the past few time points (for example, every second) to be as follows: h(59) = 0.8, h(58) = 0.7, h(57) = 0.6. When calculating g(t), first calculate the integral part:
[0166]
[0167] Then substitute the real-time behavior recognition formula:
[0168] g(60) = tanh(1.5·1.4258+0.2)
[0169] =tanh(2.1387+0.2)
[0170] =tanh(2.3387)≈0.9821
[0171] This result shows that the system can accurately identify the behavioral state at the current moment, provide the surgical team with immediate and accurate behavioral analysis support, and enhance the early warning function.
[0172] The embodiment further strengthens the early warning mechanism by predicting the future trend function, and the prediction function used is:
[0173]
[0174] h(s) is the LSTM output at time s, representing the behavioral features extracted from the current surgical data; Δt is the length of the prediction window, which determines how far into the future the trend will be represented; W p and b p They are the weight and bias of the prediction layer, respectively, which adjust the sensitivity and bias of the prediction function.
[0175] Set the prediction window Δt = 30 seconds and the weight W p is 0.8, bias b p = -0.1, t = 100 seconds, predict the behavior trend in the next 30 seconds. Set the LSTM output h(s) from time 100 seconds to 130 seconds to be uniformly distributed with an average value of 0.7. Then the integral part of the prediction function is calculated as:
[0176]
[0177] Then substitute into the prediction function:
[0178]
[0179] The result is close to 1, indicating that within the next 30 seconds, the system predicts that an important behavioral event will occur, thus activating the early warning mechanism.
[0180] Embodiment 5:
[0181] Combined with Figure 3 , the embodiment implements a dynamic region extraction algorithm, enhances the system's tracking capability of the key surgical area through adaptive focus technology, and uses real-time analysis functions:
[0182]
[0183] x and y are spatial coordinates on the image plane, indicating the relative position between the camera and the target; t is time; α, β, γ, ω are adjustment parameters. The setting of these parameters directly affects the sensitivity and reaction speed of focus adjustment, allowing the camera to automatically adjust to continuously track dynamic changes during surgery.
[0184] Parameter settings:
[0185] α = 0.5, controls the sensitivity of focus adjustment;
[0186] β = 2, ensuring nonlinear acceleration of focus adjustment with increasing distance;
[0187] γ = 1, so that time changes can moderately affect focus adjustment;
[0188] ω = 2π, corresponding to a periodic adjustment of one second to match the heartbeat or other common medical monitoring rhythms.
[0189] During the operation, the surgical tool moves from position (5,5) to (10,10) at t = 10 seconds. Calculate the value of the focus adjustment function under this movement:
[0190]
[0191]
[0192] This extremely small value indicates that after the rapid movement, the system is able to adjust the parameters to regain focus and continue tracking the surgical tool.
[0193] The embodiment further adopts multi-scale feature fusion technology, by processing feature inputs at different levels and combining multi-level integral equations, so that the system can accurately extract and fuse surgical information from a multi-scale perspective, using multi-level integral equations:
[0194]
[0195] z represents different levels of feature input, including image features extracted from surgical videos, vital signs data or other sensor data; c n and d n are layer-specific coefficients that control the weights of each layer and the decay rate of features; N is the number of layers, which determines the depth and complexity of feature extraction.
[0196] Set three levels of feature extraction (N=3), and the coefficient c of each layer n is [0.3, 0.5, 0.2], the attenuation coefficient d n is [0.1, 0.05, 0.01], and the feature z is 0.6. Calculate the integral equation for each layer and accumulate the results:
[0197] For n=0, c0=0.3, d0=0.1:
[0198]
[0199] For n=1, c1=0.5, d1=0.05:
[0200]
[0201] For n=2, c2=0.2, d2=0.01:
[0202]
[0203] Adding these values together gives the result of calculating g(z):
[0204] g(0.6)=0.28935+0.29463+0.07174=0.65572
[0205] This result shows that the system is able to synthesize input data of different depths based on multi-level feature fusion technology to effectively capture and identify key events in surgical scenarios.
[0206] The final embodiment combines local and global information to identify and predict surgical behavior. The nonlinear processing function used is:
[0207]
[0208] In this formula, u(s) and v(s) represent the local behavior feature set and the global scene feature set at time s, respectively. These feature sets come from surgical video analysis, vital sign monitoring, or other sensing devices, providing data at different levels during the surgical process.
[0209] This processing function does not involve direct adjustment parameters, but the structure of the function ensures a trade-off between local and global features. The local feature u(s) is set to represent the frequency of scalpel use, and the global feature v(s) represents the activity level of the entire operating room.
[0210] Assume that within a period of time, the sample data of local features u(s) and global features v(s) are as follows:
[0211] The integral value of u(s) from time 0 to 1 is 0.8;
[0212] The integral value of v(s) from time 0 to 1 is 1.2;
[0213] The value of is 0.7.
[0214] Compute the value of a nonlinear processing function:
[0215]
[0216]
[0217]
[0218]
[0219] By fusing local and global information through nonlinear processing, the system can effectively evaluate the relationship between local behavior and global scenes.
[0220] Embodiment 6:
[0221] View attached Figure 3 The key to establishing a behavior prediction model in the embodiment is to build a behavior intention analysis model, use a deep learning network, use an LSTM network for modeling, configure the network's forget gate, and the function used by the model is:
[0222] f(s t )=σ(W f ·[h t-1 ,x t ]+b f )
[0223] s t is the state at time t, h t-1 is the output of the previous time step, x t is the current input. The weight of the forget gate is W f and bias b f is a learning parameter, which is optimized by training data, and σ is a sigmoid activation function, which is used to ensure that the network output is between 0 and 1, indicating the probability of forgetting.
[0224] In order to ensure the effectiveness of the model, W f and b f The specific value of needs to be adjusted through actual training data. Set the initial weight range to [-0.5, 0.5] and the bias to [-0.1, 0.1].
[0225] Assume that at a specific time t, the output h of the previous state t-1 is 0.6, the current input x t is 0.8.
[0226] Use initial weights and biases, such as W f =0.3, b f =0.05.
[0227] Calculate the output of the forget gate:
[0228]
[0229] The forget gate calculates a state update value based on the data from the current and previous time steps, indicating that the system believes there is a 61.5% probability of retaining old information, which helps the model decide whether to "forget" some old information or "remember" more new information, thereby making accurate predictions about upcoming nursing behaviors.
[0230] The embodiment further introduces an abnormal behavior detection module to enhance the system's ability to identify unconventional or abnormal surgical activities. The isolation forest algorithm is used and improved to meet the needs of real-time anomaly detection, combined with the high-dimensional space mapping formula:
[0231]
[0232] x represents a feature vector extracted from real-time data, including image features, vital signs or other relevant surgical data extracted from surgical monitoring equipment. Each feature x i The weight θi , mean μ i and standard deviation σ i Adjust the algorithm's sensitivity and response to different types of features separately.
[0233] Feature weight θ i The value range is 0.1, 1.0, indicating a slight to strong weight adjustment to indicate the importance of different features.
[0234] Mean μ i and standard deviation σ i Statistical analysis based on past surgical data was performed to ensure that the treatment of each feature was based on historical performance in actual surgical scenarios.
[0235] The system is set to extract a feature vector x = [0.8, 0.5, 0.9] from surgical monitoring, and the corresponding mean and standard deviation are set to μ = [0.7, 0.5, 0.85], σ = [0.1, 0.05, 0.1], and weight θ = [0.8, 1.0, 0.9]:
[0236]
[0237] The results show that the system successfully identifies abnormal behavior based on the deviation of the current feature vector from the preset normal range.
[0238] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. Intraoperative nursing behavior recognition system based on computer vision, characterized by The following steps are involved: First, multi-angle high-resolution cameras are deployed in the operating room to collect video data including the behaviors of surgeons and nursing staff, and professional medical staff will annotate and classify the data; Secondly, deep learning models, including convolutional neural networks (CNN) or recurrent neural networks (RNN), are used in combination with multi-task learning techniques to identify and classify nursing behaviors, and long short-term memory networks (LSTM) are used to analyze time series data in the video to identify the temporal patterns of behaviors. We further adopt a dynamic region extraction algorithm to adjust the focus area according to the dynamic changes in the scene, combining local behavior characteristics with global scene information; Finally, through behavioral intention analysis and abnormal behavior detection, a behavior prediction model is established to predict the next nursing behavior and timely detect abnormal nursing behaviors that are inconsistent with the routine surgical process.
2. The intraoperative nursing behavior recognition system based on computer vision according to claim 1 is characterized in that The operating room is equipped with multi-angle high-resolution cameras, and a focus adjustment function is used: Where x, y, z represent the relative position coordinates of the camera and the target, t is the time, and a, σ, ω are adjustment parameters to control the key surgical actions to always be captured at the best focus and viewing angle; Then a deep learning model is used to annotate the video data, and the annotation optimization model uses: Where p(t) represents the predicted behavior probability distribution, q(t) represents the actual behavior label, and λ is the learning rate; The annotation process is assisted by augmented reality technology, controlling the real-time correction equation: Where u represents the virtual information level in augmented reality, v represents the interaction data between actual and virtual information, N is the number of items, and α n , β n is a coefficient used to control the interactive accuracy of the optimization annotation process.
3. The intraoperative nursing behavior recognition system based on computer vision according to claim 1 is characterized in that The deep learning model construction includes: A customized convolutional neural network (CNN) model is deployed to identify specific visual features in the operating room, including medical tools, surgical clothing, and hand movements. The recurrent neural network (RNN) with a long short-term memory (LSTM) network is combined to analyze the time series of behaviors during surgery to capture the development process and continuity of behaviors, including tracking the delivery, use, and recovery of surgical tools. Furthermore, by constructing a multi-task learning framework, shared hidden layers are used to learn common features between multiple tasks, while retaining specific network branches for each specific nursing behavior task to control the recognition and classification of different nursing tasks commonly used in surgery, including disinfection, instrument delivery, and monitoring of patient vital signs; Finally, the LSTM network is used to control the recognition of current behavior and the prediction of future behavior trends to enhance the early warning function.
4. The intraoperative nursing behavior recognition system based on computer vision according to claim 3 is characterized in that The identification of specific visual features in the operating room covers the following steps: First, the feature extraction formula is used: To optimize the recognition ability of each pixel in the image, where x, y, z represent the pixel coordinates in the image, α i and β i is the adjustment coefficient, N represents the depth of multi-level feature extraction; Then, the long short-term memory network (LSTM) enhanced recurrent neural network (RNN) is used to perform time series analysis on the identified features to track the behavioral dynamics during the operation. The time series analysis is performed through the function: , where t represents the time step, h(t) is the input feature at time t, and γ j is the weight coefficient, λ(t) is the modulation function, and T is the time window considered; Finally, the MTL strategy is learned through multi-task learning to simultaneously handle the recognition and classification of multiple nursing behaviors. Each behavior is learned through the learning function: Optimize, where k is the number of tasks, l j is a task-specific feature, θ j are learning parameters, δ and ∈ regulate the complexity and regularization of learning.
5. The intraoperative nursing behavior recognition system based on computer vision according to claim 3 is characterized in that The process of constructing a multi-task learning framework includes: S1. First, deploy multiple shared hidden layers and use the advanced feature extraction formula: Processing input data, where X represents the input vector obtained from surgical videos and vital signs monitoring equipment, ω i and β i is the adjustment parameter, M represents the number of input features; S2. Then set up independent network branches for each nursing behavior task, and each branch uses a task-specific processing formula: Perform recognition and classification for specific tasks, where Y represents the features passed from the shared layer, θ kj is a task-specific learning parameter, N k is the number of features of the kth task, α k is the normalization constant; S3. Finally, use the comprehensive multi-task loss function: To train the network, where λ k and γ k are the weight and adjustment parameters of the kth task respectively.
6. The intraoperative nursing behavior recognition system based on computer vision according to claim 3 is characterized in that The deep learning architecture for identifying the current behavior using the LSTM network control: First use the composite function: To process the continuous time series data in surgery, where X t represents the input data at time t, W hi and W xi is the learning weight, h t-1 is the hidden state at the previous moment, σ is the activation function used to capture long-term dependencies and short-term changes in surgical behavior; Then use the real-time behavior recognition formula: Where h(s) is the LSTM output at time s, λ is the decay coefficient, and W c and b c are the weights and biases of the classification layer, which are used to identify the current surgical behavior in real time; Finally, the function of predicting future trends is: Where Δt is the prediction window length, W p and b p are the weights and biases of the prediction layer, which are used to predict future events during surgery and enhance the early warning function.
7. The intraoperative nursing behavior recognition system based on computer vision according to claim 1 is characterized in that The steps of implementing the dynamic region extraction algorithm include: S1. First, through adaptive focus technology, use the real-time analysis function: Where x, y represent the spatial coordinates on the image plane, t represents time, and α, β, γ, ω are adjustment parameters used to dynamically adjust the camera focus to track key activities in the surgical area; S2, then use multi-scale feature fusion technology, through the multi-level integral equation: Where z represents the feature input at different levels, c n and d n is a layer-specific coefficient, N represents the number of layers, and controls the accuracy of extracting key events from multi-scale perspectives; S3. Finally, a nonlinear processing function is used to integrate local and global information: Where u represents the local behavior feature set and v represents the global scene feature set, which are used to control aspects of the surgical environment to ensure a comprehensive understanding of the surgical process.
8. The intraoperative nursing behavior recognition system based on computer vision according to claim 1 is characterized in that The process of establishing the behavior prediction model includes: First, we use the behavior intention analysis model to build a model through a deep learning network, especially the long short-term memory network LSTM, using the function: f(s t )=σ(W f ·[h t-1 ,x t ]+b f ) Calculate the forget gate, where s t is the state at time t, h t-1 is the output of the previous time step, x t is the current input, W f , b f is the learning parameter, σ is the sigmoid activation function to analyze and predict the next nursing behavior; Then, the abnormal behavior detection module is introduced, and the real-time anomaly detection function improved by the isolation forest algorithm is used to map the high-dimensional space: Where x represents the feature vector extracted from real-time data, θ i , μ i , σ i They are the feature weight, mean, and standard deviation, respectively, which are used to instantly identify and warn of abnormal nursing behaviors that are inconsistent with routine surgical procedures.
Citation Information
Patent Citations
Sea surface target trajectory prediction method, terminal equipment and storage medium
CN114757426A
Smart park multi-source data dynamic monitoring and real-time analysis system and method
CN118072255A
Supply chain financial application method based on supervisible block chain private transaction
CN118229420A
Surgical instrument, behavior and target tissue joint identification method and apparatus
WO2023193238A1
Cited By
Banking outlet employee behavior prediction method based on multi-stream network
CN119418412A
Medical assistance method and system based on voice recognition and medical information knowledge base
CN119742020A
Method and system for regional automatic sound wave expelling of violation personnel
CN120014712A
Perioperative period hypothermia real-time prediction model based on multi-modal data and deep learning
CN120636803A
Flow control method based on neural network
CN120980034A
