Operation personnel video behavior identification method and system based on machine learning

Abstract

The invention relates to the technical field of behavior recognition, in particular to a machine learning-based surgical personnel video behavior recognition method and system. The method comprises the following steps: acquiring a video frame sequence and a point cloud frame sequence from a hand washing area of an operator through an RGB camera and laser ranging equipment to obtain the video frame sequence and the point cloud frame sequence; performing hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; performing action original data generation according to the video frame sequence and the hand point cloud data to obtain action original data; and extracting position and attitude information of hand key points from the original motion data to obtain video key point feature data. Visual information is captured through the RGB camera, three-dimensional depth information is obtained through the laser ranging device, fine analysis and evaluation are carried out on actions in combination with the deep learning network, and finally higher recognition precision, robustness and automation level are achieved.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition technology, and in particular to a method and system for video behavior recognition of surgical personnel based on machine learning. Background Art

[0002] Traditional methods rely on manual monitoring or video playback to manage the hand washing process of surgical staff. However, this method is time-consuming and relies on manual judgment, and it is difficult to ensure real-time and accuracy. In order to solve these problems, automated behavior recognition technology based on machine learning and computer vision has gradually been applied to the hand washing monitoring of surgical staff. The rise of machine learning-based video behavior recognition methods, especially deep learning technology, has promoted the rapid development of this field. By automatically identifying and evaluating the hand washing actions of surgical staff, large-scale, real-time, and high-precision monitoring can be achieved. However, existing methods mainly rely on a single data source (such as RGB cameras) for action recognition, which has certain limitations.

[0003] In existing methods, RGB cameras are usually used as the main data source to capture video frames and perform behavior recognition on surgical personnel. Although RGB cameras can capture rich visual information, their performance in complex scenes has some obvious limitations: Occlusion problem: In the hand-washing scenario, the surgeon's hand movements are blocked by clothing, surrounding objects or other people, resulting in the RGB camera being unable to accurately capture the details of the hand movements, affecting the accuracy of motion recognition.

[0004] Lighting changes: The hand-washing areas in hospitals are often affected by different light sources. The intensity of light, reflections, or shadows will interfere with the camera's imaging quality, thereby affecting recognition accuracy.

[0005] Lack of depth information: RGB cameras can only capture two-dimensional image information and cannot obtain depth information of the surgeon's movements (such as the forward and backward displacement of the hand, the three-dimensional movement of the torso, etc.). In the accurate recognition of hand movements, the lack of depth information leads to the loss of movement details, affecting the recognition effect.

[0006] Difficulty in identifying details: Hand movements usually involve complex and subtle changes. It is difficult to accurately capture the dynamic details of each movement using RGB images alone. Especially when identifying delicate movements such as the seven-step hand washing method, RGB cameras are unable to provide enough detailed visual information. Summary of the invention

[0007] Based on this, it is necessary to provide a method and system for surgical personnel video behavior recognition based on machine learning to solve at least one of the above technical problems.

[0008] To achieve the above purpose, a method for identifying surgical personnel's video behavior based on machine learning includes the following steps: Step S1: acquiring a video frame sequence and a point cloud frame sequence of the surgical staff's hand washing area through an RGB camera and a laser ranging device to obtain a video frame sequence and a point cloud frame sequence; performing hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; generating action raw data according to the video frame sequence and the hand point cloud data to obtain action raw data; Step S2: extracting the position and posture information of the key points of the hand from the original action data to obtain the key point feature data of the video; extracting the video motion feature from the key point feature data to obtain the video motion feature data; extracting the point cloud global feature from the original action data to obtain the point cloud global feature data; performing weighted feature fusion on the point cloud global feature data and the video motion feature data to obtain a fused feature vector; Step S3: arranging the fused feature vectors in time order to obtain time series feature sequence data; predicting the probability of action segmentation points on the time series feature sequence data to obtain action segmentation point probability data; performing action segmentation according to the action segmentation point probability data, and performing step recognition to obtain a step recognition label sequence; Step S4: obtaining standard action template data of the seven-step hand washing method; performing time regularization matching processing according to the standard action template data of the seven-step hand washing method and the step identification label sequence to obtain time regularization matching distance data; performing action quality evaluation according to the time regularization matching distance data to obtain an action quality score; Step S5: Perform compliance verification on the action quality score to obtain overall compliance assessment data; generate a compliance report based on the overall compliance assessment data to obtain a compliance report to implement the hand washing and disinfection monitoring operation of the surgical staff.

[0009] The present invention obtains video and point cloud data through RGB camera and laser rangefinder, and performs hand point cloud segmentation, which can effectively obtain multimodal data of the hand washing area of ​​the surgical staff, accurately extract hand information, eliminate background interference, and provide high-quality data basis for subsequent action recognition and analysis. Extracting hand key point information and point cloud global features from the original action data, and performing weighted feature fusion, can effectively combine multimodal information, more comprehensively describe hand movements, and improve feature expression ability and recognition accuracy. By performing time series analysis, action segmentation and step recognition on the fused feature vector, the continuous hand washing action can be decomposed into discrete steps, and the category of each step can be identified, providing fine-grained analysis results for subsequent action quality evaluation. By performing time regularization matching with the standard action template, and combining spatial matching, time matching and action detail scoring, each hand washing step can be refined for quality evaluation, providing more objective and comprehensive evaluation results. By performing compliance verification on the action quality score and generating a compliance report, the automated monitoring and evaluation of the hand washing operation of the surgical staff can be realized, and the level of hygiene management in the operating room can be improved. Therefore, the present invention provides a method for identifying surgical personnel's video behavior based on machine learning. By combining RGB cameras and laser ranging equipment, the deep learning method is used to effectively overcome the limitations of the single data source in the traditional surgical personnel video behavior recognition method, especially in the application of hand washing video behavior recognition monitoring scene. This multimodal data fusion method uses RGB cameras to capture visual information, laser ranging equipment to obtain three-dimensional depth information, and then combines deep learning networks to perform detailed analysis and evaluation of actions, ultimately achieving higher recognition accuracy, robustness, and automation levels.

[0010] Preferably, step S1 comprises the following steps: Step S11: Use the RGB camera to detect the screen motion of the surgical staff's hand washing area. When continuous motion changes are detected, collect video stream data to obtain a video frame sequence; synchronously collect point cloud data according to the laser ranging device and the timestamp of the video frame sequence to obtain a point cloud frame sequence; Step S12: performing point cloud denoising processing on the point cloud frame sequence to obtain point cloud denoised data; Step S13: performing hand point cloud segmentation on the point cloud denoising data to obtain hand point cloud data; Step S14: extracting key frames from the video frame sequence to obtain video key frame data; Step S15: performing multimodal data alignment on the hand point cloud data and the video key frame data to obtain multimodal data alignment data; generating action raw data according to the multimodal data alignment data to obtain action raw data.

[0011] The present invention triggers video stream data acquisition through picture motion detection, which can effectively avoid the acquisition of invalid static pictures and save storage space and computing resources. At the same time, the point cloud data is synchronously acquired according to the timestamp of the video frame sequence, which ensures the time synchronization of RGB data and depth data, and lays the foundation for subsequent multimodal data fusion. Point cloud denoising processing can remove noise points in point cloud data, improve the quality of point cloud data, make subsequent hand point cloud segmentation more accurate, reduce the possibility of mis-segmentation, and improve the robustness of the algorithm. Hand point cloud segmentation can separate the hand point cloud from the background environment, remove the interference of background objects such as wash basins and walls, so that subsequent feature extraction and action recognition are more focused on the hand area, and improve recognition accuracy. Key frame extraction can reduce data redundancy and reduce computational complexity, while retaining the key information of the action, which is conducive to subsequent action analysis and recognition, and improves the efficiency of the algorithm. Multimodal data alignment can align the hand point cloud data and the video key frame data in time and space, ensure the consistency of the two data, provide reliable data input for subsequent multimodal feature fusion, and improve the expression ability of features.

[0012] Preferably, step S13 comprises the following steps: Step S131: performing preliminary detection of the hand area using the point cloud denoising data and a preset depth range threshold to obtain preliminary hand point cloud data; Step S132: using the pre-trained hand detection model to perform hand contour detection on the video frame sequence to obtain hand contour data; Step S133: Mapping the hand contour data to the preliminary hand point cloud data, removing non-hand point clouds, and obtaining RGB auxiliary hand point cloud data; Step S134: performing normal estimation on the RGB-assisted hand point cloud data to obtain normal estimation data; Step S135: separating the hand point cloud from the background objects on the RGB-assisted hand point cloud data according to the normal estimation data to obtain refined hand point cloud data; Step S136: performing hand point cloud morphology correction on the refined hand point cloud data to obtain hand point cloud segmentation data.

[0013] The present invention uses the depth range threshold to perform preliminary detection of the hand area, which can quickly and effectively remove point clouds that are too far or too close in the background, preliminarily screen out the point cloud data of the hand area, reduce the amount of data for subsequent processing, and improve efficiency. Using the pre-trained hand detection model to perform hand contour detection, the position and contour information of the hand in the image can be accurately obtained, providing important reference information for subsequent point cloud segmentation and improving segmentation accuracy. Mapping the hand contour data to the preliminary hand point cloud data and removing the point cloud outside the contour can effectively remove the point cloud noise that does not belong to the hand, further improve the accuracy of hand point cloud segmentation, and reduce mis-segmentation. Normal estimation of RGB-assisted hand point cloud data can obtain the normal vector information of each point cloud, provide important geometric features for subsequent distinction between hand point clouds and background objects, and improve the robustness of segmentation. Separating the hand point cloud from the background objects based on the normal estimation data can effectively remove the interference of background objects such as the edge of the wash basin and water flow, obtain more refined hand point cloud data, and improve the accuracy and completeness of segmentation. Morphological correction of the refined hand point cloud data, such as filling holes and smoothing edges, can further optimize the shape and contour of the hand point cloud, making the segmentation result more complete and beautiful, and improving the effectiveness of subsequent feature extraction.

[0014] Preferably, step S2 comprises the following steps: Step S21: extracting the position and posture information of the key points of the hand from the original action data using the pre-trained human posture estimation model to obtain the key point feature data of the video; Step S22: extracting video motion features from the video key point feature data to obtain video motion feature data; Step S23: extracting local features of point cloud from the original action data to obtain local feature data of point cloud; extracting global features of point cloud from the local feature data of point cloud to obtain global feature data of point cloud; Step S24: performing spatiotemporal alignment processing on the point cloud global feature data and the video motion feature data according to the timestamp of the original action data to obtain spatiotemporal alignment feature data; Step S25: Perform attention weighted processing of video features and point cloud features on the spatiotemporal alignment feature data to obtain attention weighted feature data; perform feature fusion on the attention weighted feature data to obtain a fused feature vector.

[0015] The present invention can accurately capture the fine movements of the hand by extracting the position and posture information of the key points of the hand using a pre-trained human posture estimation model, provide important feature information for subsequent action recognition and analysis, and improve the recognition accuracy. By calculating the motion features such as displacement, velocity and acceleration of the key points, the motion trajectory and dynamic changes of the hand can be effectively described, and more abundant feature information can be provided for subsequent action analysis, thereby improving the accuracy of action recognition. Local feature extraction and global feature extraction of point cloud data can effectively capture the local details and overall structural information of the hand shape, provide a more comprehensive point cloud feature expression for subsequent feature fusion, and improve recognition robustness. By aligning the point cloud global feature data and video motion feature data by timestamp, it can ensure that the data of two different modalities are synchronized in time, provide consistent data input for subsequent feature fusion, and improve the effectiveness of fusion features. Attention weighted processing of spatiotemporal alignment feature data can highlight the feature information of important time steps, and adaptively weight the video features and point cloud features, thereby improving the expression ability and discrimination of the fusion features, and ultimately improving the accuracy of action recognition.

[0016] Preferably, step S22 comprises the following steps: Step S221: extracting continuous key frame key points from the video key point feature data to obtain continuous key frame key point data; extracting depth map data from the video key point feature data to obtain depth map data; Step S222: Obtain camera intrinsic parameters of the RGB camera; perform three-dimensional coordinate conversion on the key point data of the continuous key frames and the depth map data using the camera intrinsic parameters of the RGB camera to obtain three-dimensional coordinate conversion data; Step S223: performing key point displacement calculation on the three-dimensional coordinate transformation data to obtain key point displacement data; performing key point velocity calculation based on the key point displacement data to obtain key point velocity data; performing key point acceleration calculation based on the key point velocity data to obtain key point acceleration data; Step S224: generating motion feature vectors according to the key point displacement data, the key point velocity data and the key point acceleration data to obtain motion feature vector data; Step S225: combining the key point motion feature vectors of each frame image with the motion feature vector data to obtain frame motion feature data; and performing data time sorting on the frame motion feature data to obtain video motion feature data.

[0017] The present invention can effectively reduce data redundancy by extracting key point data and depth map data of continuous key frames, while retaining complete hand information on key frames, and providing the necessary data basis for subsequent three-dimensional coordinate conversion and motion feature extraction. By using camera internal parameters to convert two-dimensional key point coordinates and depth information into three-dimensional coordinates, the hand key points can be converted from image space to three-dimensional space, more accurately describing the motion trajectory and posture changes of the hand, and improving the expressiveness of features. By calculating the displacement, velocity and acceleration of key points, the dynamic characteristics of hand movement, such as movement direction, velocity change and acceleration change, can be captured, providing richer feature information for subsequent action recognition and improving recognition accuracy. Combining the key point displacement, velocity and acceleration data into a motion feature vector, multiple motion-related features can be fused together to form a more expressive feature vector, thereby improving the discrimination of action recognition. Combining the key point motion feature vectors of each frame image and arranging them in chronological order can construct a complete hand motion sequence, retain information in the time dimension, and provide necessary data support for subsequent temporal action analysis and recognition.

[0018] Preferably, step S3 comprises the following steps: Step S31: arranging the fused feature vectors in time sequence to obtain time series feature sequence data; Step S32: Perform Bi-LSTM encoding on the time series feature sequence data to obtain Bi-LSTM encoded feature data; Step S33: inputting the feature vector of each time step in the Bi-LSTM encoded feature data into a preset fully connected layer, performing low-dimensional feature representation conversion, and obtaining fully connected layer feature data; inputting the feature value of each time step in the fully connected layer feature data into a preset Sigmoid activation function, performing Sigmoid function activation, obtaining Sigmoid activated data, and using the Sigmoid activated data as action segmentation point probability data; Step S34: determining the action segmentation points according to the action segmentation point probability data to obtain action segmentation point data; using the action segmentation point data to divide the temporal feature sequence data into action segments to obtain an action segment set; performing a maximum pooling operation on the action segment set to obtain action segment feature data; Step S35: inputting the action clip feature data into a preset Softmax classifier to perform step classification probability prediction to obtain step classification probability data; Step S36: Generate step identification labels according to the step classification probability data to obtain step identification label data; arrange the step identification label data in time sequence to obtain a step identification label sequence.

[0019] The present invention arranges the fusion feature vectors in chronological order to form time series feature sequence data, retains the information on the time dimension, and provides a basis for subsequent time series analysis and action segmentation. Using the Bi-LSTM network to encode the time series feature sequence data can effectively capture the long-term dependency and context information in the sequence data, so as to better represent the dynamic characteristics of hand movements and improve the accuracy of action segmentation and recognition. Inputting the Bi-LSTM encoded features into the fully connected layer and the Sigmoid activation function can convert high-dimensional features into action segmentation point probabilities, provide a probability basis for subsequent action segmentation, and improve the accuracy of segmentation. Determine the segmentation point according to the probability of the action segmentation point, and divide the time series feature sequence into action segments, so that the continuous hand washing action can be decomposed into discrete steps, which is convenient for subsequent classification and recognition of each step, and improves the efficiency and accuracy of recognition. The maximum pooling operation can extract the most representative features in each action segment, reduce the feature dimension, and reduce the amount of calculation. Using the Softmax classifier to classify the action segment features, it is possible to predict which hand washing step each segment belongs to, and give the probability distribution of each step, so as to provide confidence for subsequent step recognition and improve the reliability of recognition. By generating a step identification label sequence based on the step classification probability, each action segment can be mapped to a specific hand washing step, and finally a complete step sequence of the hand washing process is obtained, providing a basis for subsequent action quality evaluation.

[0020] Preferably, step S4 comprises the following steps: Step S41: Acquire standard action template data of the seven-step hand washing method; extract the start and end time points of each action segment from the step identification label sequence to obtain the start and end time data of the action segment; extract the corresponding time segment feature sequence data from the time series feature sequence data according to the start and end time data of the action segment to obtain the segmented action segment feature sequence data; Step S42: performing time-regular matching processing on the segmented action segment feature sequence data and the standard action template data to obtain time-regular matching distance data; Step S43: Calculate the spatial matching degree according to the time-warping matching distance data to obtain spatial matching degree data; Step S44: Calculate the time matching degree according to the time regularization matching distance data to obtain the time matching degree data; Step S45: extracting action detail features from the segmented action segment feature sequence data to obtain action detail feature data; Step S46: scoring the action detail feature data according to the standard action template data to obtain action detail scoring data; Step S47: Perform weighted summation on the spatial matching data, the temporal matching data and the action detail score data to obtain the action quality score.

[0021] The present invention obtains standard action template data and extracts the segmented action segment feature sequence according to the step identification label sequence, thereby providing a benchmark and evaluation object for subsequent action matching and quality evaluation. Using the DTW algorithm for time-warping matching can effectively handle the difference in action execution speed, thereby more accurately comparing the similarity between the actions of the surgeon and the standard actions, and improving the accuracy of the evaluation. Based on the time-warping matching distance, the spatial matching degree can be calculated, and the similarity between the actions performed by the surgeon and the standard actions in spatial position can be quantified, providing an important indicator for action quality evaluation. Based on the time-warping matching distance, the time matching degree can be calculated, and the similarity between the time for the surgeon to perform each step and the standard time can be quantified, thereby evaluating the rhythm and efficiency of the action. Extracting action detail features, such as the motion trajectory, speed and acceleration of key points, can more carefully depict the action execution of the surgeon, and provide richer detail information for action quality evaluation. According to the standard action template data, the action detail features are scored, and the details of the surgeon's actions can be compared with the standard requirements, thereby evaluating the standardization and accuracy of the action. By taking the weighted sum of the spatial matching, temporal matching and action detail scores, we can comprehensively consider multiple evaluation indicators to obtain a more comprehensive and objective action quality score and improve the reliability of the evaluation.

[0022] Preferably, step S42 includes the following steps: Step S421: extracting action segment feature sequences from the segmented action segment feature sequence data to obtain action segment feature sequences; Step S422: extracting a standard action template feature sequence from the standard action template data to obtain a standard action template feature sequence; Step S423: normalizing the time series length of the action clip feature sequence and the standard action template feature sequence to obtain a standardized action clip feature sequence and a standardized standard action template feature sequence; Step S424: using the DTW algorithm to perform dynamic time warping path calculation on the standardized action clip feature sequence and the standardized standard action template feature sequence to obtain a DTW matching path; Step S425: Calculate the time-warping matching distance according to the DTW matching path to obtain the time-warping matching distance data.

[0023] The present invention clarifies the evaluation object by extracting the feature sequence corresponding to the step to be evaluated from the segmented action clip feature sequence data, and provides specific action data for subsequent matching and comparison. The standard action template feature sequence corresponding to the step to be evaluated is extracted from the standard action template data, and an evaluation benchmark is established, which provides a reference standard for subsequent matching and comparison. The time series length of the action clip feature sequence and the standard action template feature sequence are standardized to eliminate the influence of the difference in the speed of execution of different individuals, so that the DTW algorithm can more accurately compare the similarity of the action. The DTW algorithm is used to calculate the dynamic time warping path of the two standardized sequences, and the best nonlinear matching relationship between the two sequences is found, which can effectively deal with the deformation and asynchrony problems on the time axis and improve the matching accuracy. The time warping matching distance is calculated according to the DTW matching path, and the similarity between the action clip and the standard action template is quantified, which provides an important quantitative index for the subsequent action quality evaluation.

[0024] Preferably, step S5 comprises the following steps: Step S51: using a preset quality score threshold to perform step quality judgment on the action quality score to obtain step quality judgment data; Step S52: performing real-time feedback on step quality according to step quality judgment data to obtain real-time feedback information data; Step S53: Perform step completion statistics according to the step quality judgment data to obtain step completion data; Step S54: performing compliance assessment of the overall hand washing process according to the step completion status data to obtain overall compliance assessment data; Step S55: Generate a compliance report based on the step quality judgment data, the step completion status data and the overall compliance assessment data to obtain a compliance report.

[0025] The present invention uses a preset quality scoring threshold to judge the step quality of the action quality score, which can convert complex scoring results into intuitive pass / fail judgments, which is convenient for subsequent feedback and statistics. Feedback of the step quality judgment results to surgical personnel in real time can help them promptly understand whether their hand washing operations are standardized, and make adjustments based on the feedback information to improve the quality of hand washing. Statistics on the completion of the steps can record the completion status and pass rate of each step, provide data support for subsequent overall compliance assessments, and help identify steps that need to be improved. Compliance assessment of the overall hand washing process is performed based on the step completion data, and a comprehensive assessment can be made as to whether the hand washing operations of surgical personnel meet the standard requirements. Generate a compliance report, which can fully record the hand washing operation data and evaluation results of surgical personnel, facilitate subsequent review, analysis and management, and provide data support for improving the level of hygiene management in the operating room.

[0026] Preferably, the present invention further provides a surgical personnel video behavior recognition system based on machine learning, which is used to execute the surgical personnel video behavior recognition method based on machine learning as described above. The surgical personnel video behavior recognition system based on machine learning includes: The multimodal data capture module is used to obtain the video frame sequence and point cloud frame sequence of the surgical staff's hand washing area through the RGB camera and the laser ranging device to obtain the video frame sequence and the point cloud frame sequence; perform hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; generate action raw data according to the video frame sequence and the hand point cloud data to obtain the action raw data; The spatiotemporal feature fusion module is used to extract the position and posture information of the key points of the hand from the original action data to obtain the video key point feature data; extract the video motion feature from the video key point feature data to obtain the video motion feature data; extract the point cloud global feature from the original action data to obtain the point cloud global feature data; perform weighted feature fusion on the point cloud global feature data and the video motion feature data to obtain the fused feature vector; The action segmentation and step recognition module is used to perform time sequence arrangement processing on the fused feature vector to obtain time series feature sequence data; perform action segmentation point probability prediction on the time series feature sequence data to obtain action segmentation point probability data; perform action segmentation based on the action segmentation point probability data, and perform step recognition to obtain a step recognition label sequence; The action quality assessment module is used to obtain the standard action template data of the seven-step hand washing method; perform time-regular matching processing according to the standard action template data of the seven-step hand washing method and the step identification label sequence to obtain time-regular matching distance data; perform action quality assessment according to the time-regular matching distance data to obtain an action quality score; The action compliance verification module is used to verify the compliance of the action quality score to obtain the overall compliance assessment data; the compliance report is generated based on the overall compliance assessment data to obtain the compliance report to realize the hand washing and disinfection monitoring operation of the surgical staff.

[0027] The present invention obtains video and point cloud data through RGB camera and laser rangefinder, and performs hand point cloud segmentation, which can effectively obtain multimodal data of the hand washing area of ​​the surgical staff, accurately extract hand information, eliminate background interference, and provide high-quality data basis for subsequent action recognition and analysis. Extracting hand key point information and point cloud global features from the original action data, and performing weighted feature fusion, can effectively combine multimodal information, more comprehensively describe hand movements, and improve feature expression ability and recognition accuracy. By performing time series analysis, action segmentation and step recognition on the fused feature vector, the continuous hand washing action can be decomposed into discrete steps, and the category of each step can be identified, providing fine-grained analysis results for subsequent action quality evaluation. By performing time regularization matching with the standard action template, and combining spatial matching, time matching and action detail scoring, each hand washing step can be refined for quality evaluation, providing more objective and comprehensive evaluation results. By performing compliance verification on the action quality score and generating a compliance report, the automated monitoring and evaluation of the hand washing operation of the surgical staff can be realized, and the level of hygiene management in the operating room can be improved. Therefore, the present invention provides a method for identifying surgical personnel's video behavior based on machine learning. By combining RGB cameras and laser ranging equipment, the deep learning method is used to effectively overcome the limitations of the single data source in the traditional surgical personnel video behavior recognition method, especially in the application of hand washing video behavior recognition monitoring scene. This multimodal data fusion method uses RGB cameras to capture visual information, laser ranging equipment to obtain three-dimensional depth information, and then combines deep learning networks to perform detailed analysis and evaluation of actions, ultimately achieving higher recognition accuracy, robustness, and automation levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the steps of video behavior recognition of surgical personnel based on machine learning; Figure 2 for Figure 1 Detailed implementation steps of step S2 in the flowchart; Figure 3 for Figure 1 Detailed implementation steps of step S4 in FIG.

[0029] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0030] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different network and / or processor methods and / or microcontroller methods.

[0032] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0033] To achieve this, please refer to Figures 1 to 3 , a method for identifying surgical personnel's video behavior based on machine learning, comprising the following steps: Step S1: acquiring a video frame sequence and a point cloud frame sequence of the surgical staff's hand washing area through an RGB camera and a laser ranging device to obtain a video frame sequence and a point cloud frame sequence; performing hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; generating action raw data according to the video frame sequence and the hand point cloud data to obtain action raw data; Step S2: extracting the position and posture information of the key points of the hand from the original action data to obtain the key point feature data of the video; extracting the video motion feature from the key point feature data to obtain the video motion feature data; extracting the point cloud global feature from the original action data to obtain the point cloud global feature data; performing weighted feature fusion on the point cloud global feature data and the video motion feature data to obtain a fused feature vector; Step S3: arranging the fused feature vectors in time order to obtain time series feature sequence data; predicting the probability of action segmentation points on the time series feature sequence data to obtain action segmentation point probability data; performing action segmentation according to the action segmentation point probability data, and performing step recognition to obtain a step recognition label sequence; Step S4: obtaining standard action template data of the seven-step hand washing method; performing time regularization matching processing according to the standard action template data of the seven-step hand washing method and the step identification label sequence to obtain time regularization matching distance data; performing action quality evaluation according to the time regularization matching distance data to obtain an action quality score; Step S5: Perform compliance verification on the action quality score to obtain overall compliance assessment data; generate a compliance report based on the overall compliance assessment data to obtain a compliance report to implement the hand washing and disinfection monitoring operation of the surgical staff.

[0034] In the embodiment of the present invention, reference Figure 1 The above is a schematic diagram of the steps of the surgical personnel video behavior recognition method based on machine learning of the present invention. In this example, the surgical personnel video behavior recognition method based on machine learning includes the following steps: Step S1: acquiring a video frame sequence and a point cloud frame sequence of the surgical staff's hand washing area through an RGB camera and a laser ranging device to obtain a video frame sequence and a point cloud frame sequence; performing hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; generating action raw data according to the video frame sequence and the hand point cloud data to obtain action raw data; In the embodiment of the present invention, the video frame sequence and point cloud frame sequence of the surgical staff's hand washing area are synchronously collected by an RGB camera and a laser ranging device. The hand point cloud is segmented in the point cloud frame sequence by using methods such as depth range threshold, pre-trained YOLOv5 hand detection model, camera intrinsic parameter matrix, and normal estimation to remove background and noise interference and obtain accurate hand point cloud data. Finally, the segmented hand point cloud data is aligned with the corresponding video frame data according to the timestamp and combined into multi-modal action raw data.

[0035] Step S2: extracting the position and posture information of the key points of the hand from the original action data to obtain the key point feature data of the video; extracting the video motion feature from the key point feature data to obtain the video motion feature data; extracting the point cloud global feature from the original action data to obtain the point cloud global feature data; performing weighted feature fusion on the point cloud global feature data and the video motion feature data to obtain a fused feature vector; In the embodiment of the present invention, the pre-trained MediaPipe Hands model is used to extract the position and posture information of the key points of the hand from the raw action data to generate the key point feature data of the video. The video motion feature data is extracted by calculating the displacement, velocity and acceleration of the key points between consecutive frames. At the same time, the PointNet++ network is used to perform local feature extraction and global feature aggregation on the point cloud data in the raw action data to obtain the point cloud global feature data. Finally, the video motion feature data and the point cloud global feature data are timestamp aligned, and weighted fusion is performed through the attention mechanism to generate a fused feature vector.

[0036] Step S3: arranging the fused feature vectors in time order to obtain time series feature sequence data; predicting the probability of action segmentation points on the time series feature sequence data to obtain action segmentation point probability data; performing action segmentation according to the action segmentation point probability data, and performing step recognition to obtain a step recognition label sequence; In an embodiment of the present invention, the fused feature vectors are arranged in chronological order to form time series feature sequence data. The Bi-LSTM network is used to encode the time series feature sequence data and learn its time dependency. The output features of the Bi-LSTM are input into the fully connected layer and the Sigmoid activation function to predict the probability that each time step is an action segmentation point. The action segmentation point is determined according to the probability value, and the time series feature sequence is divided into several action segments. The maximum pooling operation is performed on each action segment to extract the segment features, and the Softmax classifier is used for step recognition, and finally a step recognition label sequence is obtained.

[0037] Step S4: obtaining standard action template data of the seven-step hand washing method; performing time regularization matching processing according to the standard action template data of the seven-step hand washing method and the step identification label sequence to obtain time regularization matching distance data; performing action quality evaluation according to the time regularization matching distance data to obtain an action quality score; In an embodiment of the present invention, pre-collected standard action template data of the seven-step hand washing method is obtained, which contains a feature sequence of the complete action process of each step. The start and end time of each action segment is extracted from the step identification label sequence, and the feature sequence of the corresponding time period is extracted from the time series feature sequence data. The DTW algorithm is used to calculate the time-regular matching distance between the segmented action segment feature sequence and the standard action template feature sequence. The spatial matching degree and the temporal matching degree are calculated based on the matching distance. At the same time, the action detail features are extracted, and the action details are scored according to the predefined scoring criteria. Finally, the spatial matching degree, the temporal matching degree and the action detail score are weighted and summed to obtain the action quality score.

[0038] Step S5: verifying the compliance of the action quality score to obtain overall compliance evaluation data; generating a compliance report based on the overall compliance evaluation data to obtain a compliance report to implement the hand washing and disinfection monitoring operation of the surgical staff; In the embodiment of the present invention, the action quality score of each hand washing step is judged by using a preset quality score threshold, and the judgment result is fed back to the surgical staff in real time. The completion status of each step is counted, such as the number of completions and the number of qualified times. According to the step completion status, the compliance of the overall hand washing process is evaluated. Finally, according to the step quality judgment data, the step completion status data and the overall compliance evaluation data, a compliance report containing detailed evaluation results and improvement suggestions is generated.

[0039] Preferably, step S1 comprises the following steps: Step S11: Use the RGB camera to detect the screen motion of the surgical staff's hand washing area. When continuous motion changes are detected, collect video stream data to obtain a video frame sequence; synchronously collect point cloud data according to the laser ranging device and the timestamp of the video frame sequence to obtain a point cloud frame sequence; Step S12: performing point cloud denoising processing on the point cloud frame sequence to obtain point cloud denoised data; Step S13: performing hand point cloud segmentation on the point cloud denoising data to obtain hand point cloud data; Step S14: extracting key frames from the video frame sequence to obtain video key frame data; Step S15: performing multimodal data alignment on the hand point cloud data and the video key frame data to obtain multimodal data alignment data; generating action raw data according to the multimodal data alignment data to obtain action raw data.

[0040] In an embodiment of the present invention, first, an RGB camera is installed directly above the hand-washing area for surgical personnel, and a suitable frame rate (e.g., 30fps) and resolution (e.g., 1920x1080) are set. The camera is connected to a computer equipped with a motion detection algorithm. The algorithm is based on the inter-frame difference method, which compares the pixel changes between consecutive frames in real time and calculates the proportion of changed pixels. When the proportion exceeds a preset threshold (e.g., 5%) and lasts for a certain period of time (e.g., 0.5 seconds), it is determined that continuous motion changes are detected, triggering video stream data acquisition. The collected video frame sequence is marked with a timestamp and stored. At the same time, a laser ranging device, such as Kinect or RealSense, is installed near the hand-washing area, which is also connected to the computer. The laser ranging device collects depth data at a frequency synchronized with the RGB camera. Using the timestamp of the RGB video frame as a reference, the point cloud frame with the closest timestamp is synchronously associated with the corresponding video frame through timestamp matching, and a point cloud frame sequence is generated and stored.

[0041] After acquiring the point cloud frame sequence, the point cloud data is denoised using a statistical filter. The statistical filter sets a neighborhood search radius (e.g. 0.02 meters) and calculates the average distance of all points in its neighborhood for each point. If the average distance from the point to all points in the neighborhood is greater than the preset standard deviation multiple threshold (e.g. 2 standard deviations), the point is considered a noise point and is removed. Repeat this process until all points are processed, and finally the point cloud denoised data is obtained.

[0042] The hand area is preliminarily detected using the point cloud denoising data and the preset depth range threshold (e.g., 0.3 meters to 1.0 meters), and the point cloud data with depth values ​​within this range is screened out to obtain the preliminary hand point cloud data. Then, the YOLOv5 model is used to detect the hand contour of the synchronized video frames to obtain the bounding box coordinates of the two-dimensional hand contour. The bounding box coordinates are projected into the three-dimensional space through the camera intrinsic parameter matrix to obtain the corresponding rectangular area in the three-dimensional space. The preliminary hand point cloud data is compared with the three-dimensional rectangular area, and the point cloud outside the rectangular area is eliminated to obtain the RGB-assisted hand point cloud data. Then, the local surface normal estimation method is used to calculate the normal vector of each point in the RGB-assisted hand point cloud data to obtain the normal estimation data. According to the normal estimation data, the points whose normal vector direction and the preset wash basin plane normal vector direction are greater than the threshold (e.g., 60 degrees) are eliminated to separate the hand point cloud from the background objects and obtain the refined hand point cloud data. Finally, morphological filtering operations, such as opening and closing operations, are performed on the refined hand point cloud data to fill small holes in the hand point cloud and smooth the edges of the hand point cloud to obtain the hand point cloud segmentation data.

[0043] A key frame extraction method based on inter-frame difference is adopted. First, the sum of the pixel difference between two adjacent frames in the video frame sequence is calculated. Then, a threshold (for example, 10000) is set. When the sum of the pixel difference between two adjacent frames is greater than the threshold, the latter frame is extracted as a key frame. In addition, in order to avoid extracting too dense key frames, a minimum time interval (for example, 0.5 seconds) is set. If the time interval between the current frame and the previous key frame is less than the minimum time interval, the current frame will not be extracted as a key frame even if the sum of the pixel difference is greater than the threshold. Finally, the video key frame data is obtained.

[0044] Match the hand point cloud segmentation data with the video key frame data according to the timestamp. For each video key frame, select the hand point cloud segmentation data with the closest timestamp as its corresponding point cloud data. Then, project the two-dimensional hand contour bounding box of the video key frame into the three-dimensional space through the camera intrinsic parameter matrix to obtain the three-dimensional bounding box. Compare the matched point cloud data with the three-dimensional bounding box, calculate the distance between the center of the point cloud and the center of the three-dimensional bounding box, and perform a translation operation so that the center of the point cloud coincides with the center of the three-dimensional bounding box. Finally, combine the aligned hand point cloud data and the corresponding video key frame data to generate multimodal data alignment data. Arrange all multimodal data alignment data in chronological order to form the original action data.

[0045] Preferably, step S13 comprises the following steps: Step S131: performing preliminary detection of the hand area using the point cloud denoising data and a preset depth range threshold to obtain preliminary hand point cloud data; Step S132: using the pre-trained hand detection model to perform hand contour detection on the video frame sequence to obtain hand contour data; Step S133: Mapping the hand contour data to the preliminary hand point cloud data, removing non-hand point clouds, and obtaining RGB auxiliary hand point cloud data; Step S134: performing normal estimation on the RGB-assisted hand point cloud data to obtain normal estimation data; Step S135: separating the hand point cloud from the background objects on the RGB-assisted hand point cloud data according to the normal estimation data to obtain refined hand point cloud data; Step S136: performing hand point cloud morphology correction on the refined hand point cloud data to obtain hand point cloud segmentation data.

[0046] In an embodiment of the present invention, after acquiring the point cloud denoising data, a depth range threshold is set, for example, 0.3 meters to 1.0 meters, and the threshold is pre-set according to the distance between the wash basin and the depth camera and the size of the human hand. Traverse each point in the point cloud denoising data and calculate the distance from each point to the depth camera. Points whose distance values ​​fall within the preset depth range threshold (0.3 meters to 1.0 meters) are retained, and the remaining points are discarded. The retained points are formed into new point cloud data, that is, preliminary hand point cloud data. This step effectively removes the point cloud noise that is too far or too close in the background, and preliminarily extracts the point cloud data of the hand area.

[0047] Use the pre-trained YOLOv5 hand detection model to detect hand contours on video frame sequences. Input each frame of video image into the YOLOv5 model, and the model outputs the detected hand bounding box information, including the upper left corner coordinates (x1, y1), lower right corner coordinates (x2, y2) and confidence score of the bounding box. Set a confidence threshold, such as 0.8, to filter out bounding boxes with confidence scores below the threshold. Save the retained bounding box coordinate information as hand contour data for subsequent processing.

[0048] Get the intrinsic and extrinsic matrix of the RGB camera. Use the camera intrinsic matrix to project the two-dimensional hand contour bounding box obtained in S132 into three-dimensional space to form a three-dimensional rectangular area. Then, traverse each point in the preliminary hand point cloud data obtained in S131 to determine whether the point is located in the three-dimensional rectangular area. Keep the points in the three-dimensional rectangular area and remove the rest. The retained point cloud data is the RGB-assisted hand point cloud data. Through this step, RGB image information is used to assist point cloud segmentation, and the point cloud that does not belong to the hand in the preliminary hand point cloud data is further removed to improve the accuracy of hand point cloud segmentation.

[0049] Normal estimation is performed based on local surface fitting. For each point in the RGB-assisted hand point cloud data, first search for k nearest neighbor points (for example, k=10) in its neighborhood. Then, the covariance matrix of these k+1 points (including the center point itself) is calculated using principal component analysis (PCA). The eigenvalues ​​of the covariance matrix are sorted from small to large, and the eigenvector corresponding to the smallest eigenvalue is the normal vector of the point. The normal vectors of all points are saved to obtain normal estimation data.

[0050] Preset the normal vector direction of the wash basin plane, for example, vertically upward (0, 1, 0). Traverse each point in the RGB-assisted hand point cloud data and calculate the angle between its normal vector and the wash basin plane normal vector. Set an angle threshold, for example, 60 degrees. Remove points with angles greater than the threshold, and consider these points to be background objects, such as the edge of the wash basin or other debris. The remaining point cloud data is the refined hand point cloud data, which more accurately expresses the shape of the hand.

[0051] The refined hand point cloud data is processed by morphological opening and closing operations. First, the opening operation is performed, that is, corrosion first and then expansion, to remove small burrs and noise in the hand point cloud data. Then the closing operation is performed, that is, expansion first and then corrosion, to fill the small holes in the hand point cloud data to make the hand point cloud data more complete. The spherical structural element is selected as the structural element of the morphological operation, and the radius is set to 0.01 meters. The point cloud data after morphological processing is the final hand point cloud segmentation data, which has a smoother surface and clearer contours.

[0052] Preferably, step S2 comprises the following steps: Step S21: extracting the position and posture information of the key points of the hand from the original action data using the pre-trained human posture estimation model to obtain the key point feature data of the video; Step S22: extracting video motion features from the video key point feature data to obtain video motion feature data; Step S23: extracting local features of point cloud from the original action data to obtain local feature data of point cloud; extracting global features of point cloud from the local feature data of point cloud to obtain global feature data of point cloud; Step S24: performing spatiotemporal alignment processing on the point cloud global feature data and the video motion feature data according to the timestamp of the original action data to obtain spatiotemporal alignment feature data; Step S25: Perform attention weighted processing of video features and point cloud features on the spatiotemporal alignment feature data to obtain attention weighted feature data; perform feature fusion on the attention weighted feature data to obtain a fused feature vector.

[0053] As an example of the present invention, refer to Figure 2 As shown, in this example, step S2 includes: Step S21: extracting the position and posture information of the key points of the hand from the original action data using the pre-trained human posture estimation model to obtain the key point feature data of the video; In an embodiment of the present invention, a pre-trained MediaPipe Hands model is used to detect hand key points for each frame of video image in the action raw data. The MediaPipe Hands model outputs the three-dimensional coordinate information of 21 key points of each hand, including wrist, palm, and finger joints. The three-dimensional coordinate information of these key points and the hand posture information (for example, finger bending angle, palm direction, etc.) are saved to form the video key point feature data. Since the action raw data contains RGB images and depth information, MediaPipe Hands can use depth information to improve the accuracy and robustness of key point detection, especially in the case of hand occlusion or lighting changes.

[0054] Step S22: extracting video motion features from the video key point feature data to obtain video motion feature data; In an embodiment of the present invention, the key point feature data of the video is processed to extract the video motion features. First, the key points of the continuous key frames are extracted, and the three-dimensional displacement vector of each key point between the continuous frames is calculated. Then, the speed and acceleration of each key point are calculated based on the displacement vector. The displacement, speed and acceleration vectors of each key point are combined into a motion feature vector. Finally, the motion feature vectors of all the key points of each frame image are connected to form the motion feature data of the frame. The motion feature data of all frames are arranged in chronological order to obtain the video motion feature data, which contains the dynamic information of the hand movement.

[0055] Step S23: extracting local features of point cloud from the original action data to obtain local feature data of point cloud; extracting global features of point cloud from the local feature data of point cloud to obtain global feature data of point cloud; In an embodiment of the present invention, local features are extracted from the point cloud data in the original action data. A neighborhood search is performed on each hand point cloud data according to a fixed radius (for example, 0.02 meters) to construct a local point cloud block. For each local point cloud block, its normal vector, curvature, and local depth histogram and other features are calculated, and these features are connected into a vector as a feature descriptor of the local point cloud block. Then, the PointNet++ network is used to aggregate all local features and extract global features. The PointNet++ network can effectively capture the overall shape and structural information of the hand point cloud through multi-layer feature learning, and finally output a vector of a fixed length as the global feature data of the point cloud.

[0056] Step S24: performing spatiotemporal alignment processing on the point cloud global feature data and the video motion feature data according to the timestamp of the original action data to obtain spatiotemporal alignment feature data; In an embodiment of the present invention, the video motion feature data and the point cloud global feature data are time-synchronized using the timestamp information in the action raw data. Since the sampling frequencies of the video data and the point cloud data may be different, interpolation or resampling operations are required to ensure that the two data streams are aligned in time. For example, a data stream with a lower sampling frequency can be increased to the same frequency as a data stream with a higher sampling frequency by linear interpolation. The video motion feature data and the point cloud global feature data after the timestamp alignment are combined to form spatiotemporal alignment feature data.

[0057] Step S25: performing attention weighted processing of video features and point cloud features on the spatiotemporal alignment feature data to obtain attention weighted feature data; performing feature fusion on the attention weighted feature data to obtain a fused feature vector; In an embodiment of the present invention, an attention mechanism is applied to the spatiotemporal alignment feature data respectively. Two independent attention networks are constructed to process video motion features and point cloud global features respectively. Each attention network learns a weight vector for weighting features at different time steps. The weighted video motion features and point cloud global features are concatenated to obtain attention weighted feature data. Finally, the attention weighted feature data is input into a fully connected layer for feature fusion, and a fused feature vector of fixed length is output. The fused feature vector integrates the information of the video and point cloud data, and highlights the features of important time steps through the attention mechanism, thereby improving the expressiveness of the features.

[0058] Preferably, step S22 comprises the following steps: Step S221: extracting continuous key frame key points from the video key point feature data to obtain continuous key frame key point data; extracting depth map data from the video key point feature data to obtain depth map data; Step S222: Obtain camera intrinsic parameters of the RGB camera; perform three-dimensional coordinate conversion on the key point data of the continuous key frames and the depth map data using the camera intrinsic parameters of the RGB camera to obtain three-dimensional coordinate conversion data; Step S223: performing key point displacement calculation on the three-dimensional coordinate transformation data to obtain key point displacement data; performing key point velocity calculation based on the key point displacement data to obtain key point velocity data; performing key point acceleration calculation based on the key point velocity data to obtain key point acceleration data; Step S224: generating motion feature vectors according to the key point displacement data, the key point velocity data and the key point acceleration data to obtain motion feature vector data; Step S225: combining the key point motion feature vectors of each frame image with the motion feature vector data to obtain frame motion feature data; and performing data time sorting on the frame motion feature data to obtain video motion feature data.

[0059] In an embodiment of the present invention, key point information of continuous key frames is extracted from key point feature data of a video. The key frame extraction method can adopt a method based on inter-frame difference, set a difference threshold, and when the difference between two adjacent frames exceeds the threshold, the next frame is regarded as a key frame. The three-dimensional coordinates of all hand key points in each frame image are extracted to form continuous key frame key point data. At the same time, the depth value corresponding to the key point is extracted from the depth map data corresponding to the key frame to form the depth map data. The camera intrinsic parameters of the pre-calibrated RGB camera are obtained, including parameters such as focal length (fx, fy), principal point coordinates (cx, cy), etc. The two-dimensional key point coordinates (u, v) in the continuous key frame key point data and the depth value d in the depth map data are converted into three-dimensional coordinates (x, y, z) using the camera intrinsic parameters. The conversion formula is: x = (u - cx) * d / fx, y = (v - cy) * d / fy, z = d. The converted three-dimensional coordinate data is saved to obtain the three-dimensional coordinate conversion data.

[0060] For each key point, calculate the difference in its 3D coordinates between consecutive key frames to obtain the key point displacement data. The displacement vector is calculated by subtracting the 3D coordinates of the key point in the previous frame from the 3D coordinates of the key point in the next frame. Then, divide the displacement vector by the time interval (i.e., the time difference between two adjacent frames) to obtain the key point velocity data. Finally, divide the difference in velocity vectors between consecutive key frames by the time interval to obtain the key point acceleration data.

[0061] The displacement vector, velocity vector, and acceleration vector of each key point are connected to form a high-dimensional vector. In order to reduce the dimension and improve the computational efficiency, the high-dimensional vector can be subjected to principal component analysis (PCA) dimensionality reduction processing to extract the main components and finally obtain a motion feature vector of fixed length. The motion feature vectors of all key points are saved to obtain the motion feature vector data.

[0062] For each frame of image, the motion feature vectors of all key points in the frame are connected to form the frame motion feature data of the frame. Then, the frame motion feature data of all frames are arranged in time sequence to form the video motion feature data. This allows the time series information to be retained for subsequent action recognition and analysis.

[0063] Preferably, step S3 comprises the following steps: Step S31: arranging the fused feature vectors in time sequence to obtain time series feature sequence data; Step S32: Perform Bi-LSTM encoding on the time series feature sequence data to obtain Bi-LSTM encoded feature data; Step S33: inputting the feature vector of each time step in the Bi-LSTM encoded feature data into a preset fully connected layer, performing low-dimensional feature representation conversion, and obtaining fully connected layer feature data; inputting the feature value of each time step in the fully connected layer feature data into a preset Sigmoid activation function, performing Sigmoid function activation, obtaining Sigmoid activated data, and using the Sigmoid activated data as action segmentation point probability data; Step S34: determining the action segmentation points according to the action segmentation point probability data to obtain action segmentation point data; using the action segmentation point data to divide the temporal feature sequence data into action segments to obtain an action segment set; performing a maximum pooling operation on the action segment set to obtain action segment feature data; Step S35: inputting the action clip feature data into a preset Softmax classifier to perform step classification probability prediction to obtain step classification probability data; Step S36: Generate step identification labels according to the step classification probability data to obtain step identification label data; arrange the step identification label data in time sequence to obtain a step identification label sequence.

[0064] In the embodiment of the present invention, the fused feature vectors obtained in step S2 are sorted according to their corresponding timestamps to form a time series. Each timestamp corresponds to a fused feature vector, and these vectors are arranged in chronological order to form time series feature sequence data. This sequence completely records the change process of hand movement in the time dimension, providing a basis for subsequent action segmentation and step recognition.

[0065] A pre-trained bidirectional long short-term memory network (Bi-LSTM) is used to encode the time series feature sequence data. The Bi-LSTM network contains two sub-networks, the forward LSTM and the backward LSTM, which learn the time dependency in the time series feature sequence data from the forward and backward directions respectively. The time series feature sequence data is input into the Bi-LSTM network, and the network outputs the hidden state vector of each time step. The hidden state vectors of the forward LSTM and the backward LSTM are connected to obtain the Bi-LSTM encoded feature data. The Bi-LSTM network can effectively capture the contextual information in the long time series, thereby better representing the dynamic characteristics of hand movements.

[0066] The feature vector of each time step in the Bi-LSTM encoded feature data is input into a fully connected layer. The fully connected layer maps the high-dimensional feature vector to a low-dimensional space, performs low-dimensional feature representation conversion, and obtains the fully connected layer feature data. Then, the feature value of each time step in the fully connected layer feature data is input into the Sigmoid activation function. The Sigmoid function maps the feature value to between 0 and 1, representing the probability that the time step is an action segmentation point, and obtains the data after Sigmoid activation, that is, the action segmentation point probability data.

[0067] According to the action segmentation point probability data, a probability threshold is set, such as 0.5. When the probability value of a time step is greater than the threshold, the time step is determined as the action segmentation point. All determined action segmentation points constitute the action segmentation point data. The time series feature sequence data is divided into several action segments using the action segmentation point data to form an action segment set. For each action segment, the maximum pooling operation is applied to extract the most representative feature vector in the segment to obtain the action segment feature data.

[0068] The action segment feature data obtained in step S34 is input into a pre-trained Softmax classifier. The Softmax classifier classifies each action segment, outputs the probability distribution of each hand washing step, and obtains the step classification probability data. The Softmax classifier can effectively distinguish different hand washing steps and give the confidence of each step.

[0069] According to the step classification probability data, the category with the largest probability value is selected as the step identification label of the action segment. The step identification labels of all action segments are arranged in chronological order to form a step identification label sequence. This sequence clearly shows the various steps in the hand washing process and provides a basis for subsequent action quality evaluation.

[0070] Preferably, step S4 comprises the following steps: Step S41: Acquire standard action template data of the seven-step hand washing method; extract the start and end time points of each action segment from the step identification label sequence to obtain the start and end time data of the action segment; extract the corresponding time segment feature sequence data from the time series feature sequence data according to the start and end time data of the action segment to obtain the segmented action segment feature sequence data; Step S42: performing time-regular matching processing on the segmented action segment feature sequence data and the standard action template data to obtain time-regular matching distance data; Step S43: Calculate the spatial matching degree according to the time-warping matching distance data to obtain spatial matching degree data; Step S44: Calculate the time matching degree according to the time regularization matching distance data to obtain the time matching degree data; Step S45: extracting action detail features from the segmented action segment feature sequence data to obtain action detail feature data; Step S46: scoring the action detail feature data according to the standard action template data to obtain action detail scoring data; Step S47: Perform weighted summation on the spatial matching data, the temporal matching data and the action detail score data to obtain the action quality score.

[0071] As an example of the present invention, refer to Figure 3 As shown, in this example, step S4 includes: Step S41: Acquire standard action template data of the seven-step hand washing method; extract the start and end time points of each action segment from the step identification label sequence to obtain the start and end time data of the action segment; extract the corresponding time segment feature sequence data from the time series feature sequence data according to the start and end time data of the action segment to obtain the segmented action segment feature sequence data; In an embodiment of the present invention, video data of multiple professionals performing the seven-step hand washing method in a standardized manner is collected in advance, and the time series feature sequence data thereof is extracted according to steps S1 and S2 to form standard action template data. The template data of each step contains the feature sequence of the complete action process of the step. The starting time point and the ending time point of each action segment are extracted from the step identification label sequence to form the start and end time data of the action segment. According to the start and end time data of the action segment, the feature sequence of the corresponding time period is extracted from the time series feature sequence data to obtain the segmented action segment feature sequence data for subsequent comparison with the standard action template.

[0072] Step S42: performing time-regular matching processing on the segmented action segment feature sequence data and the standard action template data to obtain time-regular matching distance data; In an embodiment of the present invention, a dynamic time warping (DTW) algorithm is used to perform time warping matching processing on the segmented action clip feature sequence data and the standard action template data. The DTW algorithm can effectively handle the problem of inconsistent time series lengths by calculating the best matching path between the two time series. The segmented action clip feature sequence and the corresponding standard action template feature sequence are input into the DTW algorithm to calculate the matching distance between the two sequences, i.e., the time warping matching distance data.

[0073] Step S43: Calculate the spatial matching degree according to the time-warping matching distance data to obtain spatial matching degree data; In the embodiment of the present invention, the spatial matching degree is calculated based on the time-regular matching distance data. The spatial matching degree reflects the similarity between the hand-washing action performed by the surgical staff and the standard action in spatial position. The reciprocal of the time-regular matching distance is taken and normalized to obtain the spatial matching degree data. The smaller the distance, the higher the spatial matching degree.

[0074] Step S44: Calculate the time matching degree according to the time regularization matching distance data to obtain the time matching degree data; In the embodiment of the present invention, the time matching degree is calculated based on the time-regularized matching distance data. The time matching degree reflects the similarity between the time taken by the surgical staff to perform each hand-washing step and the standard action time. The ratio of the length of the standard action template sequence to the length of the regularized action fragment sequence is used as a measure of the time matching degree, and normalized processing is performed to obtain the time matching degree data. The closer the ratio is to 1, the higher the time matching degree.

[0075] Step S45: extracting action detail features from the segmented action segment feature sequence data to obtain action detail feature data; In the embodiment of the present invention, the segmented action segment feature sequence data is analyzed to extract action detail features. For example, the motion trajectory length, velocity change, acceleration change and other features of each key point in each action segment can be calculated. These features are combined into a vector as action detail feature data. These detail features can more finely describe the specific execution of the hand washing action.

[0076] Step S46: scoring the action detail feature data according to the standard action template data to obtain action detail scoring data; In an embodiment of the present invention, the key action details of each hand washing step and the corresponding scoring criteria are predefined based on the standard action template data. For example, for the step of "rubbing the back of the finger", the key action details can be defined as the finger bending angle and the range of motion, and the corresponding scoring interval can be set. The extracted action detail feature data is compared with the predefined scoring criteria, the score of each action detail is calculated, and all scores are weighted averaged to obtain the action detail scoring data.

[0077] Step S47: performing weighted summation on the spatial matching data, the temporal matching data and the action detail score data to obtain an action quality score; In the embodiment of the present invention, the spatial matching data, the temporal matching data and the action detail score data are weighted and summed to obtain the final action quality score. The weight coefficient can be adjusted according to the importance of different indicators. For example, a higher weight can be given to the action detail score to emphasize the standardization of the action. The final action quality score comprehensively reflects the accuracy and standardization of the surgical staff's hand washing operation.

[0078] Preferably, step S42 includes the following steps: Step S421: extracting action segment feature sequences from the segmented action segment feature sequence data to obtain action segment feature sequences; Step S422: extracting a standard action template feature sequence from the standard action template data to obtain a standard action template feature sequence; Step S423: normalizing the time series length of the action clip feature sequence and the standard action template feature sequence to obtain a standardized action clip feature sequence and a standardized standard action template feature sequence; Step S424: using the DTW algorithm to perform dynamic time warping path calculation on the standardized action clip feature sequence and the standardized standard action template feature sequence to obtain a DTW matching path; Step S425: Calculate the time-warping matching distance according to the DTW matching path to obtain the time-warping matching distance data.

[0079] In an embodiment of the present invention, a feature sequence corresponding to the hand washing step currently being evaluated is extracted from the segmented action segment feature sequence data. For example, if the "palms rubbing relative to each other" step is currently being evaluated, the feature sequence corresponding to the step is extracted, including all feature vectors between the start and end time points of the step. The extracted feature sequence is defined as an action segment feature sequence for subsequent comparison with the standard action template feature sequence. A standard action template feature sequence corresponding to the current evaluation step is extracted from the standard action template data. For example, if the "palms rubbing relative to each other" step is currently being evaluated, the feature sequence corresponding to the "palms rubbing relative to each other" step in the standard action template data is extracted. The sequence represents the standard execution method of the step and serves as a benchmark for evaluation. The action segment feature sequence and the standard action template feature sequence are normalized for time series length using a linear interpolation method. A standard length is set, such as 100. The action segment feature sequence and the standard action template feature sequence are respectively expanded or compressed to the standard length by linear interpolation. This ensures that the lengths of the two sequences are consistent, which facilitates subsequent DTW algorithm calculations. The Dynamic Time Warping (DTW) algorithm is used to calculate the best matching path between the standardized action clip feature sequence and the standardized standard action template feature sequence. The DTW algorithm constructs a cost matrix, and the matrix elements represent the Euclidean distance between the feature vectors of the corresponding time steps in the two sequences. Through the dynamic programming algorithm, a path from the upper left corner to the lower right corner of the matrix is ​​found so that the cumulative cost on the path is minimized. This path is the DTW matching path, which reflects the temporal correspondence between the two sequences. The time warping matching distance is calculated based on the DTW matching path. The Euclidean distances between the feature vectors of all corresponding time steps on the DTW matching path are accumulated to obtain the final time warping matching distance data. The smaller the distance value, the higher the similarity between the action clip and the standard action template.

[0080] Preferably, step S5 comprises the following steps: Step S51: using a preset quality score threshold to perform step quality judgment on the action quality score to obtain step quality judgment data; Step S52: performing real-time feedback on step quality according to step quality judgment data to obtain real-time feedback information data; Step S53: Perform step completion statistics according to the step quality judgment data to obtain step completion data; Step S54: performing compliance assessment of the overall hand washing process according to the step completion status data to obtain overall compliance assessment data; Step S55: Generate a compliance report based on the step quality judgment data, the step completion status data and the overall compliance assessment data to obtain a compliance report.

[0081] In an embodiment of the present invention, a quality score threshold value of each hand washing step is preset, for example, 0.8. The action quality score calculated in step S4 is compared with the preset quality score threshold value. If the action quality score is greater than or equal to the threshold value, the quality of the step is judged to be qualified; if the action quality score is less than the threshold value, the quality of the step is judged to be unqualified. The quality judgment result of each step is saved to obtain step quality judgment data.

[0082] Based on the step quality judgment data, the quality assessment results of each step are fed back to the surgeon in real time. For example, a prompt message such as "Step X completed, quality passed / failed" can be displayed on the display, or feedback can be given through voice broadcast. For steps with unqualified quality, specific improvement suggestions can also be given, such as "Please rub the fingers more".

[0083] Statistics on the completion of each hand washing step. For example, record whether each step is completed, the number of completions, the number of qualified times, etc. Save this information to obtain step completion data. This provides data support for subsequent overall compliance assessment.

[0084] According to the step completion data, the compliance of the overall hand washing process is evaluated. For example, according to the requirements of the seven-step hand washing method, it can be determined whether all steps have been completed and whether each step has met the quality requirements. If all steps are completed and the quality is qualified, the overall hand washing process is judged to be compliant; otherwise, the overall hand washing process is judged to be non-compliant. The evaluation results are saved to obtain the overall compliance evaluation data.

[0085] Compliance reports are automatically generated based on step quality judgment data, step completion data, and overall compliance assessment data. The report includes the quality score, completion status, improvement suggestions for each step, and the compliance assessment results of the overall hand washing process. The report can be presented in the form of text or charts and saved to a database or exported as a file for subsequent review and analysis.

[0086] Preferably, the present invention further provides a surgical personnel video behavior recognition system based on machine learning, which is used to execute the surgical personnel video behavior recognition method based on machine learning as described above. The surgical personnel video behavior recognition system based on machine learning includes: The multimodal data capture module is used to obtain the video frame sequence and point cloud frame sequence of the surgical staff's hand washing area through the RGB camera and the laser ranging device to obtain the video frame sequence and the point cloud frame sequence; perform hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; generate action raw data according to the video frame sequence and the hand point cloud data to obtain the action raw data; The spatiotemporal feature fusion module is used to extract the position and posture information of the key points of the hand from the original action data to obtain the video key point feature data; extract the video motion feature from the video key point feature data to obtain the video motion feature data; extract the point cloud global feature from the original action data to obtain the point cloud global feature data; perform weighted feature fusion on the point cloud global feature data and the video motion feature data to obtain the fused feature vector; The action segmentation and step recognition module is used to perform time sequence arrangement processing on the fused feature vector to obtain time series feature sequence data; perform action segmentation point probability prediction on the time series feature sequence data to obtain action segmentation point probability data; perform action segmentation based on the action segmentation point probability data, and perform step recognition to obtain a step recognition label sequence; The action quality assessment module is used to obtain the standard action template data of the seven-step hand washing method; perform time-regular matching processing according to the standard action template data of the seven-step hand washing method and the step identification label sequence to obtain time-regular matching distance data; perform action quality assessment according to the time-regular matching distance data to obtain an action quality score; The action compliance verification module is used to verify the compliance of the action quality score to obtain the overall compliance assessment data; the compliance report is generated based on the overall compliance assessment data to obtain the compliance report to realize the hand washing and disinfection monitoring operation of the surgical staff.

[0087] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0088] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A method for identifying surgical personnel's video behavior based on machine learning, characterized in that: The following steps are involved: Step S1: acquiring a video frame sequence and a point cloud frame sequence of the surgical staff's hand washing area through an RGB camera and a laser ranging device to obtain a video frame sequence and a point cloud frame sequence; performing hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; generating action raw data according to the video frame sequence and the hand point cloud data to obtain action raw data; Step S2: extracting the position and posture information of the key points of the hand from the original action data to obtain the key point feature data of the video; Perform video motion feature extraction on the video key point feature data to obtain video motion feature data; perform point cloud global feature extraction on the action original data to obtain point cloud global feature data; perform weighted feature fusion on the point cloud global feature data and the video motion feature data to obtain a fused feature vector; Step S3: arranging the fused feature vectors in time order to obtain time series feature sequence data; predicting the probability of action segmentation points on the time series feature sequence data to obtain action segmentation point probability data; performing action segmentation according to the action segmentation point probability data, and performing step recognition to obtain a step recognition label sequence; Step S4: obtaining standard action template data of the seven-step hand washing method; performing time regularization matching processing according to the standard action template data of the seven-step hand washing method and the step identification label sequence to obtain time regularization matching distance data; performing action quality evaluation according to the time regularization matching distance data to obtain an action quality score; Step S5: Perform compliance verification on the action quality score to obtain overall compliance assessment data; generate a compliance report based on the overall compliance assessment data to obtain a compliance report to implement the hand washing and disinfection monitoring operation of the surgical staff.

2. The method for identifying surgical personnel's video behavior based on machine learning according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: Use the RGB camera to detect the screen motion of the surgical staff's hand washing area. When continuous motion changes are detected, collect video stream data to obtain a video frame sequence; synchronously collect point cloud data according to the laser ranging device and the timestamp of the video frame sequence to obtain a point cloud frame sequence; Step S12: performing point cloud denoising processing on the point cloud frame sequence to obtain point cloud denoised data; Step S13: performing hand point cloud segmentation on the point cloud denoising data to obtain hand point cloud data; Step S14: extracting key frames from the video frame sequence to obtain video key frame data; Step S15: performing multimodal data alignment on the hand point cloud data and the video key frame data to obtain multimodal data alignment data; generating action raw data according to the multimodal data alignment data to obtain action raw data.

3. The method for identifying surgical personnel's video behavior based on machine learning according to claim 2 is characterized in that: Step S13 includes the following steps: Step S131: performing preliminary detection of the hand area using the point cloud denoising data and a preset depth range threshold to obtain preliminary hand point cloud data; Step S132: using the pre-trained hand detection model to perform hand contour detection on the video frame sequence to obtain hand contour data; Step S133: Mapping the hand contour data to the preliminary hand point cloud data, removing non-hand point clouds, and obtaining RGB auxiliary hand point cloud data; Step S134: performing normal estimation on the RGB-assisted hand point cloud data to obtain normal estimation data; Step S135: separating the hand point cloud from the background objects on the RGB-assisted hand point cloud data according to the normal estimation data to obtain refined hand point cloud data; Step S136: performing hand point cloud morphology correction on the refined hand point cloud data to obtain hand point cloud segmentation data.

4. The method for identifying surgical personnel's video behavior based on machine learning according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: extracting the position and posture information of the key points of the hand from the original action data using the pre-trained human posture estimation model to obtain the key point feature data of the video; Step S22: extracting video motion features from the video key point feature data to obtain video motion feature data; Step S23: extracting local feature of point cloud from the original action data to obtain local feature data of point cloud; extracting global feature of point cloud from the local feature data of point cloud to obtain global feature data of point cloud; Step S24: performing spatiotemporal alignment processing on the point cloud global feature data and the video motion feature data according to the timestamp of the original action data to obtain spatiotemporal alignment feature data; Step S25: Perform attention weighted processing of video features and point cloud features on the spatiotemporal alignment feature data to obtain attention weighted feature data; perform feature fusion on the attention weighted feature data to obtain a fused feature vector.

5. The method for identifying surgical personnel's video behavior based on machine learning according to claim 4 is characterized in that: Step S22 includes the following steps: Step S221: extracting continuous key frame key points from the video key point feature data to obtain continuous key frame key point data; extracting depth map data from the video key point feature data to obtain depth map data; Step S222: Obtain camera intrinsic parameters of the RGB camera; perform three-dimensional coordinate conversion on the key point data of the continuous key frames and the depth map data using the camera intrinsic parameters of the RGB camera to obtain three-dimensional coordinate conversion data; Step S223: performing key point displacement calculation on the three-dimensional coordinate transformation data to obtain key point displacement data; performing key point velocity calculation based on the key point displacement data to obtain key point velocity data; performing key point acceleration calculation based on the key point velocity data to obtain key point acceleration data; Step S224: generating motion feature vectors according to the key point displacement data, the key point velocity data and the key point acceleration data to obtain motion feature vector data; Step S225: combining the key point motion feature vectors of each frame image with the motion feature vector data to obtain frame motion feature data; and performing data time sorting on the frame motion feature data to obtain video motion feature data.

6. The method for identifying surgical personnel's video behavior based on machine learning according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: arranging the fused feature vectors in time sequence to obtain time series feature sequence data; Step S32: Perform Bi-LSTM encoding on the time series feature sequence data to obtain Bi-LSTM encoded feature data; Step S33: inputting the feature vector of each time step in the Bi-LSTM encoded feature data into a preset fully connected layer, performing low-dimensional feature representation conversion, and obtaining fully connected layer feature data; inputting the feature value of each time step in the fully connected layer feature data into a preset Sigmoid activation function, performing Sigmoid function activation, obtaining Sigmoid activated data, and using the Sigmoid activated data as action segmentation point probability data; Step S34: determining the action segmentation points according to the action segmentation point probability data to obtain action segmentation point data; using the action segmentation point data to divide the temporal feature sequence data into action segments to obtain an action segment set; performing a maximum pooling operation on the action segment set to obtain action segment feature data; Step S35: inputting the action clip feature data into a preset Softmax classifier to perform step classification probability prediction to obtain step classification probability data; Step S36: Generate step identification labels according to the step classification probability data to obtain step identification label data; arrange the step identification label data in time sequence to obtain a step identification label sequence.

7. The method for identifying surgical personnel's video behavior based on machine learning according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: Acquire standard action template data of the seven-step hand washing method; extract the start and end time points of each action segment from the step identification label sequence to obtain the start and end time data of the action segment; extract the corresponding time segment feature sequence data from the time series feature sequence data according to the start and end time data of the action segment to obtain the segmented action segment feature sequence data; Step S42: performing time-regular matching processing on the segmented action segment feature sequence data and the standard action template data to obtain time-regular matching distance data; Step S43: Calculate the spatial matching degree according to the time-warping matching distance data to obtain spatial matching degree data; Step S44: Calculate the time matching degree according to the time regularization matching distance data to obtain the time matching degree data; Step S45: extracting action detail features from the segmented action segment feature sequence data to obtain action detail feature data; Step S46: scoring the action detail feature data according to the standard action template data to obtain action detail scoring data; Step S47: Perform weighted summation on the spatial matching data, the temporal matching data and the action detail score data to obtain the action quality score.

8. The method for identifying surgical personnel's video behavior based on machine learning according to claim 7, characterized in that: Step S42 includes the following steps: Step S421: extracting action segment feature sequences from the segmented action segment feature sequence data to obtain action segment feature sequences; Step S422: extracting a standard action template feature sequence from the standard action template data to obtain a standard action template feature sequence; Step S423: normalizing the time series length of the action clip feature sequence and the standard action template feature sequence to obtain a standardized action clip feature sequence and a standardized standard action template feature sequence; Step S424: using the DTW algorithm to perform dynamic time warping path calculation on the standardized action clip feature sequence and the standardized standard action template feature sequence to obtain a DTW matching path; Step S425: Calculate the time-warping matching distance according to the DTW matching path to obtain the time-warping matching distance data.

9. The method for identifying surgical personnel's video behavior based on machine learning according to claim 1, characterized in that: Step S5 includes the following steps: Step S51: using a preset quality score threshold to perform step quality judgment on the action quality score to obtain step quality judgment data; Step S52: performing real-time feedback on step quality according to step quality judgment data to obtain real-time feedback information data; Step S53: Perform step completion statistics according to the step quality judgment data to obtain step completion data; Step S54: performing compliance assessment of the overall hand washing process according to the step completion status data to obtain overall compliance assessment data; Step S55: Generate a compliance report based on the step quality judgment data, the step completion status data and the overall compliance assessment data to obtain a compliance report.

10. A surgical personnel video behavior recognition system based on machine learning, characterized in that: Used to execute the method for recognizing surgical personnel video behavior based on machine learning as claimed in claim 1, the surgical personnel video behavior recognition system based on machine learning comprises: The multimodal data capture module is used to obtain the video frame sequence and point cloud frame sequence of the surgical staff's hand washing area through the RGB camera and the laser ranging device to obtain the video frame sequence and the point cloud frame sequence; perform hand point cloud segmentation on the point cloud frame sequence to obtain hand point cloud data; generate action raw data according to the video frame sequence and the hand point cloud data to obtain the action raw data; The spatiotemporal feature fusion module is used to extract the position and posture information of the key points of the hand from the original action data to obtain the video key point feature data; extract the video motion feature from the video key point feature data to obtain the video motion feature data; extract the point cloud global feature from the original action data to obtain the point cloud global feature data; perform weighted feature fusion on the point cloud global feature data and the video motion feature data to obtain the fused feature vector; The action segmentation and step recognition module is used to perform time sequence arrangement processing on the fused feature vector to obtain time series feature sequence data; perform action segmentation point probability prediction on the time series feature sequence data to obtain action segmentation point probability data; perform action segmentation based on the action segmentation point probability data, and perform step recognition to obtain a step recognition label sequence; The action quality assessment module is used to obtain the standard action template data of the seven-step hand washing method; perform time-regular matching processing according to the standard action template data of the seven-step hand washing method and the step identification label sequence to obtain time-regular matching distance data; perform action quality assessment according to the time-regular matching distance data to obtain an action quality score; The action compliance verification module is used to verify the compliance of the action quality score to obtain the overall compliance assessment data; the compliance report is generated based on the overall compliance assessment data to obtain the compliance report to realize the hand washing and disinfection monitoring operation of the surgical staff.

Images