Medical behavior quality evaluation method and system based on time sequence action segmentation

Abstract

The invention relates to a medical behavior quality evaluation method and system based on time sequence action segmentation, and the method comprises the following steps: obtaining all operation behavior videos in the whole medical process, which are collected by shooting equipment with multiple visual angles, and analyzing the dimensions and the frame number of the videos; video data are input into a time sequence segmentation model for time sequence action segmentation and recognition, the time sequence segmentation model introduces a diffusion model and comprises a feature extraction module, an encoder and a decoder, the feature extraction module extracts I3D features corresponding to a video, the encoder encodes the universal I3D features into features more conforming to a task, and the decoder decodes the I3D features into features more conforming to the task. The decoder completes an iteration process of a diffusion model, and iteratively generates a prediction action label sequence from random noise; and performing score statistics according to the action label sequence given by the time sequence segmentation model and the standard specification of the corresponding medical behavior. Compared with the prior art, the method has the advantages of high time sequence behavior segmentation accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a medical behavior quality assessment method and system based on temporal action segmentation. Background Art

[0002] As the aging population in my country continues to accelerate, the demand for medical resources and medical services is also increasing. A qualified medical staff must not only have solid theoretical knowledge, but also master medical operation skills that meet technical specifications. However, the current evaluation of medical students' clinical medical skills still requires manual assessment by senior doctors, which takes up a lot of doctors' time and energy. Existing related research is based on behavior recognition technology in general scenarios, and has made certain migration and adaptation to medical scenarios, realizing the error recognition and quality detection of some medical operations. However, most of the current technologies only conduct quality assessment on a single medical operation action, lack an overall analysis of the medical process, and cannot test complete surgical or nursing operations. Therefore, it is crucial to build an intelligent medical behavior assessment and analysis system based on long video temporal action segmentation to improve the efficiency of medical staff training and assessment and supplement the number of high-quality doctors.

[0003] At present, research related to medical behavior quality assessment can be divided into three aspects: human behavior recognition, behavior quality assessment, and temporal behavior segmentation. The first two only perform category prediction or quality assessment on videos containing a single action, while temporal behavior segmentation requires action segmentation of long videos containing multiple actions. It is an important method in the medical behavior assessment and analysis system, laying the foundation for subsequent analysis and evaluation.

[0004] In the task of temporal behavior segmentation, the model takes an untrimmed long video as input, outputs a sequence of action labels corresponding to each frame of the video, and finally realizes the analysis of behavior completeness and accuracy. Currently, the commonly used temporal behavior segmentation algorithms can be divided into methods based on recurrent neural networks (RNNs), methods based on temporal convolutional networks (TCNs), methods based on graph convolutional neural networks (GNNs), and methods based on Transformer models based on the network structure. In recent years, the development of multi-stage models has made significant progress in the task of temporal behavior segmentation. Multi-stage models use multiple stages of iteration, where the first stage produces initial predictions and the subsequent stages adjust the predictions based on the previous stage. At present, researchers have tried to implement multi-stage models with various architectures, such as MS-TCN relying on extended temporal convolutional layers and ASFormer with attention mechanism. The success of multi-stage models is largely attributed to the underlying iterative refinement paradigm, which correctly captures the temporal dynamics of actions and significantly increases the accuracy of video segmentation.

[0005] First, the current behavior quality assessment method can only evaluate a single action and is not suitable for medical behavior, which involves multiple complex operations. Second, in terms of temporal video segmentation, the existing multi-stage methods based on recurrent neural networks and temporal convolutional networks work well on datasets with a small number of action classes and video clips, but for large datasets with diverse action categories, these methods will have over-segmentation errors, reducing the accuracy of behavior recognition. Summary of the invention

[0006] The purpose of the present invention is to provide a medical behavior quality assessment method and system based on temporal action segmentation, to perform quality assessment on complex medical behavior scenarios with multiple actions, to improve the segmentation accuracy of action recognition, and to achieve accurate and effective action assessment.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A method for evaluating the quality of medical behavior based on temporal action segmentation comprises the following steps:

[0009] Obtain all operation behavior videos of the entire medical process captured by shooting equipment from multiple perspectives, and analyze the dimensions and frame number of the video;

[0010] The video data is input into a temporal segmentation model for temporal action segmentation and recognition. The temporal segmentation model introduces a diffusion model, which includes a feature extraction module, an encoder, and a decoder. The feature extraction module extracts I3D features corresponding to the video, the encoder encodes the general I3D features into features that are more suitable for the task, and the decoder completes the iterative process of the diffusion model and iteratively generates a predicted action label sequence from random noise;

[0011] The score statistics are performed based on the action label sequence given by the time series segmentation model and the standard specifications of the corresponding medical behavior. For each label in the predicted action label sequence, it is compared with the standard action sequence. If the label is inconsistent with the standard action, the corresponding deduction score is added to the deduction sequence according to the scoring rules. Each label in the predicted action label sequence is traversed, and the wrong action number and the corresponding deduction sequence are output. The final assessment score is counted to complete the medical behavior quality assessment.

[0012] The temporal segmentation model is trained using a series of training set data annotated with real action labels. For each video in the training set and its corresponding real reference value, a forward process is first performed: random noise is added to the real reference value, and then a backward process is performed: the decoder iteratively generates a probability sequence by receiving the reference value with added random noise, the video features encoded by the encoder, and the number of iterations, that is, the probability that each frame image represents each action, and performs sequence conversion based on the probability to obtain a predicted action sequence label; the goal of the temporal segmentation model in training is to minimize the difference between the value predicted by the decoder and the real reference value, even if the loss function is reduced during the iteration.

[0013] The forward process is specifically as follows:

[0014] Assume that the original true sequence distribution is x0~q(x0), the number of iterations is S, and the noise sequence is β1,β2,…,β S , β is a hyperparameter with a value range of 0 to 1. The single-step noise addition operation on the true reference value is expressed as:

[0015]

[0016] Among them, x s is the data after s times of noise addition, I is the unit matrix, and N(·) represents the normal distribution;

[0017] Let α s =1-β s , According to the additive property of normal distribution, x0 is added with noise in one step to obtain x s The process of writing:

[0018]

[0019] Given the original data x0, the data x after adding S noises is S is represented as:

[0020]

[0021] Among them, ∈~N(0,1), obeys the standard normal distribution.

[0022] The backward process samples a set of data x from a random noise distribution S ~N(0,1), and gradually restore it to the original data x0, defining the recovery function as p θ (x s ), the single-step denoising process is expressed as:

[0023]

[0024] Among them, θ is the neural network parameter, and the mean μθ (x s ,s) is estimated by neural network, the standard deviation σ s β S control;

[0025] Repeating the above single-step denoising process can restore the original data x0 and obtain the complete denoising path. Let the complete denoising path be {x0, x1, …, x S}, then the joint probability distribution of the denoising path is:

[0026]

[0027] During the training of the time series segmentation model, the decoder g is fed with the sequence x after adding noise. S As the initial input, it also receives the video features E encoded by the encoder and the set number of iterations S, and generates a probability sequence to represent the probability of each action represented by the denoised video frame:

[0028] P s =g(x S ,S,E)

[0029] Among them, P S Represents the input x S The probability that the denoised video frames represent each action;

[0030] The decoder performs sequence conversion based on the predicted action probability and iteratively generates a sequence of predicted action labels. The update rule for each iteration is as follows:

[0031]

[0032] Among them, P s Represents the input x′ s The action probability, that is, P s =g(x′ s ,S,E),x′ s Represents the action label sequence of the current iteration. In the first iteration, x′ s Take x S ; Updated x′ s-1 It is sent to the decoder again to predict P s-1 , thus proceeding to the next round of iteration;

[0033] After S rounds of iterations, the final output model predicts the action label sequence x′0.

[0034] The total loss function of the temporal segmentation model is used to measure the difference between the predicted action label sequence x′0 and the true sequence x0. It consists of cross entropy loss, temporal smoothing loss and boundary alignment loss, which are used to analyze frame-level differences, local label differences and behavior switching position prediction differences respectively.

[0035] Among them, the cross entropy loss The prediction difference for each frame is measured and expressed as:

[0036]

[0037] Among them, L is the total number of frames, C is the total number of categories, l and c represent the indexes of frames and categories respectively, x0 represents the real sequence, P S represents the predicted action probability output by the decoder;

[0038] Time series smoothing loss The log-likelihood mean square error between adjacent video frames is calculated and expressed as:

[0039]

[0040] Boundary alignment loss is used to measure the model's prediction deviation of the behavior switching position. For each benchmark original sequence x0, generate its corresponding boundary sequence B∈{0,1} N-1 , where B is 1, indicating that a behavior label switch has occurred:

[0041]

[0042] Use Gaussian filter to smooth the boundary sequence B and get the smoothed sequence

[0043] Use a binary loss function to measure x′0 and The boundary prediction error between them is used to obtain the boundary alignment loss

[0044]

[0045] Then the total loss function of the time series segmentation model is expressed as:

[0046]

[0047] The multi-angle shooting equipment includes a multi-angle camera set up at the medical site and a head-mounted camera worn on the operator's head.

[0048] The feature extraction module uses an I3D model to extract features. The I3D model is implemented by extending a 2D convolutional neural network to 3D, that is, expanding a 2D convolution kernel to a time dimension to form a 3D convolution kernel to simultaneously process the spatial and temporal information of the video. The I3D model architecture is divided into three parts: an input layer, an Inception layer, and a pooling layer. After the input video passes through the three parts of the I3D model, an I3D feature F is generated.

[0049] The encoder uses the ASFormer model to further encode the I3D features. The ASFormer model uses the Transformer architecture to process video data. By introducing the spatiotemporal attention mechanism to more effectively capture the complex spatiotemporal relationship in the video, the encoder-encoded feature E is obtained.

[0050] A medical behavior quality assessment system based on temporal action segmentation is used to implement the method described above, the system comprising:

[0051] The video receiving unit is used to obtain the video of all the operation behaviors in the entire medical process collected by the shooting equipment with multiple viewing angles, and analyze the dimension and frame number of the video;

[0052] A temporal segmentation unit is used to input video data into a temporal segmentation model for temporal action segmentation and recognition. The temporal segmentation model introduces a diffusion model and includes a feature extraction module, an encoder, and a decoder. The feature extraction module extracts I3D features corresponding to the video, the encoder encodes the general I3D features into features that are more suitable for the task, and the decoder completes the iterative process of the diffusion model and iteratively generates a predicted action label sequence from random noise;

[0053] The scoring statistics unit is used to perform scoring statistics based on the action label sequence given by the time series segmentation model and the standard specifications of the corresponding medical behavior. For each label in the predicted action label sequence, it is compared with the standard action sequence. If the label is inconsistent with the standard action, the corresponding deduction score is added to the deduction sequence according to the scoring rules. Each label in the predicted action label sequence is traversed, and the wrong action number and the corresponding deduction sequence are output. The final assessment score is counted to complete the medical behavior quality assessment.

[0054] The system also includes a result feedback unit, which is responsible for displaying and visualizing the evaluation results of the system on the entire medical behavior process. After receiving the recognition results and corresponding scores transmitted by the score statistics unit, the result feedback unit organizes the evaluation results of the evaluation algorithm on each stage of the action execution process into natural language, and gives the evaluation score, reason for deduction and correct specifications for each action, as well as visualizes the final assessment score of the entire medical behavior process and provides action improvement suggestions based on the stage scores.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) The present invention can make full use of the standardization and procedural nature of medical behavior to realize intelligent identification and analysis of medical behavior, and can complete a one-time analysis of a long video of multiple operations without the need for complicated video processing and segmentation.

[0057] (2) Compared with manual assessment methods, the present invention facilitates batch and large-scale assessment of medical behavior, saving physicians' teaching and assessment time.

[0058] (3) Compared with the current multi-stage model, the present invention avoids over-segmentation errors by introducing a diffusion model step-by-step decoding mechanism, and achieves better recognition results on the temporal behavior segmentation benchmark.

[0059] (4) The present invention integrates the subsequent evaluation analysis and visualization display of behavior prediction, which makes it easier for medical staff to understand the final evaluation results of this medical behavior and the improvement methods of each step, thereby improving the evaluation and learning efficiency of medical operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the method of the present invention;

[0061] Figure 2 It is a schematic diagram of the training process of the time series segmentation model of the present invention;

[0062] Figure 3 It is a flow chart of action sequence prediction of the time series segmentation model of the present invention;

[0063] Figure 4 It is a scoring behavior statistics and feedback flow chart of the present invention;

[0064] Figure 5 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0065] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0066] Example 1

[0067] This embodiment provides a method for evaluating the quality of medical behavior based on temporal action segmentation. Figure 1 As shown, the following steps are included:

[0068] S1, obtain all operation behavior videos of the entire medical process captured by shooting devices with multiple perspectives, and analyze the dimension and frame number of the video.

[0069] This embodiment collects all operation behavior videos during the entire medical process by setting up cameras with multiple viewing angles at the medical site. This can not only meet the richness of video data, but also increase the amount of video training set data. At the same time, the present invention uses a head-mounted camera to record the operator's first-person perspective behavior, which can capture the details of the medical behavior and is more targeted. In the prediction stage, the entire long video is read and the dimension and frame number of the video are analyzed.

[0070] This embodiment is explained in detail using cardiopulmonary resuscitation operation as an example. The process of cardiopulmonary resuscitation includes chest compression, artificial respiration, AED use and other parts, and each part includes multiple operation steps. In order to ensure that the model can identify the error causes of incorrect actions, the present invention first confirms the error-prone steps in each part and summarizes the types of errors that often occur in training, such as excessive chest compressions and artificial respiration blowing too fast. Based on the decomposed correct steps and the summarized incorrect steps, the present invention finally constructs a label space containing correct operations and incorrect operations.

[0071] In the data collection stage, the present invention uses cameras with two viewing angles to collect a total of about 2,000 pieces of operation data, which include operation actions in all label spaces and ensure the balance of label categories. In the training stage, the present invention matches the operation data and labels one by one and passes them to the time series segmentation model.

[0072] S2, input the video data into the temporal segmentation model for temporal action segmentation and recognition.

[0073] This embodiment introduces a diffusion model in the field of image generation into the temporal segmentation model to extract the action behavior label of each frame image in the video through the trained decoder. Unlike the traditional multi-stage model that iteratively optimizes from the initial classification results, this framework iteratively generates action labels from randomly generated noise, so there will be no over-segmentation errors, thereby effectively improving the accuracy.

[0074] The temporal segmentation model consists of three parts: feature extraction module, encoder and decoder. The feature extraction module extracts the I3D features corresponding to the video for subsequent analysis. The encoder encodes the general I3D features into features that are more suitable for the task. The decoder completes the iterative process of the diffusion model and iteratively generates a sequence of predicted action labels from random noise.

[0075] In order to obtain a decoder capable of generating a prediction sequence from noise, the present invention uses a series of training set data annotated with real action labels to train it. For each video in the training set and its corresponding real reference value, the forward process is first performed: random noise is added to the real reference value, and then the backward process is performed: the decoder iteratively generates a probability sequence by receiving the reference value with added random noise, the video features encoded by the encoder, and the number of iterations, that is, the probability that each frame image represents each action, and performs sequence conversion according to the probability to obtain the predicted action sequence label. The goal of the time series segmentation model in training is to minimize the difference between the value predicted by the decoder and the real reference value, even if the loss function is reduced during the iteration.

[0076] In order to measure the difference between the predicted sequence and the reference sequence, the present invention uses three loss functions, namely cross entropy loss, temporal smoothing loss and boundary alignment loss, to measure the difference between the two sequences during the training process. These three loss functions can analyze the frame level difference, local label difference and behavior switching position prediction difference respectively. The final loss function is the sum of the above three loss functions, which not only ensures the accuracy of action recognition, but also reduces the label switching frequency in the prediction results, ensuring the accuracy of the action switching position. In the prediction stage, the video to be analyzed is sequentially input into the feature extraction module and the encoder to obtain the final coding features. The trained decoder uses the coding features to iteratively generate the final predicted action label sequence from the generated random noise.

[0077] In this step, the present invention first extracts and encodes the features of the video, and then trains the model using the idea of ​​the diffusion model. In the prediction stage, the features formed by the video after two encodings are sent to the trained decoder, and the decoder generates correct predictions from the noise based on the features.

[0078] S21, I3D feature extraction

[0079] I3D (Inflated 3D ConvNet) is a deep learning model for video action recognition, which is implemented by extending the 2D convolutional neural network to 3D, that is, expanding the 2D convolution kernel to the time dimension to form a 3D convolution kernel, so that the spatial and temporal information of the video can be processed simultaneously. The feature extraction module of this embodiment uses the I3D model for feature extraction. The model architecture of I3D for feature extraction can be divided into three parts: input layer, Inception layer, and pooling layer. After the input video passes through the three parts of the I3D model, the I3D feature F is generated.

[0080] S22, ASFormer feature encoding

[0081] For long time series videos, relying solely on the I3D model may not be able to fully capture the long-term dependencies in the video. The use of additional encoders allows the model to integrate information and model dependencies over a longer time frame. Therefore, the encoder of the present invention uses the ASFormer model to further encode the features generated by I3D. The ASFormer (Action Segment Transformer) model uses the Transformer architecture to process video data, and introduces a spatiotemporal attention mechanism to more effectively capture the complex spatiotemporal relationships in the video, thereby improving the accuracy of action segmentation. The feature F obtained in the previous stage can be encoded by the encoder to generate the feature E encoded by the encoder.

[0082] S23, model training

[0083] Figure 2 The training process of the time series segmentation model is described. In the process of model training, the present invention refers to the idea of ​​the forward process and the backward process of the diffusion model. Noise is added to the reference value as input, and the predicted value is obtained through the backward process of the receiving decoder, and then the difference between the predicted value and the reference value is compared. The training goal of the model is to minimize the difference between the value predicted by the decoder and the actual reference value.

[0084] (1) Forward process

[0085] The forward process is a process of gradually adding Gaussian noise to the original data.

[0086] Assume that the original true sequence distribution is x0~q(x0), the number of iterations is S, and the noise sequence is β1,β2,…,β S , β is a hyperparameter with a value range of 0 to 1. The single-step noise addition operation on the true reference value is expressed as:

[0087]

[0088] Among them, x sis the data after s times of noise addition, I is the unit matrix, and N(·) represents the normal distribution;

[0089] Let α s =1-β s , According to the additive property of normal distribution, x0 is added with noise in one step to obtain x s The process of writing:

[0090]

[0091] Given the original data x0, the data x after adding S noises is S is represented as:

[0092]

[0093] Among them, ∈~N(0,1), obeys the standard normal distribution.

[0094] (2) Backward process

[0095] The backward process samples a set of data x from a random noise distribution S ~N(0,1), and gradually restore it to the original data x0, defining the recovery function as p θ (x s ), the single-step denoising process is expressed as:

[0096]

[0097] Among them, θ is the neural network parameter, and the mean μ θ (x s ,s) is estimated by neural network, the standard deviation σ s β S control.

[0098] Repeating the above single-step denoising process can restore the original data x0 and obtain the complete denoising path. Let the complete denoising path be {x0, x1, …, x S}, then the joint probability distribution of the denoising path is:

[0099]

[0100] During the training of the time series segmentation model, the decoder g is fed with the sequence x after adding noise. S As the initial input, it also receives the video features E encoded by the encoder and the set number of iterations S, and generates a probability sequence to represent the probability of each action represented by the denoised video frame:

[0101] P S =g(x S ,S,E) (6)

[0102] Among them, P S Represents the input x S The probability that the denoised video frames represent each action.

[0103] The decoder performs sequence conversion based on the predicted action probability and iteratively generates a sequence of predicted action labels. The update rule for each iteration is as follows:

[0104]

[0105] Among them, P s Represents the input x′ s The action probability is:

[0106] P s =g(x′ s ,S,E) (8)

[0107] x′ s Represents the action label sequence of the current iteration. In the first iteration, x′ s Take x S ; Updated x′ s-1 It is sent to the decoder again to predict P s-1 , thus proceeding to the next round of iteration;

[0108] After S rounds of iterations, the final output model predicts the action label sequence x′0.

[0109] (3) Loss Function

[0110] The total loss function of the temporal segmentation model is used to measure the difference between the predicted action label sequence x′0 and the true sequence x0. It consists of cross entropy loss, temporal smoothing loss and boundary alignment loss, which are used to analyze frame-level differences, local label differences and behavior switching position prediction differences, respectively.

[0111] Cross Entropy Loss The prediction difference for each frame is measured and expressed as:

[0112]

[0113] Among them, L is the total number of frames, C is the total number of categories, l and c represent the indexes of frames and categories respectively, x0 represents the real sequence, P S represents the predicted action probability output by the decoder.

[0114] Time series smoothing loss The log-likelihood mean square error between adjacent video frames is calculated and expressed as:

[0115]

[0116] Accurate boundary detection is the key to action segmentation. The boundary alignment loss is used to measure the model’s prediction deviation of the behavior switching position. For each benchmark original sequence x0, generate its corresponding boundary sequence B∈{0,1} N-1 , where B is 1, indicating that a behavior label switch has occurred:

[0117]

[0118] Use Gaussian filter to smooth the boundary sequence B and get the smoothed sequence

[0119] Use a binary loss function to measure x′0 and The boundary prediction error between them is used to obtain the boundary alignment loss

[0120]

[0121] Then the total loss function of the time series segmentation model is expressed as:

[0122]

[0123] In the training process of this example, the original data x0 is the benchmark value of the label corresponding to the cardiopulmonary resuscitation dataset. The forward process gradually adds noise to the benchmark label sequence through the noise addition module to obtain x S The reverse process uses the decoder that obtains the video coding features to iteratively infer the predicted value x′0 from the random noise, and obtains the loss value through the loss calculation module, so as to continuously reduce the loss function value to complete the update of the model parameters.

[0124] (4) Prediction process

[0125] Figure 3 The process of video prediction is shown. The trained denoising decoder can process sequences composed of pure random noise. Therefore, during prediction inference, the denoising decoder receives the pure noise sequence and generates the behavior probability corresponding to the sequence according to formula (8). The sequence conversion module is called to perform sequence conversion according to formula (7) by receiving the prediction sequence and behavior probability generated by the previous iteration process, and continues to iterate to generate a new denoised sequence, thereby gradually reducing the noise.

[0126] S3, score statistics are performed based on the action label sequence given by the time series segmentation model and the standard specifications of the corresponding medical behavior.

[0127] Specifically, for each label in the predicted action label sequence, it is compared with the standard action sequence. If the label is inconsistent with the standard action, the corresponding deduction score is added to the deduction sequence according to the scoring rules. Each label in the predicted action label sequence is traversed, and the wrong action number and the corresponding deduction sequence are output. The final assessment score is calculated to complete the medical behavior quality assessment.

[0128] Figure 4 The flowchart of the behavior score statistics and feedback part is shown. First, it is necessary to formulate scoring rules according to the steps and error types of cardiopulmonary resuscitation, give the deduction points for each type of wrong label contained in the label space, and give the standard for judging whether the cardiopulmonary resuscitation operation is qualified. In the prediction process, for each label in the predicted action label sequence given by the time series segmentation model, it is compared with the label corresponding to the standard action in the action judgment unit. If it is inconsistent, the score calculation unit deducts the score of this operation in the scoring rules and records the wrong action number. Finally, the total deduction is calculated based on all the deductions, and compared with the qualified standard to obtain the final assessment result. At the same time, the system will map the wrong action number to the specific step name, and give the final feedback and improvement plan in the form of natural language, and finally display it visually on the system page.

[0129] Example 2

[0130] This embodiment provides a medical behavior quality assessment system based on time sequence action segmentation, which is used to implement the method described above. Figure 5 As shown, the system includes:

[0131] 1. Video receiving unit, used to obtain all operation behavior videos of the entire medical process collected by shooting equipment with multiple perspectives, and analyze the dimension and frame number of the video.

[0132] 2. A temporal segmentation unit, which is used to input video data into a temporal segmentation model for temporal action segmentation and recognition. The temporal segmentation model introduces a diffusion model, including a feature extraction module, an encoder and a decoder. The feature extraction module extracts the I3D features corresponding to the video, the encoder encodes the general I3D features into features that are more suitable for the task, and the decoder completes the iterative process of the diffusion model and iteratively generates a predicted action label sequence from random noise.

[0133] 3. The scoring statistics unit is used to perform scoring statistics based on the action label sequence given by the time series segmentation model and the standard specifications of the corresponding medical behavior. For each label in the predicted action label sequence, it is compared with the standard action sequence. If the label is inconsistent with the standard action, the corresponding deduction score is added to the deduction sequence according to the scoring rules, and each label in the predicted action label sequence is traversed. The wrong action number and the corresponding deduction sequence are output, and the final assessment score is counted to complete the medical behavior quality assessment. The scoring statistics unit can score each operation step in the medical behavior according to the scoring standard of the medical behavior and the final result of the behavior label, and form a detailed score analysis report to provide more readable feedback results.

[0134] 4. The result feedback unit is responsible for displaying and visualizing the evaluation results of the entire medical behavior process. After receiving the recognition results and corresponding scores transmitted by the score statistics unit, the result feedback unit organizes the evaluation results of the evaluation algorithm for each stage of the action execution process into natural language, gives the evaluation score, deduction reason and correct specification of each action, and visualizes the final assessment score of the entire medical behavior process, and provides action improvement suggestions based on the stage score. The result feedback unit completes the presentation of the final score report to provide a basis for evaluating the standardization of medical behavior.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0136] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for evaluating the quality of medical behavior based on temporal action segmentation, characterized in that: The following steps are involved: Obtain all operation behavior videos of the entire medical process captured by shooting equipment from multiple perspectives, and analyze the dimensions and frame number of the video; The video data is input into a temporal segmentation model for temporal action segmentation and recognition. The temporal segmentation model introduces a diffusion model, which includes a feature extraction module, an encoder, and a decoder. The feature extraction module extracts I3D features corresponding to the video, the encoder encodes the general I3D features into features that are more suitable for the task, and the decoder completes the iterative process of the diffusion model and iteratively generates a predicted action label sequence from random noise; The score statistics are performed based on the action label sequence given by the time series segmentation model and the standard specifications of the corresponding medical behavior. For each label in the predicted action label sequence, it is compared with the standard action sequence. If the label is inconsistent with the standard action, the corresponding deduction score is added to the deduction sequence according to the scoring rules. Each label in the predicted action label sequence is traversed, and the wrong action number and the corresponding deduction sequence are output. The final assessment score is counted to complete the medical behavior quality assessment.

2. According to claim 1, a method for evaluating the quality of medical behavior based on temporal action segmentation is characterized in that: The temporal segmentation model is trained using a series of training set data annotated with real action labels. For each video in the training set and its corresponding real reference value, a forward process is first performed: random noise is added to the real reference value, and then a backward process is performed: the decoder iteratively generates a probability sequence by receiving the reference value with added random noise, the video features encoded by the encoder, and the number of iterations, that is, the probability that each frame image represents each action, and performs sequence conversion based on the probability to obtain a predicted action sequence label; the goal of the temporal segmentation model in training is to minimize the difference between the value predicted by the decoder and the real reference value, even if the loss function is reduced during the iteration.

3. According to claim 2, a method for evaluating the quality of medical behavior based on temporal action segmentation is characterized in that: The forward process is specifically as follows: Assume that the original true sequence distribution is x0~q(x0), the number of iterations is S, and the noise sequence is β1,β2,…,β S , β is a hyperparameter with a value range of 0 to 1. The single-step noise addition operation on the true reference value is expressed as: Among them, x s is the data after s times of noise addition, I is the unit matrix, and N(·) represents the normal distribution; Let α s =1-β s , According to the additive property of normal distribution, x0 is added with noise in one step to obtain x s The process of writing: Given the original data x0, the data x after adding S noises is S is represented as: Among them, ∈~N(0,1), obeys the standard normal distribution.

4. According to claim 3, a method for evaluating the quality of medical behavior based on temporal action segmentation is characterized in that: The backward process samples a set of data x from a random noise distribution S ~N(0,1), and gradually restore it to the original data x0, defining the recovery function as p θ (x s ), the single-step denoising process is expressed as: Among them, θ is the neural network parameter, and the mean μ θ (x s ,s) is estimated by neural network, the standard deviation σ s β S control; Repeating the above single-step denoising process can restore the original data x0 and obtain the complete denoising path. Suppose the complete denoising path is {x0, x1, …, x S }, then the joint probability distribution of the denoising path is: During the training of the time series segmentation model, the decoder g is fed with the sequence x after adding noise. S As the initial input, it also receives the video features E encoded by the encoder and the set number of iterations S, and generates a probability sequence to represent the probability of each action represented by the denoised video frame: P S =g(x S ,S,E) Among them, P S Represents the input x S The probability that the denoised video frames represent each action; The decoder performs sequence conversion based on the predicted action probability and iteratively generates a sequence of predicted action labels. The update rule for each iteration is as follows: Among them, P s Represents the input x′ s The action probability, that is, P s =g(x′ s ,S,E),x′ s Represents the action label sequence of the current iteration. In the first iteration, x′ s Take x S ; Updated x′ s-1 It is sent to the decoder again to predict P s-1 , thus proceeding to the next round of iteration; After S rounds of iterations, the final output model predicts the action label sequence x′0.

5. According to claim 4, a method for evaluating the quality of medical behavior based on temporal action segmentation is characterized in that: The total loss function of the temporal segmentation model is used to measure the difference between the predicted action label sequence x′0 and the true sequence x0. It consists of cross entropy loss, temporal smoothing loss and boundary alignment loss, which are used to analyze frame-level differences, local label differences and behavior switching position prediction differences respectively. Among them, the cross entropy loss The prediction difference for each frame is measured and expressed as: Among them, L is the total number of frames, C is the total number of categories, l and c represent the indexes of frames and categories respectively, x0 represents the real sequence, P S represents the predicted action probability output by the decoder; Time series smoothing loss The log-likelihood mean square error between adjacent video frames is calculated and expressed as: Boundary alignment loss is used to measure the model's prediction deviation of the behavior switching position. For each benchmark original sequence x0, generate its corresponding boundary sequence B∈{0,1} N-1 , where B is 1, indicating that a behavior label switch has occurred: Use Gaussian filter to smooth the boundary sequence B and get the smoothed sequence Use a binary loss function to measure x′0 and The boundary prediction error between them is used to obtain the boundary alignment loss Then the total loss function of the time series segmentation model is expressed as:

6. The method for evaluating the quality of medical behavior based on temporal action segmentation according to claim 1, characterized in that: The multi-angle shooting equipment includes a multi-angle camera set up at the medical site and a head-mounted camera worn on the operator's head.

7. The method for evaluating the quality of medical behavior based on temporal action segmentation according to claim 1, characterized in that: The feature extraction module uses an I3D model to extract features. The I3D model is implemented by extending a 2D convolutional neural network to 3D, that is, expanding a 2D convolution kernel to a time dimension to form a 3D convolution kernel to simultaneously process the spatial and temporal information of the video. The I3D model architecture is divided into three parts: an input layer, an Inception layer, and a pooling layer. After the input video passes through the three parts of the I3D model, an I3D feature F is generated.

8. The method for evaluating the quality of medical behavior based on temporal action segmentation according to claim 1, characterized in that: The encoder uses the ASFormer model to further encode the I3D features. The ASFormer model uses the Transformer architecture to process video data. By introducing the spatiotemporal attention mechanism to more effectively capture the complex spatiotemporal relationship in the video, the encoder-encoded feature E is obtained.

9. A medical behavior quality assessment system based on temporal action segmentation, characterized in that: For implementing the method according to any one of claims 1 to 8, the system comprises: The video receiving unit is used to obtain the video of all the operation behaviors in the entire medical process collected by the shooting equipment with multiple viewing angles, and analyze the dimension and frame number of the video; A temporal segmentation unit is used to input video data into a temporal segmentation model for temporal action segmentation and recognition. The temporal segmentation model introduces a diffusion model and includes a feature extraction module, an encoder, and a decoder. The feature extraction module extracts I3D features corresponding to the video, the encoder encodes the general I3D features into features that are more suitable for the task, and the decoder completes the iterative process of the diffusion model and iteratively generates a predicted action label sequence from random noise; The scoring statistics unit is used to perform scoring statistics based on the action label sequence given by the time series segmentation model and the standard specifications of the corresponding medical behavior. For each label in the predicted action label sequence, it is compared with the standard action sequence. If the label is inconsistent with the standard action, the corresponding deduction score is added to the deduction sequence according to the scoring rules. Each label in the predicted action label sequence is traversed, and the wrong action number and the corresponding deduction sequence are output. The final assessment score is counted to complete the medical behavior quality assessment.

10. A medical behavior quality assessment system based on temporal action segmentation according to claim 9, characterized in that: The system also includes a result feedback unit, which is responsible for displaying and visualizing the evaluation results of the system on the entire medical behavior process. After receiving the recognition results and corresponding scores transmitted by the score statistics unit, the result feedback unit organizes the evaluation results of the evaluation algorithm on each stage of the action execution process into natural language, and gives the evaluation score, reason for deduction and correct specifications for each action, as well as visualizes the final assessment score of the entire medical behavior process and provides action improvement suggestions based on the stage scores.

Images