Method for explaining the bias of an action quality assessment model based on super voxel

Abstract

The application provides a kind of action quality evaluation model based on the tendency explanation method of super voxel, comprising: according to the activation vector of the i-th super voxel, the g-th label and the corresponding value of the g-th label, the sensitivity score of the i-th super voxel to the g-th label is calculated;According to the sensitivity score of the i-th super voxel to the g-th label, the score tendency of the i-th super voxel to the action quality evaluation model is calculated.By super voxel segmentation of video, the influence of noise on experimental results can be reduced and it is easy to understand;Use the interlayer activation vector in deep neural network to represent each super voxel, then calculate the sensitivity score of super voxel to label, and finally calculate the score tendency of super voxel to action quality evaluation model according to the sensitivity score, so that the influence of action and background information in video on the judgment of action quality evaluation model can be more intuitively seen.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video motion evaluation, in particular, especially relates to a tendency explanation method of motion quality evaluation model based on super voxel. BACKGROUND

[0002] In recent years, video motion quality evaluation has become a hot research topic, aiming to evaluate the execution of a specific motion.

[0003] In the processing of videos with difficult inter-class judgments, most of the methods proposed in recent research use deep neural network models for end-to-end learning, which can correctly classify motion sequences. However, these models do not have overall mechanisms that are interpretable, i.e., so-called "black box" models. In the strictly required medical field, this defect is particularly prominent, making it difficult to support causal reasoning in medical research. For traditional interpretable methods, they mainly use pixel-level or visual concept-based methods to explain deep models. The former interpretable method scores the importance of each pixel in the video, which produces a lot of noise and is difficult for humans to understand. The latter proposes "visual concepts" that are easy to understand, but requires pre-proposed concepts. This concept-based method attributes similar motions or scenes to a concept and calculates the sensitivity score of the concept to the model, but cannot determine which motions in the concept are standard.

[0004] Therefore, it is urgent to propose a tendency explanation method of motion quality evaluation model based on super voxel. SUMMARY

[0005] In order to solve the above-mentioned technical problem of uninterpretable video motion evaluation, a tendency explanation method of motion quality evaluation model based on super voxel is provided. The present application mainly performs super voxel segmentation on the video, so that the motion or background in the video exists in the form of a three-dimensional block in chronological order, which can reduce the influence of noise on experimental results and enable human subjective understanding. Then, a classifier is trained to distinguish each super voxel, the inter-layer activation vector in the deep neural network is used to represent each super voxel, and finally the directional derivative is used to quantify the super voxel to the motion quality evaluation model score tendency, so that the influence of the motion and background information in the video on the judgment of the motion quality evaluation model can be more intuitively seen.

[0006] The technical means adopted by the present application are as follows:

[0007] The present application provides a tendency explanation method of motion quality evaluation model based on super voxel, comprising:

[0008] inputting an original video into a trained neural network model to obtain feature information, wherein the original video comprises N pixels and G class labels;

[0009] embedding time and position information of N pixels into the feature information to obtain a feature matrix;

[0010] uniformly dividing the feature matrix into K super voxels and distributing N pixels into the K super voxels;

[0011] performing Gaussian blur operation on the original video to generate a specific video;

[0012] retaining a pixel region of an i-th super voxel in the original video and processing to obtain an i-th related video, 1≤i≤K;

[0013] inputting the specific video and the i-th related video into a motion quality assessment model to obtain an i-th feature Figure Two tuple;

[0014] inputting the i-th feature Figure Two tuple into a linear classifier to obtain an activation vector of the i-th super voxel;

[0015] inputting the original video into the motion quality assessment model to obtain a gradient of a g-th label with respect to the motion quality assessment model, 1≤g≤G;

[0016] calculating a sensitivity score of the i-th super voxel with respect to the g-th label according to the activation vector of the i-th super voxel, the g-th label, and a corresponding value of the g-th label;

[0017] calculating a score tendency of the i-th super voxel with respect to the motion quality assessment model according to the sensitivity score of the i-th super voxel with respect to the g-th label, according to the following formula:

[0018]

[0019] wherein S is the score tendency of the i-th super voxel with respect to the motion quality assessment model, g is the g-th label, C is the sensitivity score, maxC is a maximum value of the sensitivity score, minC is a minimum value of the sensitivity score, Label g is a constant.

[0020] Further, the inputting the i-th feature Figure Two tuple into the linear classifier to obtain the activation vector of the i-th super voxel is calculated according to the following formula:

[0021]

[0022] wherein, is the activation vector of the i-th super voxel, V0 is the specific video, V i ′ is the i-th related video, (f l(V′ i ), f l (V0)) is the i-th feature Figure Two tuple, is a linear classifier.

[0023] Further, the sensitivity score of the i-th super voxel to the g-th label is calculated according to the activation vector of the i-th super voxel, the g-th label and the corresponding value of the g-th label, in the following way:

[0024]

[0025] wherein C i，l，g (V) is the sensitivity score of the i-th super voxel to the g-th label, is the activation vector of the i-th super voxel, and V is the original video, is a gradient operation, and h l，g is the output of the g-th label to the i-th layer of the action quality evaluation model.

[0026] Further, the distribution of N pixels into K super voxels includes:

[0027] initializing the centers of K super voxels respectively;

[0028] calculating the Euclidean distance of the pixels to the centers of K super voxels respectively, and distributing the pixels into the super voxel with the minimum Euclidean distance;

[0029] updating the centers of K super voxels, and returning to the iteration of calculating the Euclidean distance of the pixels to the centers of K super voxels respectively, and distributing the pixels into the super voxel with the minimum Euclidean distance;

[0030] when the distribution of N pixels no longer changes, the distribution is completed.

[0031] Further, the calculation of the Euclidean distance of the pixels to the center of the i-th super voxel is calculated in the following way:

[0032]

[0033] wherein A (p，i) is the Euclidean distance of the pixels to the center of the i-th super voxel, and F′ p is the feature vector of the p-th pixel.

[0034] Further, the updating of the centers of K super voxels includes:

[0035] establishing a relationship matrix of N pixels and K superpixels;

[0036] updating the center of the i-th superpixel, and the updating is calculated in the following manner:

[0037] S i =A T F′;

[0038] Wherein, A is the relationship matrix of N pixels and K superpixels, A∈[0,1] N×K , F′ is the feature matrix.

[0039] Further, the feature matrix is calculated by embedding the time and position information of N pixels into the feature information in the following manner:

[0040] F′=F+P;

[0041] Wherein, F′ is the feature matrix, F is the feature information, and P is the time and position information of N pixels.

[0042] Compared with the prior art, the application has the following advantages:

[0043] The action quality evaluation model based on the superpixel provided by the application has the following advantages: The action quality evaluation model based on the superpixel provided by the application has the following advantages:

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0045] Figure 1 A flow chart of the action quality evaluation model based on the superpixel provided by the application.

[0046] Figure 2 An effect diagram of being segmented into different number of superpixels.

[0047] Figure 3A visualization diagram of the sensitivity of the super voxel to the action quality evaluation model in the RISE method.

[0048] Figure 4 A visualization diagram of the sensitivity of the super voxel to the action quality evaluation model in the RISE method. Figure 3 A visualization diagram of the sensitivity of the super voxel to the action quality evaluation model in the RISE method.

[0049] Figure 5 A visualization diagram of the sensitivity of the super voxel to the action quality evaluation model in the RISE method.

[0050] Figure 6 A visualization diagram of the sensitivity of the super voxel to the action quality evaluation model in the RISE method. Figure 5 A visualization diagram of the sensitivity of the super voxel to the action quality evaluation model in the RISE method.

[0051] Figure 7 A comparison diagram of super voxel segmentation using different methods.

[0052] Figure 8 A sensitivity tendency statistical diagram of the super voxel to the action quality evaluation model in a low segmentation video.

[0053] Figure 9 A sensitivity tendency statistical diagram of the super voxel to the action quality evaluation model in a medium segmentation video.

[0054] Figure 10 A sensitivity tendency statistical diagram of the super voxel to the action quality evaluation model in a high segmentation video.

[0055] Figure 11 A comparison diagram of action sequences.

[0056] Figure 12 A score-time relationship diagram.

[0057] Figure 13 A visualization diagram of the tendency of the super voxel score. DETAILED DESCRIPTION

[0058] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0059] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0060] Embodiment 1

[0061] Reference Figure 1 , Figure 1 A flowchart of a bias explanation method of a super-voxel-based action quality evaluation model provided by the present application is used to illustrate a specific embodiment of the bias explanation method of the super-voxel-based action quality evaluation model provided by the present embodiment, which includes the following steps:

[0062] The original video is input into the trained neural network model to obtain feature information, and the original video includes N pixels and G labels.

[0063] The time and position information of the N pixels are embedded into the feature information to obtain a feature matrix.

[0064] The feature matrix is evenly divided into K super-voxels, and the N pixels are distributed into the K super-voxels.

[0065] The original video is subjected to a Gaussian blur operation to generate a specific video.

[0066] The pixel area of the i-th super-voxel in the original video is retained and processed to obtain an i-th related video, 1≤i≤K.

[0067] The specific video and the i-th related video are input into the action quality evaluation model to obtain an i-th feature Figure Two tuple.

[0068] The i-th feature Figure Two tuple is input into a linear classifier to obtain an activation vector of the i-th super-voxel.

[0069] The original video is input into the action quality evaluation model to obtain a gradient of the g-th label with respect to the action quality evaluation model, 1≤g≤G.

[0070] According to the activation vector of the i-th super-voxel, the g-th label, and the corresponding value of the g-th label, a sensitivity score of the i-th super-voxel with respect to the g-th label is calculated.

[0071] According to the sensitivity score of the ith super voxel to the gth label, the score tendency of the ith super voxel to the action quality evaluation model is calculated according to the following formula:

[0072]

[0073] Wherein, S is the score tendency of the ith super voxel to the action quality evaluation model, g is the gth label, C is the sensitivity score, maxC is the maximum value of the sensitivity score, minC is the minimum value of the sensitivity score, Label g is a constant.

[0074] Embodiment 2

[0075] With reference to the above description of the embodiment, another specific embodiment of the tendency explanation method of the super voxel-based action quality evaluation model provided by the embodiment is explained, which includes: Figure 1

[0076] The original video V is input into the trained neural network model USC to obtain the feature information F, and the original video V includes N pixels and G class labels;

[0077] The time and position information P of the N pixels is embedded into the feature information to obtain the feature matrix F';

[0078] Specifically, the time and position information P of the N pixels is embedded into the feature information to obtain the feature matrix F', which is calculated in the following manner:

[0079] F' = F + P;

[0080] Wherein, F' is the feature matrix, F is the feature information, and P is the time and position information of the N pixels. The feature matrix F' can be expressed as F' ∈ R N×m , R N×m is a matrix containing N pixels, and the dimension of each pixel is m.

[0081] The feature matrix F' is evenly divided into K super voxels, and the N pixels are allocated into the K super voxels;

[0082] Specifically, the N pixels are allocated into the K super voxels, which includes:

[0083] The centers of the K super voxels are initialized respectively;

[0084] The Euclidean distances of the pixels to the centers of the K super voxels are calculated respectively, and the pixels are allocated into the super voxel with the minimum Euclidean distance;

[0085] The centers of the K super voxels are updated, and the Euclidean distances of the pixels to the centers of the K super voxels are calculated respectively, and the pixels are allocated into the super voxel with the minimum Euclidean distance.​

[0086] When the assignment of N pixels no longer changes, the assignment is completed.

[0087] In initializing the centers of K super voxels, the centers of super voxels are initialized in the following manner:

[0088] S 0 ∈R K×m ;

[0089] Wherein, S 0 is the center of the initialized super voxel.

[0090] And the Euclidean distance between the pixel and the center of the i-th super voxel is calculated in the following manner:

[0091]

[0092] Wherein, A (p，i) is the Euclidean distance between the pixel and the center of the i-th super voxel, F p ′ is the feature vector of the p-th pixel. It should be noted that through calculation, each pixel will obtain a minimum Euclidean distance corresponding thereto, and the sum of the weights of the N minimum Euclidean distances corresponding to the N pixels is 1.

[0093] When updating the centers of K super voxels, it includes:

[0094] Establishing a relationship matrix between N pixels and K super voxels;

[0095] Wherein, the relationship matrix between N pixels and K super voxels can be expressed as A∈[0,1] N×K The relationship matrix between N pixels and K super voxels includes the Euclidean distance between the pixel and the center of the i-th super voxel.

[0096] Updating the center of the i-th super voxel is calculated in the following manner:

[0097] S i =A T F′;

[0098] Wherein, A is the relationship matrix between N pixels and K super voxels, A∈[0,1] N×K , and F′ is the feature matrix.

[0099] In the process of continuously updating the relationship matrix between N pixels and K super voxels, the final pixel-super voxel relationship matrix is obtained, which is calculated in the following manner:

[0100] H∈R N ;

[0101] H p =argmaxi A (p，i) ;

[0102] Among them, R N argmax represents an N x 1 vector matrix. i A (p，i) Indicates that A (p，i) Extract the index i of the largest item in H. p This is the final matrix relating pixels to supervoxels.

[0103] To generate a specific video V0 by performing a Gaussian blur operation on the original video V, the following calculation method is used:

[0104] V0 = GaussianBlur(V);

[0105] Where V0 represents a specific video and V represents the original video.

[0106] The pixel region of the i-th supervoxel in the original video is preserved and processed to obtain the i-th related video V. i ′, 1≤i≤K, specific video V0, first relevant video V1′, ..., Kth relevant video V K The video dataset V' = {V0, V1', V2', ..., V'} is composed of... K ′};

[0107] Connect the specific video V0 and the i-th related video V i The input action quality assessment model USDL obtains the i-th feature. Figure Two tuple (f l (V i ′), f l (V0));

[0108] The i-th feature Figure Two tuple (f l (V i ′), f l (V0)) Input linear classifier The activation vector of the i-th hypervoxel is obtained.

[0109] Among them, in the i-th feature Figure Two tuple (f l (V i ′), f l (V0)) Input linear classifier Before that, we also need to consider the i-th feature. Figure Two tuple (f l (V i ′), f l (V0)) is reshaped, and the i-th feature is... Figure Twotuple (f l (V i ′), f l (V0)) input linear classifier , the activation vector of the ith super voxel is obtained , and the following formula is used for calculation:

[0110]

[0111] wherein, is the activation vector of the ith super voxel, V0is a specific video, V i ′ is the ith related video, (f l (V i ′), f l (V0)) is the ith feature Figure 2 tuple, is a linear classifier.

[0112] The original video V is input into the action quality evaluation model USDL to obtain the gradient h l，g of the action quality evaluation model for the gth label, 1≤g≤G; wherein, the obtained gradient h l，g is the gradient of the feature layer l.

[0113] According to the activation vector of the ith super voxel , the gth label and the corresponding numerical value of the gth label, the sensitivity score of the ith super voxel to the gth label is calculated.

[0114] Specifically, according to the activation vector of the ith super voxel , the gth label and the corresponding numerical value of the gth label, the sensitivity score of the ith super voxel to the gth label is calculated, C i，l，g (V), which is calculated in the following manner:

[0115]

[0116] wherein, C i，l，g (V) is the sensitivity score of the ith super voxel to the gth label, is the activation vector of the ith super voxel, V is the original video, is the gradient operation, h l，g is the reverse gradient of the lth layer of the action quality evaluation model for the gth label.

[0117] According to the sensitivity score C i，l，g (V) of the ith super voxel to the gth label, the score tendency S of the ith super voxel to the action quality evaluation model is calculated according to the following formula:

[0118]

[0119] wherein S is the score propensity of the i-th super voxel pair motion quality assessment model, g is the g-th label, C is the sensitivity score, maxC is the maximum sensitivity score, minC is the minimum sensitivity score, Label g is a constant.

[0120] The score propensity S of the i-th super voxel pair motion quality assessment model can explain whether the motion is standard.

[0121] Embodiment 3

[0122] The motion quality assessment model USDL and the neural network model UCS, which is a video feature map model UCS, are trained using three public data sets KnotTying, NeedlePassing and Suturing of JIGSAWS, wherein the KnotTying data set is a "knot tying data set", which is a data set for surgical skill assessment, and is a video record of a surgeon practicing ligation skills in medical training; the NeedlePassing is a "needle passing data set"; and the Suturing data set is a "suture data set".

[0123] First, a super voxel segmentation experiment of a video is performed, the original video V is input to the trained video feature map model UCS, the feature information F of the last layer of the Encoder framework of the original video V is extracted, and then the time and position information P of the pixels of the original video V is embedded in F to obtain a feature matrix F′.

[0124] The super voxels are initialized in a gridding manner, and the number of super voxels is set to a suitable value K = 80, a soft clustering operation is performed on the feature matrix F′, the number of iterations is set to Epoch = 500, and a final pixel and super voxel relationship matrix H is obtained.

[0125] Next, a sensitivity evaluation experiment of the super voxel pair motion quality assessment model USDL is performed.

[0126] First, a specific video V0 with the same dimension as the original video V is generated using a Gaussian blur method, and then the pixels contained in the i-th super voxel are restored to the pixel values of the original video V based on the specific video V0, and the remaining part is unchanged to obtain the i-th related video V i ′, the specific video V0 and the i-th related video V i ′ are input into the motion quality assessment model USDL model respectively, two feature maps of a specific layer l are obtained, and then the two feature maps of the specific layer l are input into a linear classifier to obtain a layer activation vector The original video V is then fed back into the motion quality assessment model USDL to obtain the gradient of each category label at a specific layer l of USDL. This gradient is used to quantify the hypervoxel's tendency to score with USDL. When using USDL and the video feature map model UCS, the video is segmented into L frames. 160 frames are extracted from these frames at intervals, with 16 frames per folder, resulting in ten groups of segments. During training of both models, the size of each frame is cropped from 256x340 to 224x224.

[0127] Example 4

[0128] Reference Figure 2 , Figure 2 To obtain the desired effect from segmented images with varying numbers of supervoxels, for the JIGSAWS dataset, when conducting supervoxel segmentation experiments on videos, it is necessary to segment the video into a specific number of supervoxels, referring to... Figures 3 to 6 It is evident that setting the number of supervoxels to K=80 is optimal, as this number allows for the most complete segmentation of motion and background information in the video. Simultaneously, segmentation experiments with other numbers of supervoxels were also conducted, and their qualitative analysis revealed that when setting a small number of supervoxels, some supervoxels would encapsulate the motion and background in the video, making effective segmentation impossible. When setting a large number of supervoxels, the experiments showed that a complete motion or background in the video would be segmented into many fine supervoxels, failing to preserve a continuous motion and also leading to multiple score biases for the video background, resulting in computational overhead.

[0129] Example 5

[0130] Reference Figure 3 , Figure 4 This is a visualization of the sensitivity of supervoxels to the motion quality evaluation model in the RISE method. Figure 3 To and Figure 5 The corresponding visualization of the sensitivity of hypervoxels to the motion quality evaluation model in the propensity interpretation method based on the hypervoxel motion quality evaluation model provided by this invention is shown in the figure. Figure 6 This is another visualization of the sensitivity of the supervoxel to the motion quality assessment model in the RISE method. Figure 5 To and Figure 7The visualization effect diagram of the sensitivity of the super voxel to the action quality evaluation model in the tendency explanation method of the super voxel-based action quality evaluation model provided by the application. The method provided by the application can quantize the score tendency of the super voxel to the action evaluation model, and the IAUC index and the DAUC index are used to compare the method provided by the application with other explainable methods of the neural network model, wherein the IAUC is the AUC curve under insertion, the DAUC is the AUC curve under deletion, the AUC curve is commonly used for model evaluation and comparison in the field of machine learning, and is a commonly used index for evaluating the performance of a classifier, and can be used to measure the accuracy of the classifier; when the pixels in the video are gradually deleted or added, the decision made by the model will change, so that the probability of predicting the true label rises or falls. For the DAUC index, a lower area under the probability curve will mean a better explanation method; when the IAUC index is used, a higher area under the probability curve will mean a better explanation method. Referring to Table 1, five explainable methods are provided in Table 1, wherein CAM, Grad-CAM, RISE and LIME are existing explainable technologies, CAM is a method for quantizing a feature map using global average pooling to represent the sensitivity of each pixel in the video; Grad-CAM is a method for quantizing the sensitivity of each pixel in the video using a back gradient; RISE is a method for quantizing the sensitivity of each pixel using a random mask; LIME is a method for dividing a video into multiple frames of pictures, performing super pixel segmentation on the pictures, and calculating the sensitivity of each super pixel in a linear fitting method; Ours is the tendency explanation method of the super voxel-based action quality evaluation model provided by the application, which applies the same action evaluation model and data set to the five explainable methods respectively, and calculates the IAUC value and the DAUC value of each method. It can be seen from the experimental results that the DAUC index of the tendency explanation method of the super voxel-based action quality evaluation model provided by the application is lower, and the IAUC index is higher, but the sensitivity tendency calculation method based on Voxel provided by the application is superior to the explainable method of the prior art.

[0131] Table 1. Comparison table of explainable method indexes

[0132]

[0133] Example 6

[0134] Referring to Figure 7 , Figure 7For the comparison of super voxel segmentation using different methods, the video is segmented into super voxels using an unsupervised method. First, deep feature information of the video is extracted. In order to better capture the semantic spatio-temporal structure information of the video, the time and position information of the pixels is embedded into the feature information. Then, a soft clustering method is used to complete the segmentation of the super voxels. The super voxels are initialized in a grid manner, and the relationship matrix between the super voxels and the pixels in the video is iteratively calculated.

[0135] In the deep feature information of the video, the time and position information of each pixel is added, referring to Figure 8 , it is found that the actions of the two scalpels in the same frame belong to the same super voxel. In order to ensure the integrity and continuity of the action, the scalpels in the same frame should belong to the same super voxel, which is a better result. At the same time, the UCS method experiment (UCS method refers to the method disclosed in the paper "Unsupervised copart segmentation through assembly") is also carried out. It proposes a component-based unsupervised method that extracts a component-based generic representation from the video. During training, the repositioning feature map is decoded into the restoration degree of the target image, and the source latent feature map is decoded into the restoration degree of the source image. From the segmentation result, the soft clustering method can segment the scalpels and the needle thread of the panel in the form of super voxels, which is better than the UCS method.

[0136] Example 7

[0137] Referring to Figure 9 , Figure 10 , Figure 11 and Figure 8 , Figure 9 is a sensitivity tendency statistical chart of the super voxel to the action quality evaluation model in the low-segmented video, Figure 10 is a sensitivity tendency statistical chart of the super voxel to the action quality evaluation model in the medium-segmented video, Figure 11 is a sensitivity tendency statistical chart of the super voxel to the action quality evaluation model in the high-segmented video, Figures 8 to 10 is a comparison chart of the action sequence. In the experiment, the directional derivative is used to quantify the score tendency of the super voxel to the action quality evaluation model. Each classification node of the action quality evaluation model has a score significance, and the score label needs to be added to the sensitivity calculation. The value range of the score label of the video is converted from 6-30 to -12-12. According to the activation vector of the ith super voxel, the gth label and the value corresponding to the gth label, the sensitivity score of the ith super voxel to the gth label is calculated. According to the sensitivity score of the ith super voxel to the gth label, the score tendency of the ith super voxel to the action quality evaluation model is calculated. Referring to Figure 11When the score of the super voxel is high, it is determined that the action is standard; when the score of the super voxel is close to 0, the super voxel contains background information of the video, which has no obvious influence on the action evaluation model; when the score of the super voxel is low, it is determined that the action detail is poor, which will lower the judgment score of the action evaluation model.

[0138] In the form of a statistical chart, the score tendency of each super voxel in a high-score, medium-score and low-score video to the action quality evaluation model is given. In each chart, representative super voxels are given matching specific frame pictures. Referring to Figure 12 , two groups of pictures are from a high-score and a low-score video respectively. In the first group of pictures, it can be seen that the action of the surgical knife is more standard, which can pierce the needle into the panel at the right position, and the action is reasonable and decisive; in the second group of pictures, it can be seen that the action of the right surgical knife is unreasonable, and the action is hesitant, and the needle line of the surgical panel is unqualified. The two Voxel scores containing the surgical knife and the surgical panel calculated by the tendency interpretation method of the action quality evaluation model provided in the embodiment are low.

[0139] Embodiment 8

[0140] Referring to Figure 13 and Figure 12 , Figure 13 is a graph of score versus time, Figure 13 is a visualization chart of the super voxel score tendency, Figure 12 in the form of a heat map, the sensitivity tendency of each super voxel in the video to the model is shown, in Figure 12 , the X axis, the Y axis and the Z axis respectively represent the Clip number, the score label and the probability of score prediction, the video is divided into 600 frame pictures, 160 frame pictures are extracted in an equal interval manner, 16 frames are a group, and there are 10 video segments, each video segment is a Clip, and Figure 13 it can be seen that the score of the video segment in the frame is low. Referring to Figure 12 , the video action visualization along the time sequence, a video in the Suturing data set in the JIGSWAS data set is selected for visualization experiment, Figure 12 shows the evolution of the score distribution over time, the score distribution of 10 Clips is obtained and plotted on the chart, the real label score of the video is high, and the prediction score given by the network is high. From the time evolution of the score distribution, it can be seen that the network gives the prediction, in the 9th clip, the operation performed has a problem, resulting in a low score prediction. In the ninth Clip, a low-score super voxel is marked, the action contained in the super voxel produces an error operation, and then lowers the prediction score given. As ​The video of the Suturing class in the JIGSWAS dataset is super voxel segmented, and the score tendency of each super voxel to the action quality evaluation model is calculated. The interpretable method proposed in the application is based on the form of super voxel, and the action of the video scalpel is quantitatively analyzed. When the score of the super voxel is low, it can be found that the action contained is not standard, the operation action is hesitant, and the thread cannot be tied into the correct position. Similarly, when the score is high, it can be found that the action contained is almost perfect. For the super voxel in the middle score segment, the background information of the video contained can be found, and the action evaluation model is little affected.

[0141] In the above-mentioned embodiments of the application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0142] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A method for explaining the tendency of an action quality evaluation model based on super voxel, characterized in that, The method comprises the following steps: inputting an original video into a trained neural network model to obtain feature information, wherein the original video comprises N pixels and G labels; embedding time and position information of the N pixels into the feature information to obtain a feature matrix; dividing the feature matrix into K super voxels uniformly and assigning the N pixels to the K super voxels; performing Gaussian blur operation on the original video to generate a specific video; retaining a pixel region of an i-th super voxel in the original video and processing the pixel region to obtain an i-th related video, wherein 1≤i≤K; inputting the specific video and the i-th related video into an action quality evaluation model to obtain an i-th feature map pair; inputting the i-th feature map pair into a linear classifier to obtain an activation vector of the i-th super voxel; inputting the original video into the action quality evaluation model to obtain a gradient of a g-th label with respect to the action quality evaluation model, wherein 1≤g≤G; calculating a sensitivity score of the i-th super voxel with respect to the g-th label according to the activation vector of the i-th super voxel, the g-th label and a corresponding value of the g-th label; calculating a score tendency of the i-th super voxel with respect to the action quality evaluation model according to the sensitivity score of the i-th super voxel with respect to the g-th label, wherein the score tendency is calculated according to the following formula: wherein S is the score propensity of the ith super voxel to the motion quality assessment model, g is the gth label, C is the sensitivity score, maxC is the maximum value of the sensitivity score, minC is the minimum value of the sensitivity score, Label g is a constant. 2.The method of claim 1, wherein, the calculation of the activation vector of the i-th super voxel by inputting the i-th feature map pair into the linear classifier is calculated according to the following formula: where T l i is the activation vector of the ith super voxel, V0is the particular video, V i ′ is the ith related video, (f l (V i ′), f l (V0)) is the ith feature map pair, is a linear classifier. 3.The method of claim 1, wherein, the calculation of the sensitivity score of the i-th super voxel with respect to the g-th label according to the activation vector of the i-th super voxel, the g-th label and the corresponding value of the g-th label is calculated according to the following formula: wherein C i，l，g (V) is a sensitivity score of the ith super voxel to the gth label, T l i is an activation vector of the ith super voxel, V is the original video, is a gradient operation, h l，g is to solve the output of the gth label to the action quality evaluation model lth layer of the back propagation. 4.The method of claim 1, wherein, the assigning of the N pixels to the K super voxels comprises the following steps: initializing centers of the K super voxels respectively; calculating Euclidean distances of the pixels to centers of the K super voxels respectively and assigning the pixels to the super voxels with the minimum Euclidean distances; updating the centers of the K super voxels and returning to the step of calculating the Euclidean distances of the pixels to the centers of the K super voxels respectively and assigning the pixels to the super voxels with the minimum Euclidean distances for iteration; when the assignment of the N pixels no longer changes, the assignment is completed. 5.The method of claim 4, wherein, the calculation of the Euclidean distance of the pixel to the center of the i-th super voxel is calculated according to the following formula: wherein A (p，i) is the Euclidean distance of the pixel to the center of the i-th super voxel, F′ p is the feature vector of the p-th pixel. 6.The method of claim 4, wherein, the updating of the centers of the K super voxels comprises the following steps: establishing a relationship matrix of the N pixels and the K super voxels; updating the center of the i-th super voxel according to the following formula: S i = A T F'; Wherein, A is a relationship matrix of N pixels and K superpixels, A∈[0, 1] N×K , F' is the feature matrix. 7.The method of claim 1, wherein, the embedding of the time and position information of the N pixels into the feature information to obtain the feature matrix is calculated according to the following formula: F′=F+P; wherein F′ is the feature matrix, F is the feature information, and P is the time and position information of the N pixels.

Images