Practical operation process stability evaluation method and device based on visual analysis

Abstract

The invention provides a practical operation process stability assessment method and device based on visual analysis, and particularly relates to the field of stability assessment, and the method comprises the steps: extracting each frame of image from a teacher practical operation video, generating a multi-channel heat map for a preset key point, constructing a two-dimensional human skeleton posture of each frame based on preset limb connection, and constructing a multi-channel heat map; and performing three-dimensional reconstruction on the continuous two-dimensional attitude sequence to obtain a three-dimensional space coordinate sequence of the key points, and calculating a comprehensive stability evaluation score based on the three-dimensional space coordinate sequence to obtain the comprehensive stability evaluation score. According to the method, the stability of limb connection in a complex shielding environment is enhanced, accurate construction of a two-dimensional posture is ensured, and an objective and accurate comprehensive evaluation result of the stability in the practical operation process can be provided.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of stability evaluation, and particularly relates to a real operation process stability evaluation method and device based on visual analysis. BACKGROUND

[0002] In the applications such as vocational skill training, sports teaching or medical operation demonstration, it is of great significance to evaluate the stability and standardization of the real operation process of teachers or experts. This not only requires high-precision human posture data, but also needs deep motion quality analysis of the posture sequence. Existing motion quality evaluation methods usually rely on kinematic features extracted from the posture sequence, such as the displacement, velocity or acceleration of key points. However, these simple low-order kinematic indicators are difficult to fully measure the smoothness and fluency unique to expert-level operations. Human posture estimation methods locate human key points such as joints, facial features, etc. and construct human skeleton structures, which are applied to human-computer interaction, action recognition and other scenarios. The mainstream multi-person posture estimation method adopts a bottom-up paradigm, that is, all possible human key points are first detected in the image, and the key points are assembled into different individual skeletons through learned inter-component association information such as component affinity field (PAF). When generating a response map (heat map) of key point positions, this kind of method cannot adjust the confidence of the response peak value according to the motion blur degree of the key points, resulting in a decrease in positioning accuracy in fast motion scenes; when the human limbs are shortened due to the perspective effect, the isotropic circular Gaussian kernel cannot accurately describe the non-uniform positioning uncertainty of the key points in the image plane, affecting the accuracy of posture estimation. When the associated key points form a limb, the PAF method is easily disturbed by the occlusion of non-human targets such as tools, equipment, tables and chairs in the environment when calculating the connection confidence, resulting in incorrect limb connection, especially in complex environments such as real operation teaching by teachers. It is an urgent problem to be solved in the current technical field to obtain posture in a complex occlusion environment and to comprehensively evaluate the motion smoothness and limb coordination. SUMMARY

[0003] The purpose of the present application is to provide a real operation process stability evaluation method and device based on visual analysis, which is used to solve the positioning inaccuracy problem caused by fast motion blur and limb perspective shortening, comprising the following steps: Obtaining a video image sequence of the real operation process of the teacher, for each frame of image in the sequence, generating a key point heat map of each key point in a preset human key point set, and constructing a two-dimensional human skeleton posture based on each limb connection in a preset limb connection set; performing three-dimensional reconstruction on the two-dimensional human skeleton posture sequence of the continuous frames to obtain a three-dimensional spatial coordinate sequence of each key point; Extract the motion trajectory of the specified teaching core limb key point, calculate the time third derivative of the motion trajectory as the motion smoothness index; select two functionally coupled key points, respectively, to the motion trajectory of the point, calculate the phase consistency of the two in the main motion frequency band as the limb coordination degree index; through the nonlinear mapping function, the motion smoothness index and the limb coordination degree index are fused, and the comprehensive stability evaluation score of the teacher's operation process is obtained.

[0004] Optionally, the generating of the key point heat map of each key point in the preset human key point set comprises: A motion state prediction model is maintained for each key point, and a confidence scaling coefficient of a heat map Gaussian peak is determined based on a state prediction residual between a predicted value and an actual observed value of a current frame key point position of the motion state prediction model, and the greater the state prediction residual, the smaller the confidence scaling coefficient; Based on the three-dimensional pose reconstruction result of the previous frame, the direction vector of the limb connecting the key points in the three-dimensional space is calculated, the rotation principal axis of the elliptical Gaussian kernel is determined according to the projection of the direction vector in the image plane, and the flatness of the elliptical Gaussian kernel is determined according to the cosine of the angle between the three-dimensional direction vector and the line-of-sight vector of the camera optical center; The confidence scaling coefficient and the elliptical Gaussian kernel are combined to generate the heat map of the key point.

[0005] Optionally, a two-dimensional human skeleton pose is constructed based on each limb connection in the preset limb connection set, comprising: Instance segmentation is performed on the current frame image to obtain a human region mask and a pixel mask and semantic category of a non-human target; Based on the human region mask, the distance transform of the pixels in the human region to the region boundary is calculated, and the gradient field of the distance transform is obtained; A vector field is generated between the two key points constituting the limb connection, and the vector at any position in the vector field is obtained by nonlinear combination of the reference direction vector connecting the two key points and the gradient field vector at the position; Line integration is performed on the vector field along the candidate path to calculate the candidate limb confidence of connecting the two key points, and when the candidate path passes through the pixel mask of the non-human target, the integral value of the path section is multiplied by a path attenuation coefficient, and the path attenuation coefficient is determined by the occlusion weight corresponding to the semantic category of the non-human target and the normalized position of the occlusion section on the candidate path. Based on the confidence of all limb connections, the two-dimensional human skeleton pose construction of the current frame is completed.

[0006] Optionally, maintaining a motion state prediction model for each key point, and determining the confidence scaling factor of the Gaussian peak in the heatmap based on the state prediction residual between the predicted value and the actual observed value of the key point position in the current frame by the motion state prediction model, includes: The motion state prediction model is a Kalman filter, and the motion state vector includes the position, velocity, and acceleration of the key points in the image plane. The state prediction residual is the quadratic form of the inverse of the innovation covariance matrix and the innovation vector in the Kalman filter update step, i.e., the square of the Mahalanobis distance between the predicted and observed positions. ; The confidence scaling system Through formula The calculation yielded, where This is a preset positive sensitivity coefficient.

[0007] Optionally, determining the flatness of the elliptical Gaussian kernel based on the cosine of the angle between the three-dimensional direction vector and the camera's optical center line-of-sight vector includes: The orientation vector of the limbs connecting the key points in three-dimensional space. Projecting onto the image plane yields a two-dimensional projection vector. ,by The direction of this direction serves as the major axis direction of the elliptical Gaussian kernel; calculate The line-of-sight vector from the camera's optical center to the keypoint The included angle The minor axis of the elliptical Gaussian kernel With long axis The length ratio, i.e., the flatness, is .

[0008] Optionally, the vector at any position in the vector field is obtained by nonlinearly combining the reference direction vector connecting two key points with the gradient field vector at that position, including: set up For any point on the path connecting two key points, start from the key point. Pointing to key points The unit reference direction vector is , The distance transformation gradient field vector at the point is ; exist The last vector at the point Through formula The weights are obtained by combining them. The calculation method is as follows , for Euclidean distance from the point to the human region boundary, is a preset scale parameter.

[0009] Optionally, the path attenuation coefficient is determined by an occlusion weight corresponding to a semantic category of a non-human target and a normalized position of an occlusion segment on the candidate path, and includes: a fixed occlusion weight is preset for the semantic category of the non-human target , and the value range is [0, 1]; the normalized position of the midpoint of the occlusion segment on the candidate path , wherein 0 represents the starting point of the path, and 1 represents the end point of the path; The path attenuation coefficient λ is calculated by the formula .

[0010] Optionally, the motion trajectory of the specified teaching core limb key point is extracted, and the time third derivative of the motion trajectory is calculated as a motion smoothness index, including: The specified teaching core limb key point is the left wrist and right wrist key point; For a three-dimensional space coordinate sequence of the wrist key point , the third derivative of the sequence is calculated using the central difference method : , wherein is the frame sampling time interval; The motion smoothness index is obtained by time averaging the impulse vector amplitude on the entire time sequence.

[0011] Optionally, two functionally coupled key points are selected, and short-time Fourier transform is performed on the motion trajectories of the two points respectively, and the phase consistency of the two points in the main motion frequency band is calculated as a limb coordination index, including: The two functionally coupled key points are the left wrist and right wrist key points; Short-time Fourier transform is performed on the three-dimensional motion trajectories of the two wrist key points respectively, the power spectral density is estimated, the upper limit of the frequency at which the cumulative energy reaches 90% of the total energy from zero hertz is selected , and the main motion frequency band is defined as ; In the main motion frequency band, the cross-spectral density and the power spectral density of the motion trajectories of the two wrists are calculated , and the limb coordination index is obtained by calculating the frequency average value of the amplitude squared coherence function in the main motion frequency band.

[0012] The application also provides a visual analysis-based practical operation process stability evaluation device, comprising the following modules. A preprocessing module is configured to acquire a video image sequence of a teacher's practical operation process, generate a key point heat map of each key point in a preset human body key point set for each frame image in the sequence, and construct a two-dimensional human body skeleton pose based on each limb connection in a preset limb connection set; and perform three-dimensional reconstruction on a two-dimensional human body skeleton pose sequence of consecutive frames to obtain a three-dimensional spatial coordinate sequence of each key point. An evaluation module is configured to extract a motion trajectory of a specified teaching core limb key point, calculate a time third-order derivative of the motion trajectory as a motion smoothness index; select two functionally coupled key points, perform short-time Fourier transform on the motion trajectories of the two key points respectively, calculate a phase consistency of the two key points in a main motion frequency band as a limb coordination degree index; and fuse the motion smoothness index and the limb coordination degree index through a nonlinear mapping function to obtain a comprehensive stability evaluation score of the teacher's practical operation process.

[0013] The application adopts confidence scaling of motion state prediction residual and an elliptical Gaussian kernel based on a three-dimensional pose, improves the generation quality of the human body key point heat map, in the aspect of constructing a human body skeleton, combines instance segmentation to identify non-human occlusions, uses a boundary gradient field of a human body region to guide the generation of a limb connection vector field, simultaneously applies semantic-based attenuation to a candidate path that penetrates through an occlusion, enhances the stability of the limb connection in a complex occlusion environment, and ensures the accurate construction of the two-dimensional pose. In the stability evaluation stage, the application uses a time third-order derivative of a motion trajectory as a smoothness index, calculates a phase consistency of a key limb in a main motion frequency band to calculate a coordination degree, and provides an objective and accurate comprehensive evaluation result of the practical operation process stability. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A schematic diagram for vector field generation Figure 2 A schematic diagram for confidence calculation of a candidate path considering occlusions Figure 3 A schematic diagram for motion smoothness evaluation Figure 4 A schematic diagram for limb coordination degree evaluation DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0016] The first embodiment is a visual analysis-based practical operation process stability evaluation method, comprising the following steps: Step one, obtaining a video image sequence of the teacher's practical operation process, for each frame image in the sequence; Step two, generating a key point heat map for each key point in the preset human body key point set: S21, maintaining a motion state prediction model for each key point, determining a confidence scaling coefficient of the heat map Gaussian peak based on the state prediction residual between the predicted value and the actual observed value of the current frame key point position based on the motion state prediction model, the greater the state prediction residual, the smaller the confidence scaling coefficient; The motion state prediction model preferably uses a Kalman filter. For each key point, an independent Kalman filter is established, and the state vector contains the two-dimensional position and velocity of the key point in the image plane. In an optional embodiment, the motion state vector contains the position, velocity and acceleration of the key point in the image plane. The actual observed value of the current frame is obtained by a convolutional neural network, such as HRNet. The state prediction residual is calculated as the Mahalanobis distance between the predicted state of the Kalman filter and the actual observed value. The confidence scaling coefficient is generated by an exponential decay function from the Mahalanobis distance, for example, the coefficient is equal to , d is the square of the Mahalanobis distance, where is a preset constant.

[0017] S22, based on the three-dimensional pose reconstruction result of the previous frame, calculating the direction vector of the limb connected to the key point in the three-dimensional space, determining the rotation principal axis of the elliptical Gaussian kernel according to the projection of the direction vector on the image plane, and determining the flatness of the elliptical Gaussian kernel according to the cosine of the angle between the three-dimensional direction vector and the camera optical center sight vector; For the wrist key point, the connected limb is the forearm, and the three-dimensional direction vector is obtained by subtracting the three-dimensional coordinates of the elbow and wrist in the previous frame three-dimensional pose. The three-dimensional direction vector is projected onto the current image plane using the camera intrinsic matrix to obtain a two-dimensional vector, and the direction of the two-dimensional vector is the direction of the long axis of the elliptical Gaussian kernel. The camera optical center sight vector refers to the vector from the camera optical center to the three-dimensional position of the key point. Calculate the absolute value of the cosine of the angle between the limb three-dimensional direction vector and the sight vector. When the value is close to 1, it indicates that the limb is facing or facing away from the camera, and the maximum perspective shortening occurs, and the flatness is set to the maximum value at this time; when the value is close to 0, it indicates that the limb is parallel to the image plane, and there is no shortening, and the flatness is set to 0 at this time, and the Gaussian kernel is circular.

[0018] In an optional embodiment, the flatness of the elliptical Gaussian kernel is determined according to the cosine of the angle between the three-dimensional direction vector and the camera optical center sight vector, comprising: a direction vector of the limb connecting the key points in three-dimensional space projected to the image plane to obtain a two-dimensional projection vector , the direction of the elliptical Gaussian kernel as the long axis direction; calculate the included angle θ between the three-dimensional direction vector of the limb and the line-of-sight vector from the camera optical center to the key point , the short axis of the elliptical Gaussian kernel is , and the length ratio of the long axis is the flatness .

[0019] In human pose estimation, a region needs to be drawn on a heat map to represent a limb part, such as a forearm. The present embodiment uses an elliptical Gaussian kernel whose shape and direction can reflect the actual orientation of the forearm in three-dimensional space and the morphological changes caused by different viewing angles. The direction vector of the forearm in three-dimensional space, i.e., the vector from the elbow to the wrist , is determined. This three-dimensional vector is projected onto the two-dimensional image plane of the camera to obtain a two-dimensional vector . The direction of this two-dimensional vector is used as the direction of the long axis of the elliptical Gaussian kernel, so that the elliptical shape on the heat map is completely aligned with the orientation of the limb seen in the image.

[0020] To determine the flatness of the ellipse, i.e., the flatness, the included angle θ between the three-dimensional direction vector of the limb and the line-of-sight vector of the camera is calculated. The line-of-sight vector refers to the vector from the camera optical center to the key point of the limb. The included angle θ reflects the degree of inclination of the limb relative to the viewing angle of the observer. For example, when the user stretches the arm completely straight and points it sideways, the is approximately perpendicular to the , and the included angle θ is close to 90 degrees, with a cosine value close to 0. According to the setting, the length ratio of the short axis to the long axis of the ellipse is |cos(θ)|, i.e., close to 0, which will generate a very elongated ellipse, a complete length of the arm seen in the image.

[0021] On the contrary, if the user stretches the arm straight and points it at the camera, is almost parallel to the , and the included angle θ is close to 0 degrees, with a cosine value close to 1. At this time, the length ratio of the short axis to the long axis is close to 1, and the generated ellipse is very close to a circle. This also conforms to visual observation, because the arm directly facing the camera will appear shorter and more circular in cross-section due to perspective shortening. The generated limb representation can reflect the three-dimensional space pose.

[0022] S23, combine the confidence scaling coefficient and the elliptical Gaussian kernel to generate a heat map of the key point.

[0023] Centered at the predicted position of the keypoint, a two-dimensional elliptical Gaussian distribution function is constructed using the rotation axis and the flattening determined in step S22. The peak of the Gaussian distribution, i.e. the maximum amplitude of the Gaussian function, is scaled by the confidence score determined in step S21. The value of the scaled elliptical Gaussian function at each pixel is calculated on the heatmap channel, completing the generation of the keypoint response heatmap.

[0024] Step three, construct a two-dimensional human skeleton pose based on each limb connection in the preset limb connection set: S31, perform instance segmentation on the current frame image to obtain a human region mask and pixel masks and semantic categories of non-human targets; An instance segmentation model, such as a Mask R-CNN model, is used to process each frame of image. At the same time, the positions, class labels and pixel-level masks of all targets in the image are detected. Thus, a human region mask belonging to the teacher's body contour, and pixel masks and corresponding semantic category labels of non-human targets such as operation table, beaker and computer screen are obtained.

[0025] S32, based on the human region mask, calculate the distance transform of the pixels in the human region to the region boundary, and obtain the gradient field of the distance transform; The human region mask obtained in step S31 is binarized. The Euclidean distance transform algorithm is applied to calculate the distance from each pixel in the mask to the nearest background pixel, i.e. the human boundary, to generate a distance transform map. The partial derivatives of the distance transform map in the x and y directions are calculated using a sobel operator, etc., to obtain a two-dimensional gradient vector field. The vector at each position in the gradient vector field points to the direction in which the distance increases the fastest, i.e. from the human boundary to the inside of the body.

[0026] S33, generate a vector field between the two key points constituting the limb connection, wherein the vector at any position in the vector field is obtained by nonlinearly combining the reference direction vector connecting the two key points and the gradient field vector at the position; The reference direction vector is a unit vector pointing from the starting key point to the ending key point of the limb connection, e.g. from the shoulder to the elbow. In the rectangular region between the two key points, the final vector at any pixel point is obtained by weighted summation of the reference direction vector and the gradient field vector at the point. The weight can be preset, e.g. the final vector is equal to 0.7 times the reference vector plus 0.3 times the gradient vector, so that the final vector field tends to remain in the center of the human region, avoiding deviation.

[0027] S34, line integral of the vector field along the candidate path to calculate the candidate limb confidence connecting two key points, when the candidate path passes through the non-human body target pixel mask, the integral value of the path segment is multiplied by the path attenuation coefficient, which is determined by the occlusion weight corresponding to the semantic category of the non-human body target and the normalized position of the occlusion segment on the candidate path; The candidate path is a straight line segment connecting two candidate key points. The line integral is calculated by sampling points equidistantly on the line segment, calculating the dot product of the vector field at each point and the direction of the line segment, and summing up. When the sampling point falls into the detected non-human body target mask, such as the mask of an operating table, the corresponding occlusion weight of the operating table is obtained by consulting the preset weight table, such as 0.8. At the same time, the normalized position of the sampling point on the line segment is calculated, such as the midpoint position of 0.5. The path attenuation coefficient is obtained by multiplying the occlusion weight by a Gaussian function centered at 0.5. The dot product value at the sampling point needs to be multiplied by 1 minus the path attenuation coefficient before accumulation.

[0028] S35, based on the confidence of all limb connections, complete the two-dimensional human skeleton pose construction of the current frame.

[0029] All detected key points are regarded as nodes of a graph, and all calculated candidate limb connection confidences are regarded as weights of edges of the graph. A greedy algorithm is used to select the highest confidence connection for each type of limb connection, such as shoulder to elbow, from all candidate connections, and ensure that no two limb connections share the same key point, thereby gradually constructing a complete single-person two-dimensional skeleton pose with the highest total confidence.

[0030] In an optional embodiment, the vector at any position in the vector field is obtained by nonlinear combination of the reference direction vector connecting two key points and the gradient field vector at the position, including: Let be any point on the line path connecting two key points, and be the key point . The unit reference direction vector pointing to the key point is . The distance transform gradient field vector at the point is . The final vector at the point is obtained by combination according to the formula , where the weight is calculated according to . is the Euclidean distance from the point to the boundary of the human body region, is a preset scale parameter.

[0031] For example, the key points connecting the left shoulder and the left elbow, the traditional part affinity field will generate a vector field with a uniform direction from the left shoulder to the left elbow on all the pixels in the left arm region. But in the multi-person interaction or the limbs crossing scene, it is easy to connect someone's shoulder to another person's elbow by mistake. The embodiment optimizes the vector field by fusing two kinds of information. The first kind is the reference direction vector v, which is the unit vector from the left shoulder to the left elbow, providing the overall direction of the connection. The second kind of information comes from the distance transform of the human body segmentation mask. First, a binary image is generated, where the pixels belonging to the human body are 1 and the background is 0. Then the distance transform is calculated, so that the value of each pixel p in the human body region is equal to the distance from it to the nearest body contour boundary . The gradient g(p) of the distance field is a vector whose direction always points to the centermost position of the limb region, as shown in Figure 1 , which can be understood as pointing to the center axis of the limb. This ensures that the final vector field is limited within the human body region and away from its boundary.

[0032] The final vector L(p) is obtained by fusing the reference vector v and the gradient vector g(p) through a nonlinear combination formula. The fusion weight w is determined by the distance of the point p to the body boundary. For example, if the point p is very close to the edge of the arm, is small, the weight w will be close to 1, and the final vector L(p) will be mainly determined by the gradient vector g(p) pointing to the center of the limb, pulling the vector field back to the inside of the limb. If the point p is in the central region of the arm, is large, the weight w will be close to 0, and the final vector L(p) will be mainly determined by the reference vector v, maintaining the correct direction from the shoulder to the elbow. So that the generated vector field has both the correct connection direction and is constrained within the target limb.

[0033] In an optional embodiment, the path attenuation coefficient is determined by the occlusion weight corresponding to the semantic category of the non-human body target and the normalized position of the occlusion section on the candidate path, comprising: preset fixed occlusion weights for the semantic categories of the non-human body targets , with a value range of [0, 1]; calculate the normalized position of the midpoint of the occlusion section on the candidate path , wherein 0 represents the starting point of the path and 1 represents the end point of the path; The path attenuation coefficient λ is calculated by the formula .

[0034] In assessing the reliability of the connection between two key points, an intelligent penalty mechanism for the occlusions on the path is introduced. In scenarios such as chemical experiments, the teacher's arm is likely to be occluded by objects such as operating tables, beakers, or test tubes. First, semantic segmentation technology is used to identify these non-human targets in the image, and a preset occlusion weight is assigned to each category . For example, a completely opaque operating table weight may be 0.9, while a semi-transparent glass beaker weight is 0.4, and a small test tube weight is only 0.2. This represents the degree of influence of this object on limb visibility.

[0035] When considering a candidate connection path from the elbow to the wrist, it checks whether the path passes through an identified occlusion. If the path passes through an area identified as a beaker, the relative position of the center point of this occlusion section on the entire path is calculated . The position is normalized to between 0 and 1, with 0 representing the elbow and 1 representing the wrist, as shown in Figure 2 . For example, if the beaker is exactly occluding the center of the forearm, the value of is 0.5.

[0036] Substituting 0.5 and the weight of the beaker 0.4, the final attenuation coefficient is 0.6. A lower coefficient value represents a large penalty on the score of the path. If the occlusion occurs near the wrist, such as 0.9, the final attenuation coefficient is approximately 0.856, and the penalty is much smaller.

[0037] Step four, three-dimensional reconstruction is performed on the two-dimensional human skeleton pose sequence of continuous frames to obtain a three-dimensional spatial coordinate sequence of each key point; A two-dimensional to three-dimensional pose lifting model based on a time convolution network (TCN), such as the VideoPose3D model, is used. By learning the mapping relationship between the motion pattern of the two-dimensional pose in the time sequence and the three-dimensional spatial structure, the three-dimensional spatial coordinates of each human key point in each frame are regressed, and a continuous three-dimensional spatial coordinate sequence is obtained.

[0038] Step five, the motion trajectory of the specified teaching core limb key point is extracted, and the time third derivative of the motion trajectory is calculated as a motion smoothness indicator; two functionally coupled key points are selected, and short-time Fourier transform is performed on the motion trajectories of the two key points, and the phase consistency of the two in the main motion frequency band is calculated as a limb coordination degree indicator; through a nonlinear mapping function, the motion smoothness indicator and the limb coordination degree indicator are fused to obtain a comprehensive stability evaluation score of the teacher's actual operation process.

[0039] The right wrist key point is selected as the core key point, and its motion trajectory is a time series of three-dimensional coordinates. The central difference method is used to calculate the velocity sequence from the position sequence, and then the central difference method is used to calculate the acceleration sequence from the velocity sequence. The jerk, i.e., the time third derivative sequence, is calculated from the acceleration sequence using the central difference method again. The motion smoothness index is finally defined as the integral of the jerk vector modulus length over the entire time period. The integral is approximated by summing the square of the jerk modulus length at all time points and then taking the square root. The left wrist and right wrist are selected as the functionally coupled key points. The time series of the velocity magnitude, i.e., the speed, of the two key points is applied to the short-time Fourier transform, respectively, to obtain the respective time-frequency spectrum. The power spectrum is obtained by integrating the time-frequency spectrum in the time dimension to find the frequency range with the maximum power as the main motion frequency band. In this frequency band, the phase difference at each time point is calculated, and then the modulus of the exponential mean of the phase difference at all time points is calculated. This value is the phase locking value PLV, which is the limb coordination index.

[0040] The nonlinear mapping function is preferably a multilayer perceptron MLP. The multilayer perceptron has two input nodes, several hidden layers, and an output node. The motion smoothness index and the limb coordination index obtained are normalized and input into the MLP. The output of the MLP is a comprehensive stability evaluation score ranging from 0 to 1.

[0041] In an optional embodiment, the extraction of the motion trajectory of the specified teaching core limb key point and the calculation of the time third derivative of the motion trajectory as the motion smoothness index comprises: The specified teaching core limb key point is the left wrist and right wrist key point; For the three-dimensional space coordinate sequence of the wrist key point , the third derivative of the sequence is calculated using the central difference method : , wherein is the frame sampling time interval; The motion smoothness index is obtained by time averaging the impulse vector amplitude over the entire time sequence.

[0042] Both hands are the key to operation, and the core of attention is also the hands. Therefore, the left wrist and right wrist are selected as the analysis objects. In the entire teaching video, the three-dimensional space coordinates of the left and right wrists in each frame are tracked and recorded to form two time sequence trajectories P(t). The smoothness of the motion is measured by calculating the impulse of the trajectory, i.e., the -order derivative of the position with respect to time. The impulse represents the rate of change of acceleration, and a high impulse value means that the motion has abrupt starts and stops or direction mutations, which are manifested as incoherence or jitter, Figure 3The expert's trajectory is smooth and has small jerk, while the novice's trajectory is jittery, has large jerk and is multi-peak. In order to calculate the impulse from the discrete video frame data, the embodiment adopts a numerical calculation method of central difference. For any time t of a frame, in order to calculate the impulse J(t) of the point, the coordinate data of the two frames before and after the time are needed, that is, the coordinates of the time , , and . Wherein Δt is the frame interval of the video, for example, for a video of 30 frames per second, is about 0.033 seconds. By substituting the coordinate values of the five points into the given formula, the impulse vector at time t can be estimated. The above process is performed for each point on the trajectory, thereby obtaining a sequence of impulse vectors varying with time.

[0043] In order to obtain the smoothness score, the magnitude of the impulse vector at each time is calculated, that is, its size, and the magnitude values of all times are averaged in time. For example, when an expert performs titration operation, the hand movement is stable and coherent, and the calculated average impulse amplitude may be a low value, such as 5.2. When a beginner performs the same operation due to nervousness and inexperience, the hand may have slight tremor and hesitation, resulting in a significantly higher calculated average impulse amplitude, such as 28.5. By comparing the quantitative indicators, the smoothness level of the operator's movement can be evaluated.

[0044] In an optional embodiment, the two functionally coupled key points are selected, and short-time Fourier transform is performed on the motion trajectories of the points, respectively, to calculate the phase consistency of the two in the main motion frequency band as the limb coordination degree index, including: The two functionally coupled key points are left wrist and right wrist key points; Short-time Fourier transform is performed on the three-dimensional motion trajectories of the two wrist key points respectively, the power spectral density is estimated, and the upper limit of the frequency at which the cumulative energy reaches 90% of the total energy from zero hertz is selected , the main motion frequency band is defined as ; In the main motion frequency band, the cross-spectral density of the two wrist motion trajectories and the power spectral density , of each are calculated, and the limb coordination degree index is obtained by calculating the frequency average value of the amplitude squared coherence function in the main motion frequency band.

[0045] Using the above example, the left and right wrists are chosen as the keypoint pair for functional coupling. The motion trajectories of the two wrists in 3D space over time are obtained by the pose estimation algorithm. Short-time Fourier transform is applied to each trajectory to convert the time-domain motion signal to the frequency domain, and analyze the frequency components contained in the motion. By analyzing the power spectral density obtained after the transform, it is determined which frequencies are the main motion frequencies of the action. For example, for an action of shaking a beaker back and forth, the main energy may be concentrated around 1 Hz. The energy will accumulate from 0 Hz until an upper frequency limit is found , so that the energy in this frequency band accounts for 90% of the total motion energy. This frequency band, such as 0 to 2.5 Hz, is the main motion frequency band, and subsequent analysis will focus on this interval to ignore high-frequency noise and insignificant small jitter.

[0046] After determining the main motion frequency band, the index of the amplitude squared coherence function is further calculated. The value of this function at each frequency f is between 0 and 1, and is used to measure the linear correlation and phase stability of the motion of the two wrists at that frequency. If the two wrists move at the same frequency and with a fixed phase difference, for example, one hand holds the test tube steadily and the other hand performs the drop, their coherence value at that motion frequency will be close to 1. If the two wrists move independently, the coherence value will be close to 0, as shown in Figure 4 . The coherence values in the main motion frequency band are averaged in frequency to obtain an overall limb coordination index. A coordination score of 0.9 indicates that the two hands are in perfect coordination, while a score below 0.4 may indicate that the two hands are in disarray or not coordinated.

[0047] In an optional embodiment, the teacher's comprehensive stability evaluation score in the actual operation process is obtained by fusing the motion smoothness index and the limb coordination index through a nonlinear mapping function, including: inverse operation and normalization processing is performed on the motion smoothness index, so that its value range is mapped to the interval [0, 1], and the larger the value, the higher the smoothness; the normalized motion smoothness index and the limb coordination index are weighted and summed; the weighted sum is input to a Sigmoid function to obtain the comprehensive stability evaluation score in the range (0, 1).

[0048] ​The original motion smoothness indicator is pre-processed. The original average impulse value is the smaller the better, and it is converted into the larger the better form for easy fusion. For example, by taking the reciprocal, that is, 1 divided by the average impulse value. Then, the converted value is normalized, so that its range is scaled to 0 to 1. For example, by recording a large number of samples, the smoothness score range of experts and novices is determined, and then linear mapping is performed so that the highest level of smoothness corresponds to 1 and the lowest level corresponds to 0. The value range of the limb coordination degree indicator itself is between 0 and 1, and the larger the better, without conversion.

[0049] The normalized smoothness score and the coordination score C are weighted and summed. The allocation of weights can be adjusted according to the nature of the specific teaching task, which is not specifically limited by the present application. For example, for a task that emphasizes fine single-hand operation, the weight of smoothness can be set to 0.7, and the weight of coordination to 0.3. For a task that requires close cooperation of both hands, the weights can be set to smoothness 0.5 and coordination 0.5. Assuming that the weights are each 0.5, the weighted sum is In order to make the final score have better interpretability and fall within a fixed interval, the weighted sum is input into a Sigmoid function. The Sigmoid function can map any real number to the open interval (0, 1), and the shape is similar to an S-shaped curve, which can nonlinearly compress the input value. For example, the weighted sum 2.0 may get a comprehensive stability score close to 0.88 through the Sigmoid function, while the weighted sum -1 may get a score of 0.27.

[0050] The second embodiment is a visual analysis-based practical operation process stability evaluation device, comprising the following modules: A preprocessing module for obtaining a video image sequence of the teacher's practical operation process, generating a key point heat map for each key point in a preset human key point set for each frame image in the sequence, and constructing a two-dimensional human skeleton pose based on each limb connection in a preset limb connection set; performing three-dimensional reconstruction on the sequence of two-dimensional human skeleton poses of consecutive frames to obtain a sequence of three-dimensional spatial coordinates of each key point; An evaluation module for extracting the motion trajectory of the specified teaching core limb key point, calculating the time third derivative of the motion trajectory as a motion smoothness indicator; selecting two functionally coupled key points, performing short-time Fourier transform on the motion trajectories of the two key points respectively, and calculating the phase consistency of the two in the main motion frequency band as a limb coordination degree indicator; and fusing the motion smoothness indicator and the limb coordination degree indicator through a nonlinear mapping function to obtain a comprehensive stability evaluation score of the teacher's practical operation process.

[0051] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code.

[0052] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing machine, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in one or more of the flowchart and / or block diagrams. Figure 1 one or more functions specified in one or more of the flowchart and / or block diagrams.

[0053] The embodiments of the present application described above are intended to be merely exemplary and those skilled in the art will readily suggest modifications and variations to the specific embodiments. Any and all modifications, variations or equivalent arrangements that are within the spirit and scope of the underlying principles should be considered within the scope of the claims.

Claims

1. A method for evaluating stability of a practical operation process based on visual analysis, characterized in that, The method comprises the following steps: obtaining a video image sequence of a teacher's operation process, for each frame image in the sequence, generating a key point heat map of each key point in a preset human body key point set, and constructing a two-dimensional human body skeleton pose based on each limb connection in a preset limb connection set; three-dimensional reconstruction is performed on the two-dimensional human body skeleton pose sequence of the continuous frames to obtain a three-dimensional spatial coordinate sequence of each key point; extracting the motion trajectory of a specified teaching core limb key point, calculating the time third derivative of the motion trajectory as a motion smoothness index, selecting two functionally coupled key points, performing short-time Fourier transform on the motion trajectories of the two key points respectively, and calculating the phase consistency of the two key points in the main motion frequency band as a limb coordination index; through a nonlinear mapping function, the motion smoothness index and the limb coordination index are fused to obtain a comprehensive stability evaluation score of the teacher's operation process.

2. The method of claim 1, wherein, The generation of the key point heat map of each key point in the preset human body key point set specifically includes: maintaining a motion state prediction model for each key point, determining a confidence scaling coefficient of the heat map Gaussian peak based on the state prediction residual between the predicted value and the actual observed value of the current frame key point position based on the motion state prediction model, and the greater the state prediction residual, the smaller the confidence scaling coefficient; based on the three-dimensional pose reconstruction result of the previous frame, calculating the direction vector of the limb connecting the key points in the three-dimensional space, determining the rotation principal axis of the elliptical Gaussian kernel according to the projection of the direction vector on the image plane, and determining the flatness of the elliptical Gaussian kernel according to the cosine of the angle between the three-dimensional direction vector and the camera optical center sight vector; combining the confidence scaling coefficient and the elliptical Gaussian kernel to generate the heat map of the key point.

3. The method of claim 1, wherein, The construction of the two-dimensional human body skeleton pose based on each limb connection in the preset limb connection set specifically includes: performing instance segmentation on the current frame image to obtain a human body region mask and a pixel mask and semantic category of a non-human body target; based on the human body region mask, calculating the distance transform of the pixels in the human body region to the region boundary, and obtaining the gradient field of the distance transform; generating a vector field between the two key points constituting the limb connection, wherein the vector at any position in the vector field is obtained by nonlinearly combining the reference direction vector connecting the two key points and the gradient field vector at the position; line integration is performed on the vector field along the candidate path to calculate the candidate limb confidence of connecting the two key points, when the candidate path passes through the pixel mask of the non-human body target, the integral value of the path section is multiplied by a path attenuation coefficient, and the path attenuation coefficient is determined by the occlusion weight corresponding to the semantic category of the non-human body target and the normalized position of the occlusion section on the candidate path; based on the confidence of all limb connections, the two-dimensional human body skeleton pose construction of the current frame is completed.

4. The method of claim 2, wherein, The maintenance of a motion state prediction model for each key point, the determination of a confidence scaling coefficient of the heat map Gaussian peak based on the state prediction residual between the predicted value and the actual observed value of the current frame key point position based on the motion state prediction model, includes: The motion state prediction model is a Kalman filter, and the motion state vector includes positions, velocities and accelerations of the key points in the image plane. The state prediction residual is a quadratic form of the inverse of the innovation covariance matrix in the Kalman filter update step and the innovation vector, i.e., the square of the Mahalanobis distance between the predicted and observed locations ; The confidence scaling coefficient is calculated by the formula wherein is a preset sensitivity positive coefficient.

5. The method of claim 2, wherein, The flatness of the elliptical Gaussian kernel is determined according to a cosine of an angle between the three-dimensional direction vector and a line-of-sight vector of a camera optical center, and includes: a direction vector of the limb connecting the key points in a three-dimensional space projected to an image plane to obtain a two-dimensional projection vector , the direction of as the long axis direction of the elliptical Gaussian kernel calculate The line-of-sight vector from the camera's optical center to the keypoint The included angle The minor axis of the elliptical Gaussian kernel With long axis The length ratio, i.e., the flatness, is .

6. The method of claim 3, wherein, The vector at any position in the vector field is obtained by nonlinearly combining a reference direction vector connecting two key points and a gradient field vector at the position, and includes: Let be the unit reference direction vector of the key point , , , the distance transform gradient field vector at the point ; At the last vector at the point is combined by formula , where the weight is calculated as , is the Euclidean distance from the point to the boundary of the human body region, is a preset scale parameter.

7. The method of claim 3, wherein, The path attenuation coefficient is determined by a shielding weight corresponding to a semantic category of the non-human target and a normalized position of the shielding section on the candidate path, and includes: A fixed occlusion weight is preset for a semantic category of a non-human target , and the value range is [0, 1] calculating a normalized position of the midpoint of the occlusion segment on the candidate path wherein 0 represents the start of the path and 1 represents the end of the path. The path attenuation coefficient λ is calculated by the formula .

8. The method of claim 1, wherein, The motion trajectory of the specified teaching core limb key point is extracted, and a time third-order derivative of the motion trajectory is calculated as a motion smoothness index, and includes: The specified teaching core limb key point is a left wrist and a right wrist key point. Three-dimensional spatial coordinate sequence of key points of the wrist The third derivative of the sequence is calculated using the central difference method. : ,in The frame sampling time interval; The motion smoothness indicator is obtained by time-averaging the impulse vector magnitude over the entire time series ​ 9. The method of claim 1, wherein, Two functionally coupled key points are selected, short-time Fourier transforms of motion trajectories of the two key points are calculated, phase consistency of the two key points in a main motion frequency band is calculated as a limb coordination index, and includes: The two functionally coupled key points are a left wrist and a right wrist key point. The short-time Fourier transform is performed on the three-dimensional motion trajectories of the two wrist key points respectively, power spectral density is estimated, and an upper limit of frequency is selected, which is from zero hertz and accumulates energy of 90% of the total energy , and the main motion frequency band is defined as ; The cross spectral density of the two wrist trajectories is computed in the primary movement frequency band and the respective power spectral densities , The limb coordination index is obtained by computing the frequency average of the amplitude squared coherence function in the primary movement frequency band.

10. A visual analysis-based real operation process stability evaluation device, characterized by, The following modules are included: A preprocessing module is configured to obtain a video image sequence of a teacher's actual operation process, generate a key point heat map of each key point in a preset human key point set for each frame of image in the sequence, and construct a two-dimensional human skeleton pose based on each limb connection in a preset limb connection set; perform three-dimensional reconstruction on a sequence of two-dimensional human skeleton poses of consecutive frames to obtain a sequence of three-dimensional spatial coordinates of each key point; An evaluation module is configured to extract a motion trajectory of a specified teaching core limb key point, and calculate a time third-order derivative of the motion trajectory as a motion smoothness index. Two functionally coupled key points are selected, short-time Fourier transforms of motion trajectories of the two key points are calculated, phase consistency of the two key points in a main motion frequency band is calculated as a limb coordination index, and includes: The motion smoothness index and the limb coordination index are fused through a nonlinear mapping function to obtain a comprehensive stability evaluation score of the teacher's actual operation process.

Images