Method and device for evaluating stability of practical operation process based on visual analysis

Abstract

The application provides a kind of based on visual analysis's practical operation process stability evaluation method and device, specifically related to stability evaluation field, this method includes: extracting each frame image from teacher practical operation video, generating multiple channel heat map for preset key point, based on preset limb connection, the two-dimensional human skeleton posture of each frame is constructed, three-dimensional reconstruction is carried out to continuous two-dimensional posture sequence, obtains the three-dimensional space coordinate sequence of key point, based on three-dimensional space coordinate sequence, calculate comprehensive stability evaluation score, obtain comprehensive stability evaluation score.The application enhances the stability of limb connection under complex occlusion environment, ensures the accurate construction of two-dimensional posture, can provide an objective and accurate comprehensive evaluation result of practical operation process stability.

Description

Technical Field

[0001] This application belongs to the field of stability assessment, and in particular relates to a method and apparatus for assessing the stability of a practical process based on visual analysis. Background Technology

[0002] In applications such as vocational skills training, physical education, and medical surgical demonstrations, evaluating the stability and standardization of teachers' or experts' practical operations is of great significance. This requires not only high-precision human posture data but also in-depth motion quality analysis of posture sequences. Existing motion quality assessment methods typically rely on kinematic features extracted from posture sequences, such as displacement, velocity, or acceleration of key points. However, these simple, low-order kinematic indicators are insufficient to adequately measure the smoothness and fluidity characteristic of expert-level operations. Human posture estimation methods locate key human points such as joints and facial features and construct human skeletal structures for applications in human-computer interaction and action recognition. Mainstream multi-person posture estimation methods mostly adopt a bottom-up paradigm, that is, first detecting all possible human key points in the image, and then assembling the key points into the skeletons of different individuals through learned inter-part association information such as part affinity fields (PAFs). These methods, when generating response maps (heatmaps) of keypoint locations, cannot adjust the confidence of response peaks based on the degree of motion blur of keypoints, leading to decreased positioning accuracy in fast-moving scenes. When human limbs shorten due to perspective effects, isotropic circular Gaussian kernels cannot accurately describe the non-uniform positioning uncertainty of keypoints on the image plane, affecting the accuracy of pose estimation. When associating keypoints to form limbs, PAFs methods are easily affected by occlusion interference from non-human targets in the environment, such as tools, equipment, tables, and chairs, when calculating connection confidence, leading to incorrect limb connections, especially in complex environments such as those used by teachers for practical teaching. The ability to acquire poses in complex occluded environments and integrate motion smoothness and limb coordination for comprehensive evaluation is a pressing issue in the current technological field. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for evaluating the stability of practical operations based on visual analysis, which can solve the problem of inaccurate positioning caused by rapid motion blur and limb perspective shortening, including the following steps:

[0004] The system acquires a video image sequence of the teacher's practical operation process. For each frame of the sequence, it generates a key point heatmap of each key point in a preset human key point set, and constructs a two-dimensional human skeleton posture based on each limb connection in a preset limb connection set. The system then performs three-dimensional reconstruction on the continuous frame sequence of the two-dimensional human skeleton posture to obtain the three-dimensional spatial coordinate sequence of each key point.

[0005] Extract the motion trajectory of key points of the specified core limbs in the teaching process, and calculate the third time derivative of the motion trajectory as a motion smoothness index; select two functionally coupled key points, perform short-time Fourier transform on the motion trajectory of each point, and calculate the phase consistency of the two in the main motion frequency band as a limb coordination index; through a nonlinear mapping function, fuse the motion smoothness index and the limb coordination index to obtain the comprehensive stability evaluation score of the teacher's practical operation process.

[0006] Optionally, generating a keypoint heatmap for each keypoint in a preset set of human body keypoints specifically includes:

[0007] For each key point, maintain a motion state prediction model. Based on the state prediction residual between the predicted value and the actual observed value of the key point position in the current frame, determine the confidence scaling factor of the Gaussian peak in the heat map. The larger the state prediction residual, the smaller the confidence scaling factor.

[0008] Based on the 3D pose reconstruction results of the previous frame, the direction vector of the limb connecting the key points in 3D space is calculated. The rotation principal axis of the elliptical Gaussian kernel is determined according to the projection of the direction vector on the image plane. The flatness of the elliptical Gaussian kernel is determined according to the cosine of the angle between the 3D direction vector and the line-of-sight vector of the camera optical center.

[0009] By combining the confidence scaling factor and the elliptic Gaussian kernel, a heatmap of the key points is generated.

[0010] Optionally, a two-dimensional human skeleton pose is constructed based on each limb connection in a preset set of limb connections, specifically including:

[0011] Perform instance segmentation on the current frame image to obtain the human region mask and the pixel mask and semantic category of non-human targets;

[0012] Based on the human body region mask, calculate the distance transformation from pixels within the human body region to the region boundary, and obtain the gradient field of the distance transformation.

[0013] A vector field is generated between two key points that constitute the limb connection. The vector at any position in the vector field is obtained by nonlinearly combining the reference direction vector connecting the two key points with the gradient field vector at that position.

[0014] Line integrals are performed on the vector field along the candidate path to calculate the confidence of the candidate limb connecting two key points. When the candidate path passes through the pixel mask of the non-human target, the integral value of the path segment is multiplied by the path attenuation coefficient. The path attenuation coefficient is determined by the occlusion weight corresponding to the semantic category of the non-human target and the normalized position of the occlusion segment on the candidate path.

[0015] Based on the confidence of all limb connections, the 2D human skeleton pose of the current frame is constructed.

[0016] Optionally, maintaining a motion state prediction model for each key point, and determining the confidence scaling factor of the Gaussian peak in the heatmap based on the state prediction residual between the predicted value and the actual observed value of the key point position in the current frame by the motion state prediction model, includes:

[0017] The motion state prediction model is a Kalman filter, and the motion state vector includes the position, velocity, and acceleration of the key points in the image plane.

[0018] The state prediction residual is the quadratic form of the inverse of the innovation covariance matrix and the innovation vector in the Kalman filter update step, i.e., the square of the Mahalanobis distance between the predicted and observed positions. ;

[0019] The confidence scaling system Through formula The calculation yielded, where This is a preset positive sensitivity coefficient.

[0020] Optionally, determining the flatness of the elliptical Gaussian kernel based on the cosine of the angle between the three-dimensional direction vector and the camera's optical center line-of-sight vector includes:

[0021] The orientation vector of the limbs connecting the key points in three-dimensional space. Projecting onto the image plane yields a two-dimensional projection vector. ,by The direction of this direction serves as the major axis of the elliptical Gaussian kernel;

[0022] calculate The line-of-sight vector from the camera's optical center to the keypoint The included angle The minor axis of the elliptical Gaussian kernel With long axis The length ratio, i.e., the flatness, is .

[0023] Optionally, the vector at any position in the vector field is obtained by nonlinearly combining the reference direction vector connecting two key points with the gradient field vector at that position, including:

[0024] set up For any point on the path connecting two key points, start from the key point. Pointing to key points The unit reference direction vector is , The distance transformation gradient field vector at the point is ;

[0025] exist The last vector at the point Through formula The weights are obtained by combining them. The calculation method is as follows , for The Euclidean distance from the point to the boundary of the human body region. These are the preset scale parameters.

[0026] Optionally, the path attenuation coefficient is jointly determined by the occlusion weight corresponding to the semantic category of the non-human target and the normalized position of the occluded segment on the candidate path, including:

[0027] Pre-set fixed occlusion weights for the semantic categories of non-human targets. The value range is [0,1];

[0028] Calculate the normalized position of the midpoint of the occluded segment on the candidate path. ,in A value of 0 represents the starting point of the path, and a value of 1 represents the ending point of the path.

[0029] The path attenuation coefficient λ is obtained through the formula calculate.

[0030] Optionally, the step of extracting the motion trajectory of key points of the specified core limbs in the teaching process and calculating the third time derivative of the motion trajectory as a motion smoothness index includes:

[0031] The designated core limb key points for teaching are the key points of the left and right wrists;

[0032] Three-dimensional spatial coordinate sequence of key points of the wrist The third derivative of the sequence is calculated using the central difference method. : ,in The frame sampling time interval;

[0033] The motion smoothness index is determined by the magnitude of the impulse vector over the entire time series. The result is obtained by averaging over time.

[0034] Optionally, the step of selecting two functionally coupled key points, performing short-time Fourier transforms on the motion trajectories of each point, and calculating the phase consistency of the two points in the main motion frequency band as an indicator of limb coordination includes:

[0035] The key points for the coupling of the two functions are the key points of the left wrist and the right wrist.

[0036] Short-time Fourier transforms were performed on the three-dimensional motion trajectories of the key points of both wrists to estimate the power spectral density. The upper frequency limit was selected to accumulate energy to 90% of the total energy, starting from zero Hertz. The main motion frequency band is defined as ;

[0037] Within the main motion frequency band, calculate the cross spectral density of the motion trajectories of both wrists. and their respective power spectral densities , The limb coordination index is calculated by using the squared amplitude coherence function. The average frequency within the main motion frequency band is obtained.

[0038] The present invention also provides a visual analysis-based device for evaluating the stability of a practical process, comprising the following modules:

[0039] The preprocessing module is used to acquire video image sequences of the teacher's practical operation process. For each frame of the sequence, it generates a key point heatmap of each key point in the preset human key point set, and constructs a two-dimensional human skeleton posture based on each limb connection in the preset limb connection set. It then performs three-dimensional reconstruction on the two-dimensional human skeleton posture sequence of consecutive frames to obtain the three-dimensional spatial coordinate sequence of each key point.

[0040] The evaluation module is used to extract the motion trajectory of key points of the specified core limbs in teaching, and calculate the time third derivative of the motion trajectory as a motion smoothness index; select two functionally coupled key points, perform short-time Fourier transform on the motion trajectories of the two key points respectively, and calculate the phase consistency of the two in the main motion frequency band as a limb coordination index; through a nonlinear mapping function, the motion smoothness index and the limb coordination index are fused to obtain a comprehensive stability evaluation score of the teacher's practical operation process.

[0041] This invention improves the quality of human keypoint heatmap generation by employing confidence scaling of motion state prediction residuals and an elliptical Gaussian kernel based on 3D pose. In constructing the human skeleton, it combines instance segmentation to identify non-human occlusions and utilizes the boundary gradient field of the human region to guide the generation of limb connection vector fields. Simultaneously, it applies semantic-based attenuation to candidate paths traversing occlusions, enhancing the stability of limb connections in complex occlusion environments and ensuring accurate 2D pose construction. In the stability evaluation stage, this invention uses the time third derivative of the motion trajectory as a smoothness index and calculates the phase consistency of key limbs in the main motion frequency band to calculate coordination, providing an objective and accurate comprehensive evaluation result of practical process stability. Attached Figure Description

[0042] Figure 1 Schematic diagram of vector field generation

[0043] Figure 2 A schematic diagram illustrating the calculation of candidate path confidence considering occlusion;

[0044] Figure 3 This is a schematic diagram for evaluating motion smoothness.

[0045] Figure 4 This is a diagram illustrating the assessment of limb coordination. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0047] The first embodiment is a method for evaluating the stability of a practical process based on visual analysis, including the following steps:

[0048] Step 1: Obtain the video image sequence of the teacher's practical operation process, and analyze each frame of the sequence;

[0049] Step 2: Generate a key point heatmap for each key point in the preset human body key point set:

[0050] S21, maintain a motion state prediction model for each key point, and determine the confidence scaling factor of the Gaussian peak in the heat map based on the state prediction residual between the predicted value and the actual observed value of the key point position in the current frame by the motion state prediction model. The larger the state prediction residual, the smaller the confidence scaling factor.

[0051] The motion state prediction model preferably employs a Kalman filter. For each keypoint, an independent Kalman filter is established, whose state vector contains the keypoint's two-dimensional position and velocity in the image plane. In an optional embodiment, the motion state vector contains the keypoint's position, velocity, and acceleration in the image plane. The actual observations of the current frame are initially detected using a convolutional neural network, such as HRNet. The state prediction residual is calculated as the Mahalanobis distance between the Kalman filter's predicted state and the actual observation. The confidence scaling factor is generated from this Mahalanobis distance using an exponential decay function; for example, the coefficient equals... d is the squared Mahalanobis distance, where This is a preset constant.

[0052] S22, based on the three-dimensional pose reconstruction result of the previous frame, calculate the direction vector of the limb connecting the key points in three-dimensional space, determine the rotation principal axis of the elliptical Gaussian kernel according to the projection of the direction vector on the image plane, and determine the flatness of the elliptical Gaussian kernel according to the cosine of the angle between the three-dimensional direction vector and the line-of-sight vector of the camera optical center.

[0053] For the wrist keypoint, the connected limb is the forearm, and its 3D direction vector is obtained by subtracting the 3D coordinates of the elbow and wrist from the previous frame's 3D pose. This 3D direction vector is projected onto the current image plane using the camera intrinsic matrix, resulting in a 2D vector. The direction of this 2D vector is the major axis of the elliptical Gaussian kernel. The camera optical center line-of-sight vector is the vector pointing from the camera optical center to the 3D position of this keypoint. The absolute value of the cosine of the angle between the limb's 3D direction vector and the line-of-sight vector is calculated. When this value is close to 1, it indicates that the limb is facing away from the camera, resulting in maximum perspective shortening; in this case, the flatness is set to the maximum value. When this value is close to 0, it indicates that the limb is parallel to the image plane, with no shortening; in this case, the flatness is set to 0, and the Gaussian kernel is circular.

[0054] In an optional embodiment, determining the flatness of the elliptical Gaussian kernel based on the cosine of the angle between the three-dimensional direction vector and the camera's optical center line-of-sight vector includes:

[0055] The orientation vector of the limbs connecting the key points in three-dimensional space. Projecting onto the image plane yields a two-dimensional projection vector. ,by The direction of this direction serves as the major axis of the elliptical Gaussian kernel;

[0056] calculate The line-of-sight vector from the camera's optical center to the keypoint The included angle The minor axis of the elliptical Gaussian kernel With long axis The length ratio, i.e., the flatness, is .

[0057] When performing human pose estimation, a region needs to be plotted on a heatmap to represent limbs such as the forearm. This embodiment uses an elliptical Gaussian kernel, the shape and orientation of which can reflect the actual orientation of the forearm in three-dimensional space and the morphological changes caused by different viewing angles. The direction vector of the forearm in three-dimensional space is determined, that is, the vector from the elbow to the wrist. Projecting this three-dimensional vector onto the camera's two-dimensional image plane yields a two-dimensional vector. The direction of this two-dimensional vector is used as the direction of the major axis of the elliptical Gaussian kernel, so that the elliptical shape on the heatmap is perfectly aligned with the limb orientation seen in the image.

[0058] To determine the flatness of the ellipse, i.e., the degree of flatness, the three-dimensional orientation vectors of the limbs are calculated. With camera gaze vector The angle θ between the two points. The gaze vector is the vector pointing from the camera's optical center to the key points of the limb. The angle θ reflects the degree of tilt of the limb relative to the observer's viewpoint. For example, when a user fully extends their arm and points to the side, at this time... and Roughly perpendicular, with an included angle θ close to 90 degrees, and the remaining chord values ​​close to 0. By setting, the ratio of the minor axis to the major axis of the ellipse is |cos(θ)|, which is close to 0, which will generate a very slender ellipse, with the full length of an arm as seen in the image.

[0059] Conversely, if the user extends their arm and points it at the camera, and With the axes nearly parallel, the included angle θ close to 0 degrees, and the remaining chord values ​​close to 1, the ratio of the minor axis to the major axis is close to 1, resulting in an ellipse that closely resembles a circle. This also aligns with visual observation, as an arm facing the camera will appear shorter and more circular in the image due to perspective shortening. The generated limb representation can reflect the posture in three-dimensional space.

[0060] S23, combine the confidence scaling factor and the elliptical Gaussian kernel to generate a heatmap of the key points.

[0061] Centered on the predicted location of the keypoint, a two-dimensional elliptical Gaussian distribution function is constructed using the principal axis of rotation and flatness determined in step S22. The peak value of this Gaussian distribution, i.e., the maximum amplitude of the Gaussian function, is multiplied by the confidence scaling factor determined in step S21. The value of this scaled elliptical Gaussian function at each pixel is calculated on the heatmap channel, completing the generation of the keypoint response heatmap.

[0062] Step 3: Construct a two-dimensional human skeleton pose based on each limb connection in the preset set of limb connections:

[0063] S31, Perform instance segmentation on the current frame image to obtain the human body region mask and the pixel mask and semantic category of non-human targets;

[0064] An instance segmentation model, such as Mask R-CNN, is used to process each frame of the image. Simultaneously, the location, category label, and pixel-level mask of all targets in the image are detected. This yields the human body region mask belonging to the teacher's body contour, as well as the pixel masks and corresponding semantic category labels of non-human targets such as the workbench, beaker, and computer screen.

[0065] S32, based on the human body region mask, calculate the distance transformation from pixels within the human body region to the region boundary, and obtain the gradient field of the distance transformation;

[0066] The human body region mask obtained in step S31 is binarized. The Euclidean distance transform algorithm is applied to calculate the distance from each pixel within the mask to the nearest background pixel, i.e., the human body boundary, generating a distance transform map. The partial derivatives in the x and y directions of the distance transform map are calculated using operators such as the Sobel operator, resulting in a two-dimensional gradient vector field. The vector at each position in the gradient vector field points in the direction of the fastest increase in distance, i.e., from the human body boundary towards the interior of the body.

[0067] S33, a vector field is generated between two key points that constitute the limb connection. The vector at any position in the vector field is obtained by nonlinearly combining the reference direction vector connecting the two key points with the gradient field vector at that position.

[0068] The reference direction vector is a unit vector pointing from the starting keypoint of a limb connection to the ending keypoint, such as from the shoulder to the elbow. Within a rectangular region between two keypoints, the final vector of any pixel is obtained by weighted summing the reference direction vector and the gradient field vector at that point. The weights can be preset; for example, the final vector is equal to 0.7 times the reference vector plus 0.3 times the gradient vector, so that the final vector field tends to stay centered in the human body region and avoids deviation.

[0069] S34, perform line integration on the vector field along the candidate path to calculate the confidence of the candidate limb connecting the two key points. When the candidate path passes through the pixel mask of the non-human target, multiply the integral value of the path segment by the path attenuation coefficient. The path attenuation coefficient is determined by the occlusion weight corresponding to the semantic category of the non-human target and the normalized position of the occlusion segment on the candidate path.

[0070] A candidate path is a straight line segment connecting two candidate keypoints. The line integral is calculated by sampling points at equal intervals along this line segment, calculating the dot product of the vector field at each point with the direction of the line segment, and summing the results. When a sampling point falls into a detected non-human target mask, such as the mask of an operating console, a preset weight table is consulted to obtain the occlusion weight corresponding to the operating console, such as 0.8. Simultaneously, the normalized position of the sampling point on the line segment is calculated; for example, a position of 0.5 is given if it is located at the midpoint. The path attenuation coefficient is obtained by multiplying a Gaussian function centered at 0.5 with the occlusion weight. The dot product value at the sampling point is accumulated and needs to be multiplied by 1 and subtracted from the path attenuation coefficient.

[0071] S35, based on the confidence of all limb connections, completes the 2D human skeleton pose construction for the current frame.

[0072] All detected keypoints are treated as nodes in a graph, and the calculated confidence scores of all candidate limb connections are used as the weights of the graph edges. A greedy parsing algorithm is employed to select the connection with the highest confidence score from all candidate connections for each type of limb connection, such as shoulder to elbow, while ensuring that no two limb connections share the same keypoint. This process gradually constructs a complete single-person 2D skeletal pose with the highest total confidence score.

[0073] In an optional embodiment, the vector at any position in the vector field is obtained by a nonlinear combination of the reference direction vector connecting two key points and the gradient field vector at that position, including:

[0074] set up For any point on the path connecting two key points, start from the key point. Pointing to key points The unit reference direction vector is , The distance transformation gradient field vector at the point is ;

[0075] exist The last vector at the point Through formula The weights are obtained by combining them. The calculation method is as follows , for The Euclidean distance from the point to the boundary of the human body region. These are the preset scale parameters.

[0076] Taking the key point connecting the left shoulder and left elbow as an example, a traditional part affinity field generates a vector pointing in a uniform direction from the left shoulder to the left elbow across all pixels in the left arm region. However, in multi-person interaction or limb-crossing scenarios, it's easy to mistakenly connect one person's shoulder to another person's elbow. This embodiment optimizes the vector field by fusing two types of information. The first is the reference direction vector v, which is the unit vector pointing from the left shoulder to the left elbow, providing the overall direction of the connection. The second type of information comes from the distance transformation of the human body segmentation mask. First, a binary image is generated, where pixels belonging to the human are 1 and the background is 0. Then, the distance transformation is calculated so that the value of each pixel p within the human body region is equal to its distance to the nearest body contour boundary. The gradient g(p) of the distance field is a vector whose direction always points to the center of the limb region, such as... Figure 1 As shown, this can be understood as pointing towards the central axis of the limb. This ensures that the final vector field is confined within the human body region and away from its boundaries.

[0077] The final vector L(p) is obtained by fusing the baseline vector v and the gradient vector g(p) using a nonlinear combination formula. The fusion weight w is the distance from point p to the body boundary. Decision. For example, if point p is very close to the edge of the arm, When the value is very small, the weight w will be close to 1. In this case, the final vector L(p) will be mainly determined by the gradient vector g(p) pointing towards the center of the limb, pulling the vector field back into the limb. If point p is located in the central region of the arm, The weight w is very large, and the final vector L(p) is mainly determined by the reference vector v, maintaining the correct direction from shoulder to elbow. This ensures that the generated vector field has both the correct connection direction and is constrained within the range of the target limb.

[0078] In an optional embodiment, the path attenuation coefficient is determined jointly by the occlusion weight corresponding to the semantic category of the non-human target and the normalized position of the occluded segment on the candidate path, including:

[0079] Pre-set fixed occlusion weights for the semantic categories of non-human targets. The value range is [0,1];

[0080] Calculate the normalized position of the midpoint of the occluded segment on the candidate path. ,in A value of 0 represents the starting point of the path, and a value of 1 represents the ending point of the path.

[0081] The path attenuation coefficient λ is obtained through the formula calculate.

[0082] When assessing the reliability of the connection between two key points, an intelligent penalty mechanism for occlusions along the path is introduced. In scenarios such as chemical experiments, a teacher's arm is likely to be obscured by objects such as workbenches, beakers, or test tubes. First, semantic segmentation technology is used to identify these non-human targets in the image, and an occlusion weight is preset for each category. For example, a completely opaque workbench might have a weight of 0.9, while a translucent glass beaker might have a weight of 0.4, and a small test tube might have a weight of only 0.2. This indicates the degree to which such objects affect the visibility of limbs.

[0083] When considering a candidate connection path from the elbow to the wrist, it checks whether the path passes through an identified occlusion. If the path passes through a region identified as a beaker, the relative position of the center point of this occluded segment on the entire path is calculated. The position is normalized to between 0 and 1, where 0 represents the elbow and 1 represents the wrist, such as... Figure 2 As shown. For example, if the beaker exactly covers the center of the forearm, then... The value is 0.5.

[0084] Substitution Equal to 0.5 and the weight of the beaker The coefficient is 0.4, resulting in a final attenuation factor of 0.6. A lower coefficient value represents a large penalty for the path score. If the occlusion occurs near the wrist, for example... If the value is 0.9, the final attenuation coefficient will be approximately 0.856, resulting in a much smaller penalty.

[0085] Step 4: Perform 3D reconstruction on the 2D human skeleton pose sequence of consecutive frames to obtain the 3D spatial coordinate sequence of each key point.

[0086] The input consists of a continuous sequence of multiple frames, such as 16 frames, of 2D human skeleton poses. A 2D-to-3D pose enhancement model based on a Temporal Convolutional Network (TCN), such as the VideoPose3D model, is employed. By learning the mapping relationship between the motion patterns of 2D poses over time and the 3D spatial structure, the 3D spatial coordinates of each key point of the human body in each frame are regressed, thereby obtaining a continuous sequence of 3D spatial coordinates.

[0087] Step 5: Extract the motion trajectory of the key points of the designated core limbs in the teaching process, and calculate the time third derivative of the motion trajectory as the motion smoothness index; select two functionally coupled key points, perform short-time Fourier transform on the motion trajectories of the two key points respectively, and calculate the phase consistency of the two in the main motion frequency band as the limb coordination index; through a nonlinear mapping function, fuse the motion smoothness index and the limb coordination index to obtain the comprehensive stability evaluation score of the teacher's practical operation process.

[0088] The right wrist was selected as the core key point, and its motion trajectory was a three-dimensional coordinate time series. The velocity sequence was calculated from the position sequence using the central difference method, and then the acceleration sequence was calculated from the velocity sequence using the central difference method. The jerkiness, i.e., the third derivative sequence of time, was then calculated from the acceleration sequence using the central difference method again. The motion smoothness index was ultimately defined as the integral of the magnitude of the jerkiness vector over the entire time period, approximated by taking the square root of the sum of the squares of the jerkiness magnitudes at all time points. The left and right wrists were selected as key points for functional coupling. Short-time Fourier transforms were applied to the time series of the velocity magnitudes (i.e., rates) of each of the two key points to obtain their respective time spectra. The power spectrum was obtained by integrating the time spectra in the time dimension, and the frequency range with the highest power was identified as the main motion frequency band. Within this frequency band, the phase difference between the two signals at each time point was calculated, and then the magnitude of the exponential mean of the phase difference at all time points was calculated. This value is the phase-locked value (PLV), which is the limb coordination index.

[0089] The nonlinear mapping function is preferably a multilayer perceptron (MLP). This MLP has two input nodes, several hidden layers, and one output node. The obtained motion smoothness and limb coordination indices are normalized and then input into the MLP. The output of the MLP is a comprehensive stability evaluation score ranging from 0 to 1.

[0090] In an optional embodiment, the step of extracting the motion trajectory of a specified core limb key point in the teaching process and calculating the time third derivative of the motion trajectory as a motion smoothness index includes:

[0091] The designated core limb key points for teaching are the key points of the left and right wrists;

[0092] Three-dimensional spatial coordinate sequence of key points of the wrist The third derivative of the sequence is calculated using the central difference method. : ,in The frame sampling time interval;

[0093] The motion smoothness index is determined by the magnitude of the impulse vector over the entire time series. The result is obtained by averaging over time.

[0094] Both hands are crucial to the operation, and therefore the focus is on them; hence, the left and right wrists are chosen as the analysis objects. Throughout the instructional video, the three-dimensional spatial coordinates of the left and right wrists are tracked and recorded in each frame, forming two time-series trajectories P(t). The smoothness of the motion is measured by calculating the impulse of the trajectory, which is the first derivative of position with respect to time. Impulse represents the rate of change of acceleration; a high impulse value indicates abrupt starts or stops or sudden changes in direction, manifesting as discontinuity or jitter. Figure 3 The trajectory of an expert is smooth and has low abruptness, while the trajectory of a novice is jittery, with high abruptness and multiple peaks. To calculate the impulse from discrete video frame data, this embodiment employs a central difference numerical calculation method. For any frame at time t, to calculate the impulse J(t) at that point, the coordinate data of the two frames before and after that time are needed, i.e. , , and The coordinates of time. Where Δt is the frame interval of the video; for example, for a video of 30 frames per second... The time is approximately 0.033 seconds. Substituting the coordinates of these five points into the given formula, the impulse vector at time t can be estimated. This process is performed on every point on the trajectory, resulting in a sequence of impulse vectors that vary over time.

[0095] To obtain a smoothness score, the magnitude of the impulse vector at each moment is calculated, and the magnitude values ​​at all moments are averaged over time. For example, when an expert performs a titration operation, their hand movements are stable and fluid, and the calculated average impulse amplitude might be a lower value, such as 5.2. However, a beginner performing the same operation, due to nervousness and lack of experience, might experience slight hand tremors and hesitation, resulting in a significantly higher calculated average impulse amplitude, such as 28.5. By comparing these quantitative indicators, the smoothness level of the operator's movements can be assessed.

[0096] In an optional embodiment, selecting two functionally coupled key points, performing short-time Fourier transforms on the motion trajectories of each point, and calculating the phase consistency of the two points in the main motion frequency band as an indicator of limb coordination includes:

[0097] The key points for the coupling of the two functions are the key points of the left wrist and the right wrist.

[0098] Short-time Fourier transforms were performed on the three-dimensional motion trajectories of the key points of both wrists to estimate the power spectral density. The upper frequency limit was selected to accumulate energy to 90% of the total energy, starting from zero Hertz. The main motion frequency band is defined as ;

[0099] Within the main motion frequency band, calculate the cross spectral density of the motion trajectories of both wrists. and their respective power spectral densities , The limb coordination index is calculated by using the squared amplitude coherence function. The average frequency within the main motion frequency band is obtained.

[0100] Continuing with the example above, we select the left and right wrists as the key point pair for functional coupling. A pose estimation algorithm is used to obtain the time-varying motion trajectories of both wrists in three-dimensional space. Short-time Fourier transforms are then performed on these two trajectories to convert the time-domain motion signals to the frequency domain, allowing analysis of the various frequency components within the motion. By analyzing the power spectral density obtained after the transformation, we determine which frequencies are the dominant motion frequencies of the action. For example, for the action of shaking a beaker back and forth, the main energy might be concentrated around 1 Hz. Energy will accumulate starting from 0 Hz until an upper frequency limit is found. This makes from 0 to The energy within this frequency band accounts for 90% of the total motion energy. This frequency band, for example, 0 to 2.5 Hz, is the main motion frequency band, and subsequent analysis will focus on this range to ignore high-frequency noise and insignificant micro-jitter.

[0101] After determining the main motion frequency band, the index of the amplitude squared coherence function is further calculated. The value of this function at each frequency f is... A value between 0 and 1 measures the linear correlation and phase stability of wrist movements at that frequency. If the two wrists move in tandem with the same frequency and a fixed phase difference—for example, one hand steadily holds a test tube while the other adds liquid—their coherence value at that frequency will be close to 1. If the wrist movements are unrelated, the coherence value will be close to 0. Figure 4 As shown, the overall limb coordination index is obtained by averaging all coherence values ​​within the main motion frequency band. A coordination score of 0.9 indicates good hand-eye coordination, while a score below 0.4 may indicate chaotic or uncoordinated hand movements.

[0102] In an optional embodiment, the step of fusing the motion smoothness index and the limb coordination index through a nonlinear mapping function to obtain a comprehensive stability evaluation score for the teacher's practical operation includes:

[0103] The motion smoothness index is inversely calculated and normalized to map its value range to the interval [0, 1], and the larger the value, the higher the smoothness.

[0104] The normalized motion smoothness index is weighted and summed with the limb coordination index;

[0105] The weighted sum is fed into the Sigmoid function to obtain the comprehensive stability evaluation score in the range (0, 1).

[0106] The original motion smoothness index undergoes preprocessing. The original average impulse value is better the smaller it is; to facilitate fusion, it is converted to a form where a larger value is better. This can be achieved, for example, by taking the reciprocal, i.e., 1 divided by the average impulse value. Then, the converted value is normalized, scaling its range to between 0 and 1. For example, by recording a large number of samples, the smoothness score ranges for experts and novices are determined, and then a linear mapping is performed so that the highest level of smoothness corresponds to 1, and the lowest level corresponds to 0. The limb coordination index itself has a value range between 0 and 1, and a larger value is better, so no conversion is needed.

[0107] Normalized smoothness score The weighted sum is then calculated using the coordination score C. The weighting can be adjusted based on the nature of the specific teaching task; this invention does not impose specific limitations on this. For example, for a task emphasizing fine motor skills with one hand, the weight for smoothness can be set to 0.7, and the weight for coordination to 0.3. For a task requiring close cooperation of both hands, the weights can be set to smoothness 0.5 and coordination 0.5. Assuming each weight is 0.5, the weighted sum is... To improve the interpretability of the final score and ensure it falls within a fixed interval, the weighted sum is fed into the Sigmoid function. The Sigmoid function maps any real number to an open interval between 0 and 1, resembling an S-shaped curve, and provides non-linear compression of the input value. For example, a weighted sum of 2.0 might yield a comprehensive stability score close to 0.88 through the Sigmoid function, while a weighted sum of -1 might yield a score of 0.27.

[0108] The second embodiment is a practical process stability assessment device based on visual analysis, comprising the following modules:

[0109] The preprocessing module is used to acquire video image sequences of the teacher's practical operation process. For each frame of the sequence, it generates a key point heatmap of each key point in the preset human key point set, and constructs a two-dimensional human skeleton posture based on each limb connection in the preset limb connection set. It then performs three-dimensional reconstruction on the two-dimensional human skeleton posture sequence of consecutive frames to obtain the three-dimensional spatial coordinate sequence of each key point.

[0110] The evaluation module is used to extract the motion trajectory of key points of the specified core limbs in teaching, and calculate the time third derivative of the motion trajectory as a motion smoothness index; select two functionally coupled key points, perform short-time Fourier transform on the motion trajectories of the two key points respectively, and calculate the phase consistency of the two in the main motion frequency band as a limb coordination index; through a nonlinear mapping function, the motion smoothness index and the limb coordination index are fused to obtain a comprehensive stability evaluation score of the teacher's practical operation process.

[0111] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0112] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for evaluating stability of a practical operation process based on visual analysis, characterized in that, The method comprises the following steps: obtaining a video image sequence of a teacher's operation process, for each frame image in the sequence, generating a key point heat map of each key point in a preset human body key point set, and constructing a two-dimensional human body skeleton pose based on each limb connection in a preset limb connection set; three-dimensional reconstruction is performed on the two-dimensional human body skeleton pose sequence of the continuous frames to obtain a three-dimensional spatial coordinate sequence of each key point; extracting the motion trajectory of the specified teaching core limb key point, calculating the time third derivative of the motion trajectory as a motion smoothness index; selecting two functionally coupled key points, respectively performing short-time Fourier transform on the motion trajectory of the points, calculating the phase consistency of the two points in the main motion frequency band as a limb coordination degree index; through a nonlinear mapping function, the motion smoothness index and the limb coordination degree index are fused to obtain a comprehensive stability evaluation score of the teacher's operation process; the extraction of the motion trajectory of the specified teaching core limb key point, the calculation of the time third derivative of the motion trajectory as the motion smoothness index, comprising: the specified teaching core limb key point is the left wrist and right wrist key point; Three-dimensional spatial coordinate sequence of key points of the wrist The third derivative of the sequence is calculated using the central difference method. : ,in The frame sampling time interval; The motion smoothness indicator is obtained by time-averaging the impulse vector magnitude over the entire time series over the entire time series the selection of two functionally coupled key points, respectively performing short-time Fourier transform on the motion trajectory of the points, calculating the phase consistency of the two points in the main motion frequency band as a limb coordination degree index, comprising: the two functionally coupled key points are the left wrist and right wrist key points; The short-time Fourier transform is performed on the three-dimensional motion trajectories of the two wrist key points respectively, power spectral density is estimated, and an upper limit of frequency is selected, which is from zero hertz and accumulates energy of 90% of the total energy , and the main motion frequency band is defined as ; In the primary movement frequency band, the cross spectral density of the two wrist trajectories is calculated and the respective power spectral densities , The limb coordination index is obtained by calculating the frequency average of the amplitude squared coherence function in the primary movement frequency band.

2. The method of claim 1, wherein, the generation of a key point heat map of each key point in a preset human body key point set, specifically comprising: maintaining a motion state prediction model for each key point, determining a confidence scaling coefficient of a heat map Gaussian peak based on the state prediction residual between the predicted value and the actual observed value of the current frame key point position based on the motion state prediction model, the greater the state prediction residual, the smaller the confidence scaling coefficient; based on the three-dimensional pose reconstruction result of the previous frame, calculating the direction vector of the limb connecting the key points in the three-dimensional space, determining the rotation principal axis of the elliptical Gaussian kernel according to the projection of the direction vector in the image plane, and determining the flatness of the elliptical Gaussian kernel according to the cosine of the angle between the three-dimensional direction vector and the camera optical center sight vector; combining the confidence scaling coefficient and the elliptical Gaussian kernel to generate a heat map of the key point.

3. The method of claim 1, wherein, constructing a two-dimensional human body skeleton pose based on each limb connection in a preset limb connection set, specifically comprising: performing instance segmentation on the current frame image to obtain a human body region mask and a pixel mask and semantic category of non-human body targets; based on the human body region mask, calculating the distance transform of the pixels in the human body region to the region boundary, and obtaining the gradient field of the distance transform; generating a vector field between the two key points constituting the limb connection, the vector at any position in the vector field being obtained by nonlinearly combining the reference direction vector connecting the two key points and the gradient field vector at the position; Line integral of the vector field is performed along the candidate path to calculate a candidate limb confidence connecting two key points, when the candidate path passes through the pixel mask of the non-human body target, the integral value of the path segment is multiplied by a path attenuation coefficient, the path attenuation coefficient is determined by the occlusion weight corresponding to the semantic category of the non-human body target and the normalized position of the occlusion segment on the candidate path; Based on the confidence of all limb connections, the two-dimensional human skeleton pose construction of the current frame is completed.

4. The method of claim 2, wherein, The motion state prediction model is maintained for each key point, and based on the state prediction residual between the predicted value and the actual observed value of the key point position in the current frame, a confidence scaling coefficient of the heat map Gaussian peak is determined, including: The motion state prediction model is a Kalman filter, and the motion state vector includes the position, velocity and acceleration of the key point in the image plane; The state prediction residual is a quadratic form of the inverse of the innovation covariance matrix in the Kalman filter update step and the innovation vector, i.e., the square of the Mahalanobis distance between the predicted and observed locations ; The confidence scaling coefficient is calculated by the formula is calculated by the formula is a preset sensitivity positive coefficient.

5. The method of claim 2, wherein, The flatness of the elliptical Gaussian kernel is determined according to the cosine of the angle between the three-dimensional direction vector and the line-of-sight vector of the camera optical center, including: a direction vector of the limb connecting the key points in a three-dimensional space projected to an image plane to obtain a two-dimensional projection vector , the direction of is taken as the long axis direction of the elliptical Gaussian kernel calculate The line-of-sight vector from the camera's optical center to the keypoint The included angle The minor axis of the elliptical Gaussian kernel With long axis The length ratio, i.e., the flatness, is .

6. The method of claim 3, wherein, The vector at any position in the vector field is obtained by nonlinear combination of the reference direction vector connecting two key points and the gradient field vector at this position, including: set up For any point on the path connecting two key points, start from the key point. Pointing to key points The unit reference direction vector is , The distance transformation gradient field vector at the point is ; exist The last vector at the point Through formula The weights are obtained by combining them. The calculation method is as follows , for The Euclidean distance from the point to the boundary of the human body region. These are the preset scale parameters.

7. The method of claim 3, wherein, The path attenuation coefficient is determined by the occlusion weight corresponding to the semantic category of the non-human body target and the normalized position of the occlusion segment on the candidate path, including: A fixed occlusion weight is preset for a semantic category of a non-human target , and the value range is [0, 1] calculating a normalized position of the midpoint of the occlusion segment on the candidate path wherein 0 represents the start of the path and 1 represents the end of the path. The path attenuation coefficient λ is calculated by the formula λ = -ln (1 - P) / L 8. A visual analysis-based actual operation process stability evaluation device characterized by comprising: It includes the following modules: The preprocessing module is used to obtain a video image sequence of the teacher's actual operation process, generate a key point heat map of each key point in a preset human key point set for each frame image in the sequence, and construct a two-dimensional human skeleton pose based on each limb connection in a preset limb connection set; three-dimensional reconstruction is performed on the two-dimensional human skeleton pose sequence of the continuous frames to obtain a three-dimensional spatial coordinate sequence of each key point; The evaluation module is used to extract the motion trajectory of the specified teaching core limb key point, calculate the time third derivative of the motion trajectory as a motion smoothness index; select two functionally coupled key points, respectively perform short-time Fourier transform on the motion trajectory of the points, calculate the phase consistency of the two in the main motion frequency band as a limb coordination degree index; through a nonlinear mapping function, the motion smoothness index and the limb coordination degree index are fused to obtain a comprehensive stability evaluation score of the teacher's actual operation process; The motion trajectory of the specified teaching core limb key point is extracted, and the time third derivative of the motion trajectory is calculated as a motion smoothness index, including: The specified teaching core limb key point is the left wrist and right wrist key point; a sequence of three-dimensional spatial coordinates of wrist key points a third derivative of the sequence is calculated using central difference method : where is a frame sampling time interval; The motion smoothness indicator is obtained by time-averaging the impulse vector magnitude over the entire time series over the entire time series The two functionally coupled key points are selected, and the motion trajectory of the points is respectively subjected to short-time Fourier transform, and the phase consistency of the two in the main motion frequency band is calculated as a limb coordination degree index, including: The two functionally coupled key points are the left wrist and right wrist key points. The short-time Fourier transform is performed on the three-dimensional motion trajectories of the two wrist key points respectively, power spectral density is estimated, and an upper limit of frequency is selected, which is from zero hertz and accumulates energy of 90% of the total energy , and the main motion frequency band is defined as ; The cross spectral density of the two wrist trajectories is computed in the primary movement frequency band and the respective power spectral densities , The limb coordination index is obtained by computing the frequency average of the amplitude squared coherence function in the primary movement frequency band.

Images