A dynamic image key point and key frame positioning method

Abstract

The application designs a dynamic image key point and key frame positioning method, and belongs to the technical field of computer aided diagnosis; firstly, a dynamic image sequence containing a to-be-measured target is pretreated; the dynamic image sequence obtained through the pretreatment is encoded and decoded, and multi-level features are extracted; secondly, the key points of the dynamic image sequence are coarsely positioned; then, an adaptive Bayesian hypergraph model is constructed, and a hypernode is extended, so that the key points are finely adjusted; finally, the key frames are identified, and the synchronous detection of the key points and the key frames is realized; in addition, the application also proposes an order loss function, establishes the relative relationship between the key frames and non-key frames, realizes the accurate identification of the key frames, and promotes the accuracy improvement of the key point positioning.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided diagnostic technology, specifically relating to a method for locating key points and key frames in dynamic images. Background Technology

[0002] Accurate landmark detection in medical imaging is crucial for quantifying various anatomical structures, assisting in diagnosis, and developing treatment plans. Dynamic imaging, a common form of image storage, allows real-time viewing of tissue and organ structure and blood flow patterns, providing additional motion diagnostic information compared to static imaging and enabling the identification of subtle abnormalities. In routine clinical practice, manually identifying keyframes in dynamic images to discover, measure, and diagnose specific clinicopathological and physiological events is a fundamental task for clinicians. However, even for experienced physicians, performing this repetitive task is quite challenging, especially given complex anatomical variations, poor image quality, and the pressure of heavy clinical workloads. Therefore, there is a growing need for automated tools that can simplify and improve the accuracy of such diagnostic processes, enabling the simultaneous identification of key points and keyframes in dynamic images.

[0003] Currently, keypoint and keyframe detection tasks in dynamic images are performed independently. Keypoint detection methods are categorized as follows: 1) Traditional keypoint detection methods rely on carefully crafted feature descriptors and well-defined detectors to analyze unique features in the image. For example, statistical models and template matching methods remain suitable for anatomical keypoint localization due to their simplicity and efficiency; 2) Deep learning methods based on direct regression directly predict coordinates or coordinate offsets; 3) Deep learning methods based on heatmaps perform semantic segmentation of keypoints, where semantic labels are generated using a Gaussian distribution centered on each keypoint. For keyframe detection, existing methods are mainly divided into: 1) Classification-based methods: treating each frame in the dynamic image as a 0-1 binary classification problem for keyframes, extracting features of specific events (such as the appearance of tumor lesions or cardiac motion phases) for frame-by-frame classification prediction; 2) Indexed regression-based methods: extracting global temporal features of the dynamic sequence to regress the index value of a keyframe.

[0004] MHJafari et al., “U-land: Uncertainty-driven video landmark detection,” IEEE Trans. Med. Imag., vol. 41, no. 4, pp. 793-804, 2022. (U-Land: Uncertainty-driven video landmark detection), used deep learning technology to model an anatomical landmark detection model and utilized the uncertainty generated by landmark detection to identify keyframes. This study is the first to discover a correlation between landmark detection and keyframe identification in dynamic film data, and uses empirical rules to filter keyframes. Although this study, published in a top medical imaging journal, can identify dynamic structures with high image quality and significant dynamic differences, it has poor robustness to images with insignificant inter-frame differences under noise. Post-processing methods relying solely on empirical rules are insufficient to meet clinical needs.

[0005] Due to the inherent characteristics of dynamic imaging, image resolution and quality are low, leading to high inter-observer variability in both keypoint and keyframe detection tasks, resulting in high uncertainty in measurement and diagnostic tasks. Existing automated detection methods do not fully explore and utilize the intrinsic relationship between the two tasks, treating keypoint and keyframe detection as two independent tasks. In particular, keypoint detection methods lack sufficient modeling of the structural relationships between points, easily leading to significant measurement errors when applied to low-quality images. Keyframe detection methods face severe class imbalance between keyframes and non-keyframes, resulting in significant model overfitting. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method for locating key points and key frames in dynamic images, enabling simultaneous detection of key frames and key points. By exploring the inherent connection between these two tasks, it accurately and automatically detects anatomical key points in dynamic images, thereby helping clinicians quickly locate target structures, achieving automated tissue and organ measurement, and improving their diagnostic efficiency.

[0007] A method for locating key points and keyframes in dynamic images, comprising the following steps:

[0008] Step 1: Preprocess the dynamic image sequence containing the target to be measured;

[0009] The preprocessing includes:

[0010] The dynamic image sequence containing the target to be measured is sampled at equal intervals. The image sequence is divided into several groups, each containing N frames. Groups with less than N frames are supplemented by copying the last frame. After sampling preprocessing, the input of the detection model is an image sequence with a fixed number of frames.

[0011] Step 2: Encode and decode the dynamic image sequence obtained in Step 1: Specifically, extract multi-level features frame by frame from the dynamic image sequence using a convolutional neural network with encoder and decoder structures;

[0012] The encoder contains four sets of convolutional layers, each set containing two consecutive convolutional kernels. Each set of convolutional layers is followed by a pooling layer for feature downsampling. The feature obtained after the first set of convolutional layers is the shallow feature F. shallow The features obtained after the fourth set of convolutional and pooling layers are the bottom-level features F. bot Shallow and low-level features are used for subsequent keypoint fine-tuning and keyframe detection, respectively.

[0013] Correspondingly, the decoder contains four sets of convolutional layers, each containing two consecutive convolutional kernels. Each set of convolutional layers is followed by a transposed convolution to upsample the features. The features obtained from the fourth set of convolutions and the transposed kernel constitute the global features F. global Used for coarse positioning of key points;

[0014] Step 3: Perform coarse localization of key points in the dynamic image sequence;

[0015] Coarse localization of key points is performed based on heatmap regression; specifically, the feature map F obtained from the last layer of the decoder is... global Convolution is performed to obtain heatmaps of K key points, and a Softmax operation is performed on each heatmap to obtain a heatmap Z for coarse localization. k Furthermore, the coordinates of the key points for coarse localization are obtained through heatmap regression:

[0016] L k =∫ P P·Z k (P)

[0017] Where P represents the coordinate value of each pixel in the heatmap;

[0018] Step 4: Construct an adaptive Bayesian hypergraph model and extend it to include supernodes;

[0019] Step 4.1: Constructing supernodes and the incident matrix;

[0020] Centered on the coarse localization coordinates, the system adaptively expands in 8-neighborhood directions with a step size of L pixels, forming K×9 supernodes with the coarse localization node as the central node and its surrounding 8-neighborhood as expansion nodes. Furthermore, a learnable incident matrix is ​​constructed based on the number of keypoints to be detected. n is the number of hyperedges in the hypergraph model, and each hyperedge connects K×9 hypernodes;

[0021] Step 4.2: Extraction of local image features of supernodes;

[0022] To extract local image features, the dynamic image sequence input in step 1 and the shallow feature map F from the encoder in step 2 are used respectively. shallow Centered on the supernode, an L×L feature block is extracted; further, the extracted feature blocks are concatenated along the channel dimension, and then subjected to L×L convolution to obtain the 1D local image feature F of each supernode. local ;

[0023] Step 4.3: Extraction of supernode structural relationship features;

[0024] Calculate the normalized coordinate difference between each of the K×9 supernodes and other supernodes, and combine it with its own normalized coordinates to form the supernode structural relationship feature F. struct ;

[0025] Step 4.4: Transfer the local features F from step 4.2 local and the structural feature F obtained in step 4.3 struct The feature F, which serves as a supernode, is obtained by concatenating the channel dimensions. land ∈R i ;

[0026] Step 5: Fine-tune the key points;

[0027] Specifically, the proposed adaptive Bayesian hypergraph convolution is used to transmit information between the supernodes obtained by adaptive expansion in step 4, and to update the features of the central supernode to predict its own fine-tuning scale and direction, so as to achieve fine-tuning of the key point coordinates and obtain the final key point coordinates.

[0028] Step 5.1: Construct an adaptive Bayesian hypergraph convolution;

[0029] To enable information transfer between supernodes, an adaptive Bayesian hypergraph convolution is established, and the supernode features F are processed. land After adaptive Bayesian hypergraph convolution, the updated features of the K key points are obtained:

[0030]

[0031] Where σ represents the nonlinear activation layer RELU; γ∈R K×9K Let be a Bernoulli random vector, with each component having a probability of 0.2 of being 1; T∈R K×9K , is a learnable adaptive aggregation matrix; Let n be the learnable incident matrix, and n denote the number of hyperedges; Λ∈R n×n , is the learnable diagonal hyperedge weight matrix; for The transpose of W; W∈R i×j It is a learnable transformation layer;

[0032] Step 5.2: Predicting the fine-tuning scale and direction of key points;

[0033] The features F of the K key points obtained in step 5.1 l ′ and The fine-tuning scaling matrix S∈R is obtained through a multilayer perceptron for optimizing the coordinate position. K×2 and direction probability matrix P∈R K×9 And further obtain the coordinate offset:

[0034] O P =P·O0⊙S

[0035] Where ⊙ is the Hadamard product of the matrices; O0∈R 9×2 For the pre-defined 8-neighborhood offset matrix:

[0036]

[0037] Therefore, the result of coarse positioning in step 3, L k Based on this, proceed with O P Fine-tuning of the order of magnitude yields precise keypoint coordinates L′ k ; and use L1 loss to train the keypoint localization branch formed in steps 3, 4, and 5;

[0038] Step 6: Identify keyframes;

[0039] The bottom feature F of the encoder in step 2 bot The 1D features obtained by adaptive average pooling and the keypoint coordinates L′ obtained in step 5 k F is obtained by splicing. Bi-GRU This is then passed to a Bi-GRU to predict the activation value H of each frame t in each group as a keyframe. t Set the activation value H of keyframe M. M To find the minimum value of the entire sequence, a loss function of order L is proposed for training the Bi-GRU network. order It is composed of a weighted combination of the partial log-release loss Log(PL) and the triplet loss TriL(a,p,n):

[0040] L order =log(PL(H) M ))+0.1·TriL(M,p,n)

[0041]

[0042]

[0043] Where log represents the natural logarithm, e is the natural constant, and H a H b H d H n The values ​​represent the frame activation values ​​for different indices. T = N represents the maximum index of the image frame within the group, p represents the positive example frame within the group that is closer to the key frame M, and n represents the negative example frame within the group that is farther away from the relative p frame. The encoder's low-level features represent positive example frames. The encoder's low-level features represent keyframes. The encoder's low-level features represent negative example frames. By setting the above loss function, the activation value of keyframe M is trained to be the global minimum. Positive example frames p that are closer to M have image features that are closer to M. Steps 2, 3, 4, 5, and 6 finally form a complete dual-branch model for keypoint and keyframe localization. Multi-task synchronous training is performed using sparse dynamic image data with keypoint coordinates marked only on keyframes, thus achieving automated anatomical structure measurement.

[0044] Beneficial technical effects of the present invention:

[0045] This invention fully leverages the inherent correlation between point and frame localization tasks by establishing a multi-task end-to-end localization model and sharing multi-task feature encoding. This overcomes the class imbalance problem in keyframe localization and the need for large-sample training data for the model, enabling the training of a dynamic image localization model under sparse annotation. Furthermore, it proposes a Bayesian hypergraph convolution model, achieving precise keypoint localization from coarse to fine optimization based on the encoder and decoder, overcoming the high measurement uncertainty caused by image quality. Finally, it proposes an order loss function to establish the relative relationship between keyframes and non-keyframes, achieving accurate keyframe identification while simultaneously improving the accuracy of keypoint localization. Attached Figure Description

[0046] Figure 1 A flowchart of a method for locating key points and key frames in dynamic images according to an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of adaptive Bayesian hypergraph construction according to an embodiment of the present invention;

[0048] Figure 3 Diagram of the end-to-end keypoint and keyframe detection multi-task model of this invention embodiment;

[0049] Figure 4 Automated anatomical structure measurement diagram of this invention embodiment;

[0050] Figure 5 Bland-Altman consistency analysis diagram of an embodiment of the present invention. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0052] A method for locating key points and keyframes in dynamic images is proposed, which establishes an end-to-end multi-task detection model, as shown in the attached figure. Figure 3 As shown, multi-task shared feature encoding and a coarse-to-fine localization strategy enable synchronous detection of key points and keyframes; the method flowchart is as follows. Figure 1 As shown, the steps are as follows:

[0053] Step 1: Preprocess the dynamic ultrasound image sequence containing the target to be measured;

[0054] The preprocessing includes:

[0055] Since the detection objective is to identify key points and specific keyframes of target tissues and organs in dynamic ultrasound images, the data input to the detection model is a dynamic ultrasound image containing the target to be measured. Considering the limitations of computing resources, the dynamic ultrasound image sequence containing the target to be measured is sampled at equal intervals, and the image sequence is divided into several groups, each containing 10 frames. Groups with fewer than 10 frames are supplemented by copying the last frame. After sampling preprocessing, the input to the detection model is an ultrasound image sequence with a fixed number of frames, and the amount of computation and memory consumption are controlled, which is beneficial for deployment on low- and mid-range equipment.

[0056] Step 2: Encode and decode the dynamic ultrasound image sequence obtained in Step 1: Specifically, extract multi-level features frame by frame from the dynamic ultrasound image sequence using a convolutional neural network with encoder and decoder structures;

[0057] The encoder contains four sets of convolutional layers, each set containing two consecutive convolutional kernels. Each set of convolutional layers is followed by a pooling layer for feature downsampling. The feature obtained after the first set of convolutional layers is the shallow feature F. shallow The features obtained after the fourth set of convolutional and pooling layers are the bottom-level features F. bot Shallow and low-level features are used for subsequent keypoint fine-tuning and keyframe detection, respectively.

[0058] Correspondingly, the decoder contains four sets of convolutional layers, each containing two consecutive convolutional kernels. Each set of convolutional layers is followed by a transposed convolution to upsample the features. The features obtained from the fourth set of convolutions and the transposed kernel constitute the global features F. global Used for coarse positioning of key points;

[0059] Step 3: Coarsely locate key points in the dynamic ultrasound imaging sequence:

[0060] Coarse localization of key points is performed based on heatmap regression; specifically, the feature map F obtained from the last layer of the decoder is... globalPerform 3×3 convolution to obtain heatmaps of K key points, and perform Softmax operation on each heatmap to obtain heatmap Z for coarse localization. k Furthermore, the coordinates of the key points for coarse localization are obtained through heatmap regression:

[0061] L k =∫ P P·Z k (P)

[0062] Where P represents the coordinate value of each pixel in the heatmap;

[0063] Step 4: Construct an adaptive Bayesian hypergraph model and extend it to include supernodes;

[0064] Step 4.1: Constructing supernodes and the incident matrix;

[0065] like Figure 2 As shown, to further improve the accuracy of key point localization, this invention uses the coarse localization coordinates as the center and adaptively expands in 8-pixel steps in the 8-neighborhood direction of this center, forming K×9 supernodes with the coarse localization node as the center node and the surrounding 8-neighborhood as the expansion nodes; in addition, this invention constructs a learnable incident matrix based on the number of key points to be detected. n is the number of hyperedges in the hypergraph model, and each hyperedge connects K×9 hypernodes; the number of keypoints K is determined according to the specific imaging task. For example, if a doctor requires measuring the length and width of an organ, then 4 points are needed.

[0066] Step 4.2: Extraction of local image features of supernodes;

[0067] To extract local image features, the dynamic ultrasound image sequence input in step 1 and the shallow feature map F from the encoder in step 2 are used respectively. shallow Centered on the supernode, an 8×8 feature block is extracted; further, the extracted feature blocks are concatenated along the channel dimension, and then subjected to an 8×8 convolution to obtain the 1D local image feature F of each supernode. local ;

[0068] Step 4.3: Extraction of supernode structural relationship features;

[0069] Calculate the normalized coordinate difference between each of the K×9 supernodes and other supernodes, and combine it with its own coordinates to form the supernode structural relationship feature F. struct ;

[0070] Step 4.4: Transfer the local features F from step 4.2 local and the structural feature F obtained in step 4.3 struct The feature F, which serves as a supernode, is obtained by concatenating the channel dimensions. land ∈Ri ;

[0071] Step 5: Fine-tune the key points;

[0072] Specifically, the proposed adaptive Bayesian hypergraph convolution is used to transmit information between the supernodes obtained by adaptive expansion in step 4, and to update the features of the central supernode to predict its own fine-tuning scale and direction, so as to achieve fine-tuning of the key point coordinates and obtain the final key point coordinates.

[0073] Step 5.1: Construct an adaptive Bayesian hypergraph convolution;

[0074] To enable information transfer between supernodes, an adaptive Bayesian hypergraph convolution is established, and the supernode features F are processed. land After adaptive Bayesian hypergraph convolution, the updated features of the K key points are obtained:

[0075]

[0076] Where σ represents the nonlinear activation layer RELU; γ∈R K×9K Let be a Bernoulli random vector, with each component having a probability of 0.2 of being 1; T∈R K×9K , is a learnable adaptive aggregation matrix; Let n be the learnable incident matrix, and n denote the number of hyperedges; Λ∈R n×n , is the learnable diagonal hyperedge weight matrix; for The transpose of W; W∈R i×j It is a learnable transformation layer;

[0077] Step 5.2: Predicting the fine-tuning scale and direction of key points;

[0078] The features F′ of the K key points obtained in step 5.1 land The fine-tuning scaling matrix S∈R is obtained through a multilayer perceptron for optimizing the coordinate position. K×2 and direction probability matrix P∈R K×9 And further obtain the coordinate offset:

[0079] O P =P·O0⊙S

[0080] Where ⊙ is the Hadamard product of the matrices; O0∈R 9×2 For the pre-defined 8-neighborhood offset matrix:

[0081]

[0082] Therefore, the result of coarse positioning in step 3, L k Based on this, proceed with O PFine-tuning of the order of magnitude yields precise keypoint coordinates L′ k ; and use L1 loss to train the keypoint localization branch formed in steps 3, 4, and 5;

[0083] Step 6: Keyframe recognition;

[0084] The bottom feature F of the encoder in step 2 bot The one-dimensional feature obtained by adaptive average pooling and the keypoint coordinates L′ obtained in step 5 k F is obtained by splicing. Bi-GRU This is then passed to a Bi-GRU to predict the activation value H of each frame t in each group as a keyframe. t Set the activation value H of keyframe M. M To find the minimum value of the entire sequence, a loss function of order L is proposed for training the Bi-GRU network. order It is composed of a weighted combination of the partial log-release loss Log(PL) and the triplet loss TriL(a,p,n):

[0085] L order =log(PL(H) M ))+0.1·TriL(M,p,n)

[0086]

[0087]

[0088] Where log represents the natural logarithm, e is the natural constant, and H a H b H d H n The values ​​represent the frame activation values ​​for different indices. T = N represents the maximum index of the image frame within the group, p represents the positive example frame within the group that is closer to the key frame M, and n represents the negative example frame within the group that is farther away from the relative p frame. The encoder's low-level features represent positive example frames. The encoder's low-level features represent keyframes. This represents the encoder's low-level features of negative example frames. Through the above loss function settings, the activation value of keyframe M is trained to the global minimum. Positive example frames p, which are closer to M, have image features more similar to M. Steps 2, 3, 4, 5, and 6 ultimately form a complete keypoint and keypoint bi-branch model. Multi-task synchronous training is performed using sparsely labeled dynamic image data, thus achieving automated anatomical structure measurement. Measurement results are attached. Figure 4 As shown;

[0089] The keypoint and keyframe detection method described in this invention enables automated detection of anatomical keypoints in dynamic images and identification of keyframes in image sequences. Automated measurements of the size and depth of the left atrial appendage opening, the aortic valve annulus diameter, and the thickness of the anterior and posterior walls and the diameter of the left ventricle were performed on both transesophageal and transthoracic echocardiographic datasets. The test results were compared with manual measurements by three sonographers, and Bland-Altman consistency analysis showed high agreement. Quantitative analysis indicated a small mean absolute error compared to physician measurements, meeting the needs of clinical applications.

[0090] This embodiment uses Bland-Altman analysis to evaluate the consistency between model predictions and measurements of different human tissues and organs by experienced senior physicians. (See attached...) Figure 5 The Bland-Altman concordance analysis plot shown displays the average (x-axis) and the differences (y-axis) between the automated measurements of this invention and the physician measurements. It was observed that the average differences for the left atrial appendage diameter, left atrial appendage depth, aortic valve annulus diameter, left ventricular diameter, interventricular septum, and left ventricular posterior wall were 0.232 mm, -0.630 mm, -0.487 mm, 0.001 mm, 0.374 mm, and -0.111 mm, respectively. Furthermore, Figure 5 Most of the data points were within ±1.96 standard deviations of the mean, indicating that the automated measurements of this invention have good accuracy and consistency with experienced physicians.

Claims

1. A method for locating key points and keyframes in dynamic images, characterized in that, The steps are as follows: Step 1: Preprocess the dynamic image sequence containing the target to be measured; Step 2: Encode and decode the dynamic image sequence obtained in Step 1: Specifically, extract multi-level features frame by frame from the dynamic image sequence using a convolutional neural network with encoder and decoder structures, including shallow features. Underlying features and global features ; Step 3: Perform coarse localization of key points in the dynamic image sequence, and obtain the coordinates of the coarsely localized key points using heatmap regression. ; Step 4: Construct an adaptive Bayesian hypergraph model and extend it to include supernodes; Step 4 specifically involves: Step 4.1: Constructing supernodes and the incident matrix; Centered on the coarse positioning coordinates, and adaptively expanding in the 8-neighborhood direction of the center with L pixels as the step size, forming K×9 supernodes with the coarse positioning node as the center node and the surrounding 8-neighborhood as the expansion nodes; Furthermore, a learnable incident matrix is ​​constructed based on the number of keypoints to be detected. n is the number of hyperedges in the hypergraph model, and each hyperedge connects K×9 hypernodes; Step 4.2: Extraction of local image features of supernodes; To extract local image features, the dynamic image sequence input in step 1 and the shallow feature map of the encoder in step 2 are used respectively. Centered on the supernode, extract L×L feature blocks; concatenate the extracted feature blocks along the channel dimension, and then perform L×L convolution to obtain the 1D local image features of each supernode. ; Step 4.3: Extraction of supernode structural relationship features; Calculate the normalized coordinate difference between each of the K×9 supernodes and other supernodes, and combine it with its own normalized coordinates to form the supernode structural relationship feature. ; Step 4.4: The local features from Step 4.2 and the structural features obtained in step 4.3 Features obtained as supernodes through channel-dimensional splicing ; Step 5: Fine-tune the key points; Step 5 specifically involves: Step 5.1: Construct an adaptive Bayesian hypergraph convolution; To enable information transfer between supernodes, an adaptive Bayesian hypergraph convolution is established, and the supernode features are... After adaptive Bayesian hypergraph convolution, the updated features of the K key points are obtained. ; Step 5.2: Predicting the fine-tuning scale and direction of key points; Features of the K key points obtained in step 5.1 The fine-tuning scaling matrix for optimizing coordinate position is obtained through a multilayer perceptron. and direction probability matrix And further obtain the coordinate offset. ; Therefore, the result of coarse positioning in step 3 Based on this, Fine-tuning at the order of magnitude yields precise keypoint coordinates. ; and use L1 loss to train the keypoint localization branch formed in steps 3, 4, and 5; Step 6: Identify keyframes; Step 6 specifically involves: The bottom feature of the encoder in step 2 The 1D features obtained by adaptive average pooling and the keypoint coordinates obtained in step 5 By splicing This is then passed to a Bi-GRU to predict the activation value of each frame t in each group as a keyframe. ; Set the activation value of keyframe M To find the minimum value of the entire sequence, train a Bi-GRU network; The activation value of keyframe M is obtained as the global minimum. Positive frames p that are closer to M have image features that are closer to M. Steps 2, 3, 4, 5, and 6 finally form a complete dual-branch model for keypoint and keyframe localization. Multi-task synchronous training is performed using sparse dynamic image data with keypoint coordinates marked only on keyframes, thus achieving automated anatomical structure measurement.

2. The method for locating key points and key frames in dynamic images according to claim 1, characterized in that, The preprocessing described in step 1 includes: The dynamic image sequence containing the target to be measured is sampled at equal intervals. The image sequence is divided into several groups, each containing N frames. Groups with less than N frames are supplemented by copying the last frame. After sampling preprocessing, the input of the detection model is an image sequence with a fixed number of frames.

3. The method for locating key points and key frames in dynamic images according to claim 1, characterized in that, The encoder described in step 2 contains four sets of convolutional layers, each set containing two consecutive convolutional kernels. Each set of convolutional layers is followed by a pooling layer for feature downsampling. The features obtained after the first set of convolutional layers are shallow features. The features obtained after the fourth set of convolutional and pooling layers are the bottom-level features. Shallow and low-level features are used for subsequent keypoint fine-tuning and keyframe detection, respectively.

4. The method for locating key points and key frames in dynamic images according to claim 1, characterized in that, The decoder described in step 2 contains four sets of convolutional layers, each set containing two consecutive convolutional kernels. Each set of convolutional layers is followed by a transposed convolution to upsample the features. The features obtained from the fourth set of convolutions and the transposed kernel are global features. It is used for coarse positioning of key points.

5. The method for locating key points and key frames in dynamic images according to claim 1, characterized in that, Step 3 specifically involves coarse localization of key points based on heatmap regression; specifically, it involves analyzing the feature map obtained from the last layer of the decoder. Convolution is performed to obtain heatmaps of K key points, and a Softmax operation is performed on each heatmap to obtain a heatmap for coarse localization. The coordinates of the key points for coarse positioning are obtained: Where P represents the coordinate value of each pixel in the heatmap.

6. The method for locating key points and key frames in dynamic images according to claim 1, characterized in that, Step 5 specifically employs the proposed adaptive Bayesian hypergraph convolution to transmit information between the supernodes obtained through adaptive expansion in step 4, and updates the features of the central supernode to predict its own fine-tuning scale and direction, thereby achieving fine-tuning of the keypoint coordinates and obtaining the final keypoint coordinates.

7. The method for locating key points and key frames in dynamic images according to claim 1, characterized in that, Step 5 also includes: The updated K key points have the following features: in, Represents the ReLU nonlinear activation layer; , is a Bernoulli random vector, with each component having a probability of 0.2 of being 1; , is a learnable adaptive aggregation matrix; , where is the learnable incident matrix, and n represents the number of hyperedges; , is the learnable diagonal hyperedge weight matrix; for Transpose of; It is a learnable transformation layer; The coordinate offset is: in, The Hadamard product of the matrices; For the pre-defined 8-neighborhood offset matrix: 。 8. The method for locating key points and key frames in dynamic images according to claim 1, characterized in that, Step 6 also includes: When training the Bi-GRU network, a loss function of order is proposed. It consists of partial logarithmic explanatory loss. and triplet loss T Composed of weighted combinations: Where log represents the natural logarithm, and e is the natural constant. The values ​​represent the frame activation values ​​for different indices. T=N represents the maximum index of the image frame within the group, p represents the positive example frame within the group that is closer to the key frame M, and n represents the negative example frame within the group that is farther away from the relative p frame. Underlying features Underlying features Low-level features; training is performed using the loss function settings described above.

Images