A method and medium for automatic keyframe localization in X-ray angiography video

Abstract

This invention relates to an automatic keyframe localization method and medium for X-ray angiography videos. The method includes the following steps: acquiring an original X-ray video sequence containing a target blood vessel; extracting primary blood vessel features from the original X-ray video sequence; inputting the primary blood vessel features into a trained action localization network to obtain a two-stage action localization result, wherein the two-stage action includes the action of contrast agent gradually filling the blood vessel and the action of contrast agent gradually dissipating in the blood vessel; decoding the two-stage action localization result into a keyframe localization result, wherein the keyframe includes a start frame, a top frame, and an end frame; wherein the action localization network includes a long-term attention module, a block-level contrast learning module, a short-term attention module, and an action classification and boundary regression module arranged sequentially. Compared with the prior art, this invention has the advantages of effectively reducing the influence of complex mixed noise in the background and improving the accuracy of keyframe localization.

Description

Technical Field

[0001] This invention belongs to the field of information extraction technology, and relates to a video keyframe localization method, and more particularly to an automatic keyframe localization method and medium for X-ray angiography videos. Background Technology

[0002] In video analytics, it is frequently necessary to locate keyframes in video sequences. For example, in X-ray coronary angiography for diagnosing and treating cardiovascular diseases, clinical practice requires identifying three keyframes of contrast agent movement within the blood vessel: the start frame (contrast agent appears in the vessel), the top frame (contrast agent reaches maximum filling in the vessel), and the end frame (contrast agent disappears from the vessel). This is used to determine the time span of contrast agent movement within the vessel, which can be directly used to assess coronary microvascular function. Keyframe identification can be translated into action localization. Currently, the problem of action localization in video sequences is called temporal action localization (TAL).

[0003] TAL (Anchor-Based Action) aims to locate the category, start frame, and end frame of a specific action in a video sequence. Currently, the most efficient TAL methods are based on deep learning and full supervision, and are generally divided into two-stage and one-stage methods. The two-stage method, also known as the anchor-based method, first proposes actions, and then performs action classification and regression for each proposed action. The state-of-the-art method is usually a one-stage method, which typically adopts an end-to-end solution, directly predicting the classification score and regression for action boundaries at each time point.

[0004] Specifically, one-stage methods utilize convolutional neural networks, recurrent neural networks, and combinations of these two types of networks to aggregate contextual features at each time point to regress action boundaries. To represent long-term dependencies, most state-of-the-art single-stage methods employ the Transformer architecture to replace recurrent neural networks. Among them, Actionformer [Zhang, C., Wu, J. and Li, Y., 2022. Actionformer: Localizing moments of actions with transformers. ECCV] utilizes the Transformer architecture to calculate and weight the correlations at all time points to capture the internal correlations of the data, achieving state-of-the-art performance.

[0005] Current state-of-the-art Transformer-based TAL methods still have limitations in modeling temporal relationships. Due to the computationally intensive nature of the Transformer architecture, time windows are used in TAL. Frames within each time window are individually correlated and weighted, making it difficult to model relationships between frames from different time windows. This leads to a lack of long-term attention across time windows. Temporal pooling is used to alleviate this problem, but temporal pooling inevitably results in the loss of temporal information. Furthermore, the Transformer architecture directly models the correlation between two frames; however, in X-ray angiography videos, the movement of contrast agent within blood vessels is weak and continuous. Modeling the correlation of multiple consecutive frames is more reasonable than directly modeling the correlation between two distant frames.

[0006] Therefore, existing methods for locating actions from X-ray angiography video sequences are not precise enough, hindering diagnostic procedures such as quantifying and functionally analyzing the temporal span of contrast agent movement within blood vessels. In summary, the shortcomings of existing action localization methods are summarized as follows:

[0007] 1. The long-term dependency information and short-term dependency information between frames were not effectively combined;

[0008] 2. The spatial structure of objects related to motion localization (such as blood vessels) was ignored;

[0009] 3. The most advanced Transformer-based methods ignore the relationships of continuous changes between multiple frames;

[0010] 4. It overlooks the possibility of further enhancing the foreground features through the contrast between the foreground and the background. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and provide an automatic keyframe localization method and medium for X-ray angiography videos with higher keyframe localization accuracy and stronger interpretability.

[0012] The objective of this invention can be achieved through the following technical solutions:

[0013] An automatic keyframe localization method for X-ray angiography videos includes the following steps:

[0014] Acquire raw X-ray video sequences containing the target blood vessels, and extract primary features of the blood vessels from the raw X-ray video sequences;

[0015] The primary features of the blood vessels are input into a trained motion localization network to obtain two-stage motion localization results. The two-stage motion includes the action of contrast agent gradually filling the blood vessels and the action of contrast agent gradually dissipating in the blood vessels.

[0016] The two-stage motion localization results are decoded into keyframe localization results, wherein the keyframes include a start frame, a top frame, and an end frame.

[0017] The action localization network includes a long-term attention module, a block-level contrastive learning module, a short-term attention module, and an action classification and boundary regression module arranged sequentially.

[0018] Furthermore, primary vascular features are extracted from the original X-ray video sequence using a deep network model for vascular segmentation.

[0019] Furthermore, the long-term attention module extracts the relationship between frames within a long period, specifically including:

[0020] Build temporal attention using CLSTM;

[0021] Spatial attention is constructed based on traditional convolutional neural networks;

[0022] The weighted fusion of temporal attention and spatial attention is used as the output of the long-term attention module.

[0023] Furthermore, the block-level contrastive learning module is a convolutional module based on random tile projection, which projects the foreground blood vessels and the background into different latent spaces.

[0024] Furthermore, the short-term attention module is an attention module based on the Transformer architecture.

[0025] Furthermore, the short-term attention module includes, in sequence, a convolutional projection, a layer normalization, a self-attention module, a multilayer perceptron, a convolutional decoder, and a residual structure.

[0026] Furthermore, the action classification and boundary regression modules are implemented based on convolution.

[0027] Furthermore, the specific steps for obtaining the two-stage action localization result are as follows:

[0028] The output of the action localization network at each time t is obtained. Soft-NMS is used to select the action with the highest probability of belonging to the gradually filling action and the action with the highest probability of belonging to the gradually dissipating action as the final two-stage action localization result.

[0029] Furthermore, when training the action localization network, a graph-level contrastive learning module is embedded after the long-term attention module. Action classification is trained using the Focal loss function, and boundary regression is trained using the IoU loss function. The graph-level contrastive learning module is trained using the InfoNCE loss function in the first set number of training rounds, and is ignored in subsequent training rounds.

[0030] The present invention also provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the automatic keyframe localization method for X-ray angiography video as described above.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] First, this invention transforms the problem of keyframe localization in X-ray angiography videos into a motion localization problem, which facilitates continuous localization of multiple frames in a video sequence and ensures high localization reliability.

[0033] Second, this invention combines a long-term attention module based on CLSTM and convolutional neural networks with a short-term attention module based on the Transformer architecture, and uses convolutional neural networks to construct spatial attention, which makes up for the shortcomings of long-term dependence of Transformer and improves the accuracy of action localization.

[0034] Third, this invention takes into account the slow and continuous changes in blood vessels in X-ray angiography videos. It uses CLSTM to model the relationship of continuous changes between multiple frames, accurately identifies the time span from the start frame to the top frame as the action of contrast agent gradually filling the blood vessel, and the time span from the top frame to the end frame as the action of contrast agent gradually dissipating in the blood vessel, thereby improving the key frame recognition accuracy.

[0035] Fourth, this invention enhances the ability of deep networks to distinguish between foreground blood vessels and background through graph-level contrastive learning and block-level contrastive learning, thereby achieving more accurate keyframe localization results. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the process of the present invention;

[0037] Figure 2 A schematic diagram of the action localization network structure, long-term attention module, and short-term attention module constructed in this invention;

[0038] Figure 3 The diagrams are schematic diagrams of the graph-level contrastive learning and block-level contrastive learning modules constructed in this invention, wherein (a) is a schematic diagram of the graph-level contrastive learning module and (b) is a schematic diagram of the block-level contrastive learning module. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0040] This invention defines the time span from the start frame to the top frame as the gradual filling of the blood vessel with contrast agent, and the time span from the top frame to the end frame as the gradual dissipation of the contrast agent in the blood vessel. By locating these two phases of action, it indirectly locates the three keyframes required clinically. Based on the above problem transformation, this invention provides an automatic keyframe localization method for X-ray angiography videos, referencing... Figure 1 As shown, the method includes the following steps: acquiring an X-ray video sequence containing the target blood vessel; inputting the X-ray video sequence into a trained deep network model for blood vessel segmentation to extract primary blood vessel features from the video sequence; inputting the extracted primary blood vessel features into a trained action localization network, which is a deep network based on long short-term spatiotemporal attention, graph-level contrastive learning, and block-level contrastive learning; decoding the output of the action localization network into a two-stage action localization result, wherein the two-stage action includes the action of contrast agent gradually filling the blood vessel and the action of contrast agent gradually dissipating in the blood vessel; and decoding the two-stage action localization result into a localization result of three key frames, wherein the key frames include a start frame, a top frame, and an end frame.

[0041] Specifically, the construction process of the action localization network is as follows: Based on traditional CLSTM and convolutional neural networks, a long-term attention module is constructed. This module is responsible for modeling the relationship between frames over a long period. Specifically, firstly, the video sequence is processed using CLSTM to construct temporal attention. Secondly, to more accurately extract the spatial structure of blood vessels in the video sequence, spatial attention is constructed based on traditional convolutional neural networks. Finally, the temporal attention and spatial attention are combined into a long-term attention module, which can extract and weight the long-term spatiotemporal information of the video sequence. After the long-term attention module, a block-level contrastive learning module based on random tile projection is embedded. This module is a variant of convolution. The first frame of the video sequence is randomly selected as a convolutional kernel to convolve the entire video sequence, projecting foreground blood vessels and background into different latent spaces. After the block-level contrastive learning module, a short-term attention module based on the Transformer architecture is embedded, which is responsible for modeling the relationship between frames in the short term. After the short-term attention module, an action classification and boundary regression module is embedded to generate action localization results. When training the action localization network, the present invention also embeds a graph-level contrastive learning module after the long-term attention module, reusing the attention map of the long-term attention module to separate foreground blood vessels and background for contrast, further enhancing the network's ability to distinguish foreground blood vessels and background.

[0042] When training the above action localization network, the action classification is trained using the Focal loss function, the boundary regression is trained using the IoU loss function, and the graph-level contrast learning module is trained using the InfoNCE loss function in the first set number of training rounds (e.g., 5 rounds), and then ignored in the subsequent training rounds.

[0043] like Figure 2 As shown, an action localization network constructed based on the above method includes a Long-Term Attention Module, a Short-Term Attention Module, an Imagewise Contrastive Module, a Patchwise Contrastive Module, and an Action Classification & Boundary Regression Module. The Long-Term Attention Module includes a Temporal Attention Module and a Spatial Attention Module, while the Short-Term Attention Module and the Action Classification & Boundary Regression Module constitute an action former.

[0044] The above method utilizes a constructed motion localization network to effectively improve the accuracy of keyframe localization in X-ray angiography videos.

[0045] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0046] Example 1

[0047] like Figure 1 As shown, this embodiment achieves precise keyframe localization of X-ray angiography image sequences based on the above method, including:

[0048] 101) The deep network model SVS-Net for blood vessel segmentation was used to extract primary features of blood vessels from the entire video sequence:

[0049] X = SVS(X0)

[0050] Where X0 represents the original X-ray angiography video sequence, and X represents the feature sequence extracted by SVS-Net. In this invention, SVS-Net only serves to extract features from the original video sequence and does not participate in the subsequent training of the action localization network.

[0051] 102) Construct a long-term attention module based on traditional CLSTM and convolutional neural networks. Specifically, firstly, process the video sequence using CLSTM to construct temporal attention:

[0052] TA(X) = CLSTM(X)

[0053] To more accurately extract the spatial structure of blood vessels in video sequences, a convolutional block is constructed based on a traditional convolutional neural network:

[0054] ConvBlock(X)=Relu(BN(Conv(X)))

[0055] Where ReLU(·) is the activation function, BN(·) represents the batch normalization layer, and Conv(·) represents the convolutional layer. Based on the convolutional blocks, spatial attention is constructed:

[0056] SA(X) = BN(Conv(ConvBlock) 2 (X)))

[0057] ConvBlock 2 (·) represents two consecutive convolutional blocks. In this embodiment, the kernel sizes of the three convolutional layers for spatial attention are 3, 3, and 1, with strides of 1, 1, and 1, and padding of 1, 1, and 0, respectively. The temporal and spatial attention are combined into a long-term attention module, which can extract and weight the long-term spatiotemporal information of the video sequence.

[0058] STA(X)=SA(X+TA(X))

[0059] The final output of the long-term attention module:

[0060] Out(X) = X + X * Sigmoid(STA(X))

[0061] Where Sigmoid(·) is the activation function, and * represents the Hadamard product.

[0062] 103) Following the long-term attention module, a block-level contrastive learning module based on random tile projection is embedded. In the first frame of the video sequence processed by the long-term attention module, a tile is randomly selected. In this embodiment, the selected tile size is 5×5, and the tile contains background features without foreground vessels. This tile is used as a convolution kernel to convolve the entire video sequence, projecting other tiles containing foreground vessels and tiles containing only the background into different latent spaces to further distinguish between foreground and background.

[0063] 104) Following the block-level contrastive learning module, a short-term attention module based on the Transformer architecture is embedded. In this embodiment, the time window of the Transformer architecture is set to 4. Specifically, the short-term attention module includes convolutional projection, layer normalization, self-attention, multilayer perceptron (MLP), convolutional decoder, and residual structure. Convolutional projection projects the features processed by the block-level contrastive learning module onto a specific dimension C, which is C=512 in this embodiment. The self-attention module first projects the features onto three different subspaces Q, K, and V:

[0064] Q = XW Q K = XW K V = XW V

[0065] Among them W Q W K W V It is the projection matrix, then the self-attention is calculated:

[0066]

[0067] Q T Indicates the transpose of Q. This refers to the dimension of the vector. Other modules and their connections are as follows: Figure 2 As shown.

[0068] 105) Following the short-term attention module, an action classification and boundary regression module is embedded. Specifically, the action classification module uses a one-dimensional convolutional layer with convolution and activation functions to map the high-dimensional temporal features generated by the short-term attention module to n dimensions, and is responsible for predicting the action classification probability. Since two types of actions need to be located, n=2 in this embodiment. Boundary regression also uses a one-dimensional convolutional layer to map the high-dimensional temporal features generated by the short-term attention module to two dimensions, and predicts the distance from time t to the start and end of the action.

[0069] 106) Following the long-term attention module, a graph-level contrastive learning module is embedded. Specifically, in the long-term attention module, convolution generates an attention map. Based on the characteristics of the attention map, pixels with values ​​closer to 1 are more likely to be predicted as foreground elements requiring attention, while pixels with values ​​closer to 0 are more likely to be predicted as background elements requiring neglect. Therefore, this invention reuses the attention map from the long-term attention module to separate foreground vascular features and background features from the features processed by SVS-Net:

[0070] Foreground = X * AttentionMap

[0071] Background=X*(1-AttentionMap)

[0072] Foreground blood vessel features are used as positive examples, and background features are used as negative examples. The InfoNCE loss function is used for training.

[0073] 107) The primary vascular features extracted by SVS-Net are used to train the long-short-term spatiotemporal attention-based action localization model to obtain the trained model. Specifically, in this embodiment, the model is trained for 50 rounds. Action classification is trained using the Focal loss function, and boundary regression is trained using the IoU loss function. The graph-level contrast learning module is trained using the InfoNCE loss function in the first 5 rounds of training, and then this module is ignored.

[0074] 108) The output of the depth model is decoded into action localization results and simplified into a two-stage action. Specifically, the output size of the action localization model is T×4, meaning that four values ​​are output for each time t, representing the probability of belonging to a gradually filling action. The probability of a gradually dissipating action. Current time distance from the start of the action Current time distance from the end of the action Therefore, at each time t, a possible action and its category are predicted. Start of action frame End of action frame

[0075] Soft-NMS (Non-maximum suppression) is used to reduce overlapping actions. Among the remaining actions, the action with the highest probability of being a gradually filling action and the action with the highest probability of being a gradually dissipating action are selected as the final two-stage localization result. The gradually filling action is represented as (start1, end1), and the gradually dissipating action is represented as (start2, end2).

[0076] 109) Decode the two-stage motion localization into localization using three keyframes. The start frame equals the start frame of the first motion, the vertex frame equals the average of the end frame of the first motion and the start frame of the second motion, and the end frame equals the end frame of the second motion. That is:

[0077] StartFrame = start1

[0078] ApexFrame = (end1 + start2) / 2

[0079] EndFrame = end2

[0080] The overall implementation process of locating the three keyframes in the above X-ray angiography image sequence is as follows: Figure 1 As shown. The action localization network structure, long-term attention module, and short-term attention module in this embodiment are as follows: Figure 2 As shown, the graph-level contrastive learning module and the block-level contrastive learning module are as follows: Figure 3 As shown.

[0081] This embodiment uses 200 clinical angiography image sequences as an example to further illustrate the above method. Each angiography video sequence contains 31-379 frames of images, and the original resolution of each frame is 512×512. When SVS-Net proposes primary features, the feature map with an encoder stage size of 64×64 in SVS-Net is used as the primary features.

[0082] The keyframe localization process of the above-mentioned X-ray angiography video sequence has the advantages of effectively reducing the influence of background noise, fully focusing on the vascular region, and significantly improving the keyframe localization effect.

[0083] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for automatic positioning of key frames in an X-ray angiography video, characterized in that, The method comprises the following steps: collecting an original X-ray video sequence containing a target blood vessel, and extracting blood vessel primary features from the original X-ray video sequence; inputting the blood vessel primary features into a trained action localization network to obtain two-stage action localization results, the two-stage action including an action of contrast agent gradually filling in the blood vessel and an action of contrast agent gradually dissipating in the blood vessel; decoding the two-stage action localization results into key frame localization results, the key frames including a start frame, a top frame and an end frame; wherein the action localization network comprises a long-term attention module, a block-level contrast learning module, a short-term attention module and an action classification and boundary regression module arranged in sequence The specific construction process of the action localization network is as follows: based on a traditional CLSTM and a convolutional neural network, a long-term attention module is constructed, which is responsible for modeling the relationship between frames in a long term, specifically, first, the video sequence is processed by CLSTM to construct temporal attention, second, spatial attention is constructed based on a traditional convolutional neural network, and finally the temporal attention and the spatial attention are combined into a long-term attention module; after the long-term attention module, a block-level contrast learning module based on random patch projection is embedded, which is a convolutional variant, a patch is randomly selected from the first frame of the video sequence as a convolution kernel and the entire video sequence is convolved, and the foreground blood vessels and the background are projected into different latent spaces; after the block-level contrast learning module, a short-term attention module based on the Transformer architecture is embedded, which is responsible for modeling the relationship between frames in a short term; after the short-term attention module, an action classification and boundary regression module is embedded for generating action localization results.

2. The method of claim 1, wherein, The blood vessel primary features are extracted from the original X-ray video sequence by a deep network model for blood vessel segmentation.

3. The method of claim 1, wherein, The short-term attention module comprises a convolutional projection, a layer normalization, a self-attention module, a multi-layer perceptron, a convolutional decoding and a residual structure arranged in sequence.

4. The method of claim 1, wherein, The action classification and boundary regression module is implemented based on convolution.

5. The method of claim 1, wherein, The two-stage action localization results are obtained as follows: The action localization network obtains an output result at each time t The Soft-NMS is used to select an action with the maximum probability of belonging to the gradually filling action and an action with the maximum probability of belonging to the gradually dissipating action as the final two-stage action localization result.

6. The method of claim 1, wherein, when training the action localization network, a graph-level contrast learning module is embedded after the long-term attention module, the action classification is trained using a Focal loss function, the boundary regression is trained using an IoU loss function, the graph-level contrast learning module is trained using an InfoNCE loss function in the first preset number of training rounds, and the graph-level contrast learning module is ignored in the subsequent number of training rounds.

7. A computer readable storage medium characterized in that, one or more programs for one or more processors of an electronic device, the one or more programs comprising instructions for performing the X-ray angiography video key frame automatic localization method according to any one of claims 1-6.

Images