Deep learning time sequence alignment method and system for multi-view asynchronous sequence image
By using EfficientNet-B0 and the soft dynamic temporal warping alignment loss function, the problem of specialized design for temporal alignment of multi-view asynchronous sequence images is solved, achieving high-precision alignment under weak supervision and improving the reliability of cardiovascular image analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-03-20
AI Technical Summary
Existing technologies struggle to effectively handle the temporal alignment of multi-view asynchronous sequence images, particularly in terms of cardiac motion phase consistency and dynamic temporal regularization between sequences. They lack specialized design and rely on high-quality labeled data and stable imaging conditions.
EfficientNet-B0 is used as the backbone network for feature extraction. Combined with the soft dynamic time warping alignment loss function, the temporal alignment of multi-view asynchronous sequence images is achieved through end-to-end training. The non-differentiable operations in DTW are replaced with differentiable operations by using the softening parameter γ. Combined with bandwidth constraint encoding temporal smoothness, feature extraction and alignment are performed.
High-precision temporal alignment of multi-view asynchronous sequence images was achieved under weak supervision, overcoming the dependence on high-quality labeled data and stable imaging conditions, and improving the reliability of cardiovascular image analysis.
Smart Images
Figure CN121707987A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and is a deep learning time sequence alignment method and system for multi-view asynchronous sequence images. BACKGROUND
[0002] As a kind of two-dimensional X-ray imaging technology based on catheter, coronary angiography has long been regarded as the "gold standard" for diagnosing and guiding coronary intervention. However, due to the difficulty of a single angiography to fully display the complex structure of the coronary artery tree, it is often necessary to collect sequence images from multiple views in clinical practice to obtain more complete anatomical information. These multi-view sequences often have time sequence misalignment due to the time asynchrony of the acquisition device or the dynamic nature of the heart movement, which brings significant challenges to subsequent three-dimensional reconstruction, cardiac phase analysis and functional evaluation.
[0003] Using electrocardiogram gating technology, the R wave in the electrocardiogram signal can be used as a time reference point to align the image sequence. However, the electrocardiogram signal itself is easily affected by motion artifacts, noise and device interference, resulting in inaccurate identification of the time reference point; at the same time, this method loses the time dimension information of the sequence and cannot support functional analysis related to the dynamic movement of the heart. In recent years, image processing-based methods have been gradually proposed. For example, by analyzing the time variation of pixel intensity or the local trajectory of coronary artery movement to achieve sequence alignment. However, such methods often rely on the stability of image quality and the accuracy of motion tracking, and their performance decreases significantly when the contrast agent injection is uneven or the vascular structure is complex. In addition, deep convolutional neural networks are used to directly learn the cardiac phase from images, but this method requires large-scale labeled data support, and the scarcity of medical image data limits its widespread application in clinical practice.
[0004] In view of the above problems, transfer learning and weakly supervised learning methods have gradually attracted attention. Existing research shows that by fine-tuning the convolutional neural network pre-trained on large-scale natural image datasets, effective time sequence feature extraction can be achieved on a small amount of medical data. However, existing methods still lack specialized design for the task of time sequence alignment of multi-view asynchronous sequences, especially in handling the consistency of cardiac motion phase and dynamic time warping between sequences, which still needs further optimization.
[0005] Therefore, there is an urgent need for a method and system that can fully utilize the representation capability of deep learning, combine time sequence alignment prior knowledge, and achieve high-precision time sequence alignment of multi-view asynchronous sequence images under weak supervision, to overcome the limitations of existing technology and provide reliable support for precise analysis and clinical application of cardiovascular images. SUMMARY
[0006] The application aims to provide a deep learning time sequence alignment method and system for multi-view asynchronous sequence images, effectively process multi-view asynchronous sequence images, realize time sequence alignment, and provide a reliable alignment basis for cardiovascular image analysis.
[0007] Technical scheme: To achieve the above-mentioned application purposes, the application adopts the following technical scheme:
[0008] In a first aspect, a deep learning time sequence alignment method for multi-view asynchronous sequence images includes the following steps:
[0009] S1, image screening is performed on angiography sequences collected asynchronously at different angles to construct a short sequence composed of 3 consecutive images;
[0010] S2, a feature extraction network is constructed, EfficientNet-B0 is used as the feature extraction backbone network, the classification head is removed and replaced with an identity mapping to directly output convolutional features, a hierarchical parameter freezing mechanism is implemented, and the first five network parameters, including the initial convolutional layer and part of the MBConv module, are frozen by default; the network is designed to accept three-channel input with a size of 224x224 pixels to meet the input requirements of angiography sequence images;
[0011] S3, a fully differentiable soft dynamic time warping alignment loss function (soft DTW) is used, the non-differentiable minimum operation in the classic DTW is replaced by the differentiable log-sum-exp operation through the softening parameter γ, the time sequence smoothness is combined with the bandwidth constraint coding, the input is extracted through forward propagation and the alignment loss is calculated by using the three-frame short sequence, and end-to-end differentiable training is realized by optimizing the network parameters through back propagation;
[0012] S4, using the trained network, the preprocessed short sequence is forward propagated to extract the time sequence feature vector representing the progress of the cardiac cycle;
[0013] S5, the cosine similarity of the feature vectors between different angles is calculated, a similarity matrix is constructed, and the soft dynamic time warping algorithm (soft DTW) is used to search for the optimal alignment path in the similarity matrix;
[0014] S6, according to the optimal alignment path searched, the time sequence alignment frame pair and the corresponding similarity score are outputted;
[0015] As preferred, in step S1, pre-screening and structured processing are performed on the angiography sequences asynchronously collected at different viewing angles, specifically including: based on the time sequence characteristics of the pixel intensity changes in the sequence images, by setting a threshold for the global pixel intensity gradient, the continuous image frames from the gradient exceeding the threshold to falling below the threshold are screened and retained; then, the screened images are uniformly scaled to 224x224 pixels and converted to gray to three-channel format, and then stacked and grouped in groups of three consecutive frames in a sliding window manner to form a series of short sequence with short time sequence structure as the actual input of the feature extraction network.
[0016] As preferred, in step S2, the construction of the feature extraction network specifically includes: using EfficientNet-B0 as the feature extraction backbone network, removing its classification head and replacing it with an identity mapping to directly output convolutional features; implementing a hierarchical parameter freezing strategy, freezing the first five layers of network parameters, including the initial convolutional layer and part of the MBConv module, and only fine-tuning the deep network parameters; the network is designed to accept three-channel input with a size of 224x224 pixels, and the single-channel gray image is converted to three-channel RGB format and input into the network for feature extraction, and the feature vectors extracted from the three consecutive frames are averaged in the time dimension as the time sequence enhanced feature representation.
[0017] As preferred, in step S3, the core calculation in the soft dynamic time warping alignment loss function is a completely differentiable recursive calculation process, which replaces the non-differentiable minimum operation in the classic DTW with a differentiable log-sum-exp operation to calculate the soft alignment distance by softening the parameter γ, and supports limiting the alignment path offset range by the bandwidth parameter to encode the time sequence smoothness prior; the entire calculation process maintains differentiability from the distance matrix to the soft alignment distance output, allowing backpropagation gradients to flow through this process, providing a supervision signal for the feature extraction network; in the training phase, a three-frame short sequence pair is used as input, the features are extracted and the alignment loss is calculated through forward propagation, and the network parameters are optimized using backpropagation to drive the network to learn discriminative time sequence features representing the cardiac motion phase, realizing end-to-end differentiable alignment training.
[0018] As preferred, in step S4, the time sequence feature vector extraction specifically includes: for the angiography sequences to be synchronized, a sliding window is used to sequentially extract three consecutive frames as the basic processing unit to form a short time sequence input into the trained feature extraction network; the network independently performs forward propagation on each frame in the sequence to obtain the feature vector of each frame, and then takes the average of the three feature vectors as the final time sequence enhanced feature representation of the center frame of the sliding window; after performing the above operation on two videos at different viewing angles, two feature sequences are obtained, and each feature vector in each sequence corresponds to a specific frame in the original video.
[0019] As preferred, in step S5, the calculation of the similarity matrix specifically comprises: based on the two feature sequences extracted in step 3, first, performing L2 normalization processing on all feature vectors in sequence A and sequence B respectively; then, by calculating the dot product (i.e. cosine similarity) between each normalized feature vector in sequence A and all normalized feature vectors in sequence B, a two-dimensional similarity matrix is constructed; the rows of the matrix correspond to the frame order of sequence A, the columns correspond to the frame order of sequence B, and each element in the matrix directly reflects the degree of consistency of the directions of the corresponding frames in the feature space of the two videos; the finally generated similarity matrix serves as the basis for subsequent dynamic time warping path search.
[0020] As preferred, in step S5, the dynamic time warping path search algorithm first converts the similarity matrix into a distance matrix (distance calculation method is 1 minus the similarity value at the corresponding position), and converts the maximum similarity problem into a minimum cumulative distance problem; then, the cumulative distance matrix is filled row by row and column by column through the dynamic programming algorithm, and the cumulative distance of each position is calculated by adding the distance value of the current position and the minimum cumulative distance in the , , predecessor position, and the source of the optimal path is recorded simultaneously; the algorithm allows the path to move along the diagonal, to the right or downward, and at the same time, the search range can be limited to a strip-shaped area near the diagonal by a preset bandwidth parameter (such as 5 frames), in order to encode the temporal smoothness prior of the heart phase change; after the calculation is completed, the recorded path is traced back to the starting point from the end point of the matrix , and all coordinate points on the optimal alignment path are obtained.
[0021] As preferred, in step S6, according to the optimal alignment path obtained by the soft dynamic time warping algorithm in the similarity matrix search, a series of time sequence alignment frames are generated and the cosine similarity is taken as the confidence score of the time sequence alignment frame, and finally the time sequence correspondence relationship between the different view angiography sequences based on the consistency of the heart phase is formed.
[0022] In a second aspect, the present application provides a deep learning time sequence alignment system for multi-view asynchronous sequence images, characterized in that the system comprises:
[0023] a sequence image preprocessing module, which performs image screening on the angiography sequences collected asynchronously from different views, and constructs a short sequence composed of 3 consecutive images;
[0024] The feature extraction network construction module is configured to construct a feature extraction network, adopt EfficientNet-B0 as a backbone network, remove the classification head and replace it with an identity mapping to realize direct mapping from an input image to a convolutional feature; the network training module is configured to perform end-to-end training in combination with a soft dynamic warping alignment loss function to optimize the feature extraction network parameters; the feature extraction module is configured to extract a time sequence feature vector representing the progress of a cardiac cycle using the trained network; and the similarity calculation and path search module is configured to calculate the cosine similarity of feature vectors between different views, construct a similarity matrix, and search for an optimal alignment path using a soft dynamic time warping algorithm.
[0025] The time sequence alignment output module is configured to output a time sequence alignment frame pair and a corresponding similarity score according to the optimal alignment path.
[0026] In a third aspect, a computer system includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the deep learning time sequence alignment method for multi-view asynchronous sequence images.
[0027] Advantages: Traditional methods such as electrocardiogram gating or image intensity analysis-based techniques require accurate time sequence labeling or stable image quality, and are easily disturbed by noise, artifacts, and uneven contrast agents in actual applications; the present application constructs an end-to-end differentiable soft dynamic time warping alignment loss function, so that the model can learn discriminative features highly related to the phase of cardiac motion in the case of only rough time sequence labeling or complete unsupervision, and realize time sequence alignment of multi-view asynchronous sequence images, effectively overcoming the dependence of traditional methods on high-quality labeled data and stable imaging conditions. BRIEF DESCRIPTION OF DRAWINGS
[0028] Figure 1 is the implementation flowchart of the present application;
[0029] Figure 2 is the similarity matrix and optimal path search result graph generated by the present application;
[0030] Figure 3 is the cumulative histogram of the path score; DETAILED DESCRIPTION
[0031] The technical solutions and effects of the present application will be further described below in combination with the drawings and specific embodiments.
[0032] The multi-view asynchronous sequence image deep learning time sequence alignment method disclosed in the embodiments of the present application has an implementation flowchart as shown in Figure 1 , and specifically includes the following steps:
[0033] Step S1 data preparation and sequence extraction: pre-screening and structured processing are performed on the angiography sequences collected asynchronously at different angles, specifically including: based on the time sequence characteristics of the pixel intensity change of the sequence, by setting a threshold for the global pixel intensity gradient, the continuous image frames from the gradient exceeding the threshold to falling below the threshold are screened and retained, thereby generating a data set containing an average of 30-40 frames per sequence, corresponding to 3-4 complete cardiac cycles; then, the image is reduced from 1024x1024 pixels to 224x224 or 248x248 pixels to meet the expected input size of the selected model; in addition, the screened images are stacked and grouped in groups of three consecutive frames in a sliding window manner to form a series of small sequences with short time sequences as the actual input of the model.
[0034] Step S2 feature extraction network construction and end-to-end training: a pre-trained EfficientNet-B0 is used as the feature extraction backbone network, the classification head is removed and the first 5 layers of parameters are frozen to adapt to the characteristics of limited medical image data; for the angiography time alignment task, a differentiable soft dynamic time warping alignment loss function is constructed, which replaces the non-differentiable minimum operation in traditional DTW with log-sum-exp operation through softening parameter γ, and integrates optional bandwidth constraints to encode the continuity prior of cardiac motion phase; in the training stage, 3-frame sequence pairs are randomly sampled as input, features are extracted and alignment loss is calculated through end-to-end forward propagation, the loss gradient is back-propagated to the feature extraction network to drive it to learn to extract discriminative features that can represent the cardiac motion phase; the whole training process adopts AdamW optimizer, cosine annealing learning rate scheduling and gradient clipping strategy to realize task-driven optimization from raw images to time alignment features.
[0035] Specifically, in this embodiment, the EfficientNet-B0 pre-trained on the ImageNet large-scale visual recognition dataset is used as the basic feature extraction network, and the specific implementation includes: removing the classification head (i.e. the last fully connected layer) of the original EfficientNet-B0 and replacing it with an identity mapping (nn.Identity()), so that the network directly outputs convolutional features instead of classification probabilities; a hierarchical parameter freezing mechanism is implemented, which defaults to freezing the first 5 layers of network parameters (including the initial convolutional layer and part of the MBConv module), keeping these bottom-level feature extractors unchanged, and only fine-tuning the deep network parameters; the network is designed to accept 3-channel input with a size of 224x224 pixels, for a short sequence consisting of three consecutive frames, first convert each single-channel grayscale image to three-channel RGB format through channel replication, then input the three frames into the pre-trained model as independent input samples for feature extraction, and finally average the feature vectors extracted from the three frames in the time dimension as the time-enhanced feature representation of the sequence.
[0036] In this embodiment, the soft dynamic time warping alignment loss function is a completely differentiable alignment measure, which replaces the non-differentiable minimization operation in traditional dynamic time warping with a differentiable log-sum-exp operation through a softening parameter γ, so as to calculate the soft alignment distance between two feature sequences; the function first converts the normalized feature sequence into a distance matrix based on cosine similarity, and then accumulates the alignment cost through a recursive dynamic programming process; during the entire calculation process, the search range of the alignment path is limited by the bandwidth parameter to encode the temporal smoothness prior of the cardiac motion phase, ensuring that the gradient can be backpropagated to the feature extraction network to drive it to learn to extract discriminative features that can represent the cardiac motion phase, achieving end-to-end optimization.
[0037] The calculation process is as follows: assuming that the two normalized feature sequences are and , first calculate the distance matrix , where
[0038]
[0039] The soft dynamic time warping distance accumulates the distance matrix through recursive calculation; ; ; where is the soft minimum operator,
[0040]
[0041] > 0 is the softening parameter, which controls the smoothness of the alignment path.
[0042] In this embodiment, the training process: through a special trainer class (SoftDTWAlignmentTrainer) to achieve end-to-end optimization management, first randomly extract 3 consecutive frames from the paired angiography sequence as training samples, form batch data input feature extraction network; after the network forward propagation generates the feature vectors of the two sequences, input them into the soft dynamic time warping loss function to calculate the alignment loss, which contains both the main alignment distance based on differentiable DTW; after the loss calculation is completed, backpropagation is performed, and the gradient signal flows through the entire differentiable computation graph to update the parameters of the feature extraction network, driving the network to learn to extract discriminative features that make temporal alignment easier; training uses AdamW optimizer (initial learning rate 1e-4) with cosine annealing learning rate scheduling, and implements gradient clipping (max_norm=0.5) to ensure stability.
[0043] Step 3 Feature Extraction: For the angiography sequence to be synchronized, three consecutive frames are extracted sequentially using a sliding window as the basic processing unit to form a short temporal sequence input to the trained feature extraction network. The network performs forward propagation independently on each frame in the sequence to obtain the feature vector of each frame. Then, the average of these three feature vectors is taken as the final temporal enhancement feature representation of the center frame of the sliding window. After performing the above operation on the sequences from two different perspectives, two feature vectors are obtained. Each feature vector in each sequence corresponds to a specific frame in the original video.
[0044] Step 4: Similarity Matrix Calculation: Based on the two feature sequences extracted in Step 3, firstly, L2 normalization is performed on all feature vectors in sequence A and sequence B respectively; then, a two-dimensional similarity matrix is constructed by calculating the dot product (i.e., cosine similarity) between each normalized feature vector in sequence A and all normalized feature vectors in sequence B; the rows of this matrix correspond to the frame order of sequence A, and the columns correspond to the frame order of sequence B. Each element in the matrix... It intuitively reflects the degree of directional consistency of corresponding frames in the feature space of two videos; the final generated similarity matrix serves as the basis for subsequent dynamic time warping path search.
[0045] Step 5: Dynamic Time Warping Path Search: The Dynamic Time Warping Path Search algorithm first converts the similarity matrix into a distance matrix (distance is calculated by subtracting the similarity value of the corresponding position from 1), transforming the problem of maximizing similarity into the problem of minimizing cumulative distance. Then, it fills the cumulative distance matrix row by row and column by column using a dynamic programming algorithm, at each position... The cumulative distance is calculated by the distance from the current position to the previous position ( , , The algorithm calculates the minimum cumulative distances in the matrix by summing them, and simultaneously records the source of the optimal path. The algorithm allows the path to move diagonally, to the right, or downwards, and can limit the search range to a band-like region near the diagonal using a preset bandwidth parameter (e.g., 5 frames) to encode the temporal smoothness prior of cardiac phase changes. After calculation, the algorithm starts from the matrix endpoint. Backtracking along the recorded path to the starting point This process involves obtaining all coordinate points along the optimal alignment path; finally, mapping the feature index back to the original video frame index to generate a series of synchronized frame pairs and their corresponding cosine similarity values as alignment confidence, thus completing the precise temporal alignment of the two angiography sequences. The cosine similarity matrix and the optimal alignment path sample are shown below. Figure 2 As shown.
[0046] Step 6: Output of Temporal Alignment Results: Based on the optimal path obtained by the dynamic time warping algorithm, the feature index values in the path are mapped back to the actual positions of the original sequence, generating a one-to-one temporal alignment list. Each temporal alignment frame is accompanied by a cosine similarity value extracted from the similarity matrix, serving as the alignment confidence score. The system finally outputs complete temporal alignment results, including the temporal alignment list, a similarity matrix visualization heatmap annotated with the optimal path, and a statistical analysis report of the alignment scores. The statistical analysis of the alignment scores is as follows... Figure 3 .
[0047] The present invention discloses a deep learning temporal alignment system for multi-view asynchronous sequence images, the system comprising:
[0048] The sequence image preprocessing module filters images from angiography sequences acquired asynchronously from different perspectives and constructs a short sequence consisting of three consecutive images.
[0049] The feature extraction network construction module is used to build the feature extraction network, using EfficientNet-B0 as the backbone network, removing its classification head and replacing it with an identity mapping to achieve direct mapping from the input image to convolutional features; the network training module is used to perform end-to-end training using a soft dynamic warping alignment loss function to optimize the parameters of the feature extraction network; the feature extraction module is used to extract temporal feature vectors representing the progression of the cardiac cycle using the trained network; the similarity calculation and path search module is used to calculate the cosine similarity of feature vectors between different viewpoints, construct a similarity matrix, and use a soft dynamic time warping algorithm to search for the optimal alignment path;
[0050] The temporal alignment output module is used to output temporally aligned frame pairs and their corresponding similarity scores based on the optimal alignment path.
[0051] The present invention discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the deep learning temporal alignment method for multi-view asynchronous sequence images.
[0052] The program code used to implement the method of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the steps of the method of the present invention to be performed. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server. All aspects not detailed in this invention are well-known to those skilled in the art.
Claims
1. A deep learning-based temporal alignment method and system for multi-view asynchronous sequence images, characterized in that, Includes the following steps: S1. Image screening is performed on the asynchronously acquired angiography sequences from different perspectives to construct a short sequence consisting of 3 consecutive images; S2. Construct a feature extraction network, using EfficientNet-B0 as the backbone network for feature extraction. Remove the classification head and replace it with an identity mapping to directly output convolutional features. Implement a hierarchical parameter freezing mechanism, freezing the parameters of the first five network layers by default, including the initial convolutional layer and some MBConv modules. The network is designed to accept three-channel input with a size of 224×224 pixels to meet the input requirements of angiography sequence images. S3. The fully differentiable soft dynamic time warping alignment loss function (soft DTW) is adopted. By softening the parameter γ, the non-differentiable minimum operation in the classic DTW is replaced with a differentiable log-sum-exp operation. Combined with the bandwidth-constrained encoding of temporal smoothness, the input is used to extract features and calculate the alignment loss through forward propagation using a three-frame short sequence. The network parameters are optimized through backpropagation to achieve end-to-end differentiable training. S4. Using the trained network, perform forward propagation on the preprocessed short sequences to extract temporal feature vectors representing the progression of the cardiac cycle. S5. Calculate the cosine similarity of feature vectors between different viewpoints, construct a similarity matrix, and use the soft dynamic time warping algorithm (soft DTW) to search for the optimal alignment path in the similarity matrix; S6. Based on the optimal alignment path obtained from the search, output the temporally aligned frame pairs and their corresponding similarity scores.
2. The deep learning temporal alignment method for multi-view asynchronous sequence images according to claim 1, characterized in that, In step S1, pre-screening and structuring processing are performed on the angiography sequences acquired asynchronously from different perspectives. Specifically, this includes: based on the temporal features of pixel intensity changes in the sequence images, by setting a threshold for the global pixel intensity gradient, filtering and retaining continuous image frames from the point where the gradient exceeds the threshold until it falls back below the threshold; subsequently, the filtered images are uniformly scaled to 224×224 pixels and converted from grayscale to three-channel format, and then stacked and grouped in groups of three consecutive frames using a sliding window method to form a series of small sequences with short temporal structures, which serve as the actual input to the feature extraction network.
3. The deep learning temporal alignment method for multi-view asynchronous sequence images according to claim 1, characterized in that, In step S2, the construction of the feature extraction network specifically includes: using EfficientNet-B0 as the feature extraction backbone network, removing its classification head and replacing it with an identity mapping to directly output convolutional features; implementing a hierarchical parameter freezing strategy to freeze the parameters of the first five network layers, including the initial convolutional layer and part of the MBConv module, and only fine-tuning the parameters of the deep network layers; the network is designed to accept a three-channel input of size 224×224 pixels, converting the single-channel grayscale image into a three-channel RGB format and inputting it into the network for feature extraction, and averaging the feature vectors extracted from three consecutive frames in the time dimension as a temporal enhanced feature representation.
4. The deep learning temporal alignment method for multi-view asynchronous sequence images according to claim 1, characterized in that, In step S3, the core calculation in the soft dynamic time warping alignment loss function is a fully differentiable recursive calculation process. By softening the parameter γ, the non-differentiable minimum operation in the classic DTW is replaced with a differentiable log-sum-exp operation to calculate the soft alignment distance. This supports limiting the alignment path offset range through the bandwidth parameter to encode the temporal smoothness prior. The entire calculation process maintains differentiability from the distance matrix to the soft alignment distance output, allowing the backpropagation gradient to flow through this process and providing a supervision signal for the feature extraction network. During the training phase, three short sequence pairs of frames are used as input. Features are extracted and the alignment loss is calculated through forward propagation. Backpropagation is used to optimize the network parameters, driving the network to learn discriminative temporal features representing the phase of cardiac motion, thus achieving end-to-end differentiable alignment training.
5. The deep learning temporal alignment method for multi-view asynchronous sequence images according to claim 1, characterized in that, In step S4, the temporal feature vector extraction specifically includes: for the angiography sequence to be synchronized, three consecutive frames are sequentially extracted as basic processing units using a sliding window approach to form a short temporal sequence input to the trained feature extraction network; the network independently performs forward propagation on each frame in the sequence to obtain the feature vector of each frame, and then takes the average of these three feature vectors as the final temporal enhancement feature representation of the center frame of the sliding window; after performing the above operations on two videos from two different perspectives, two feature sequences are obtained, and each feature vector in each sequence corresponds to a specific frame in the original video.
6. The deep learning temporal alignment method for multi-view asynchronous sequence images according to claim 1, characterized in that, In step S5, calculating the similarity matrix specifically includes: based on the two feature sequences extracted in step 3, firstly, L2 normalization is performed on all feature vectors in sequence A and sequence B respectively; then, a two-dimensional similarity matrix is constructed by calculating the dot product (i.e., cosine similarity) between each normalized feature vector in sequence A and all normalized feature vectors in sequence B; the rows of this matrix correspond to the frame order of sequence A, the columns correspond to the frame order of sequence B, and each element in the matrix... It intuitively reflects the degree of directional consistency of corresponding frames in the feature space of two videos; the final generated similarity matrix serves as the basis for subsequent dynamic time warping path search.
7. The deep learning temporal alignment method for multi-view asynchronous sequence images according to claim 1, characterized in that, In step S5, the dynamic time warping path search algorithm first converts the similarity matrix into a distance matrix (the distance is calculated by subtracting the similarity value of the corresponding position from 1), transforming the problem of maximizing similarity into the problem of minimizing cumulative distance. Subsequently, the cumulative distance matrix is filled row by row and column by column using a dynamic programming algorithm, at each location The cumulative distance is calculated by the distance from the current position to the previous position ( , , The algorithm calculates the minimum cumulative distances in the range of ) by adding them together and records the source of the optimal path simultaneously. The algorithm allows the path to move along the diagonal, to the right or down, and can limit the search range to a strip-shaped area near the diagonal by a preset bandwidth parameter (such as 5 frames) to encode the temporal smoothness prior of cardiac phase changes. After the calculation is complete, start from the end of the matrix. Backtracking along the recorded path to the starting point This yields all coordinate points along the optimal alignment path.
8. The deep learning temporal alignment method for multi-view asynchronous sequence images according to claim 1, characterized in that, In step S6, a series of temporal alignment frames are generated based on the optimal alignment path obtained by the soft dynamic time warping algorithm in the similarity matrix search, and the cosine similarity is used as the confidence score of the temporal alignment frame, thus forming a temporal correspondence between the angiography sequences from different perspectives based on cardiac phase consistency.
9. A deep learning temporal alignment system for multi-view asynchronous sequence images, characterized in that, The system includes: The sequence image preprocessing module filters images from angiography sequences acquired asynchronously from different perspectives and constructs a short sequence consisting of three consecutive images. The feature extraction network construction module is used to build the feature extraction network. It uses EfficientNet-B0 as the backbone network, removes its classification head and replaces it with an identity mapping to achieve a direct mapping from the input image to the convolutional features. The network training module is used to perform end-to-end training by combining the soft dynamic regularization alignment loss function to optimize the parameters of the feature extraction network. The feature extraction module is used to extract the temporal feature vectors representing the progression of the cardiac cycle using the trained network. The similarity calculation and path search module is used to calculate the cosine similarity of feature vectors between different viewpoints, construct a similarity matrix, and use the soft dynamic time warping algorithm to search for the optimal alignment path; the temporal alignment output module is used to output temporally aligned frame pairs and their corresponding similarity scores based on the optimal alignment path.
10. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the deep learning temporal alignment method for multi-view asynchronous sequence images according to any one of claims 1-8.
Citation Information
Cited By
Cytodynamics prediction method for pseudo-sequential manifold alignment and graph attention
CN121938461A