Method, system, and medium for analyzing an image sequence of periodic physiological activities

Abstract

The present disclosure relates to a computer-implemented method, system, and medium for analyzing an image sequence of periodic physiological activities. The method includes: receiving the image sequence from an imaging device, the image sequence including a plurality of images; the method further includes identifying one or more feature portions in a selected image, the one or more feature portions moving in response to the periodic physiological activities; the method further includes detecting corresponding feature portions in other images of the image sequence by a processor; and determining, by the processor, a phase of the selected image in the image sequence based on the movement of the feature portions.

Description

[0001] Cross - Reference to Related Applications

[0002] This application is a partial continuation of U.S. Patent Application No. 16 / 689,048, filed on November 19, 2019, and claims the benefit of U.S. Provisional Application No. 63 / 081,276, filed on September 21, 2020, under 35 U.S.C.§119(e). U.S. Patent Application No. 16 / 689,048 is a continuation of U.S. Patent Application No. 15 / 864,398, filed on January 8, 2018, which has now been published as U.S. Patent No. 10,499,867 on December 10, 2019. The entire contents of the above - mentioned applications are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to methods, systems, and computer - readable storage media for analyzing image sequences of periodic physiological activities. For example, the present disclosure relates to methods, systems, and computer - readable storage media for analyzing image sequences of periodic physiological activities to obtain phase information. Background Art

[0004] Many physiological activities, such as breathing and heartbeat, are periodic. The phase of each frame in an image sequence of a periodic physiological activity provides potentially useful information for medical imaging and diagnosis. For example, when a doctor interprets a computed tomography (CT) image of the lungs, they may wish to distinguish between the inhalation phase and the exhalation phase corresponding to inhalation and exhalation. In addition, these periodic movements play an important role in synchronizing the image frames in an image sequence of the physiological activity. For example, when reconstructing the three - dimensional (3 - D) structure of a physiological structure from two - dimensional (2 - D) images at various angles, it is helpful to keep the respective 2 - D images at the same phase in the periodic physiological activity. Such a method minimizes the impact of tissue deformation that may occur at different phases on the result of the three - dimensional reconstruction. In addition, in the field of cardiovascular diagnosis and treatment, accurately determining the phase of each frame in the cardiac cycle helps to derive other important physiological parameters, such as heart rate, cardiac cycle, etc. Determining these physiological parameters can provide information about health, as these physiological parameters may be indicators of potential good health or potential health problems.

[0005] X-ray coronary angiography (CAG) is a procedure that uses a contrast agent during X-ray imaging to better distinguish vascular structures from other tissues. The contrast agent can be a radiopaque agent, including but not limited to iodine, gadolinium, or barium sulfate. In CAG, the structure of the blood vessels is typically captured as a video sequence at multiple X-ray projection angles. Medical imaging applications based on CAG can include extracting the physiological structure of blood vessels and obtaining relevant physiological parameters (such as the blood vessel volume of arteries or veins, blood vessel diameter, etc.). These results enable medical imaging applications to be used to reconstruct three-dimensional vascular structures and can also be used to simulate functional hemodynamics. Simulating functional hemodynamics is mainly used to measure the impact of lesions on the blood supply to the experimental subject. During the heartbeat, the coronary arteries may deform with the heartbeat. Therefore, automatically and accurately obtaining the cycle phase corresponding to each frame in the image sequence at each angle is very important for accurately estimating vascular characteristic parameters and successfully synchronizing the image frames for reconstruction.

[0006] Typically, to achieve the above purposes, additional cycle monitoring devices, such as electrocardiogram devices and respiratory monitors, are typically used clinically to record and monitor clinically relevant information. However, these configurations are not standard configurations in hospitals for the corresponding imaging systems. Referencing additional devices will occupy space and time at the diagnostic site and also require clinicians to be distracted to compare between the cycle monitoring devices and the imaging system. In the absence of a heartbeat monitoring signal (such as a synchronous electrocardiogram gating signal / ECG), manually determining the correspondence between the image frame and the heartbeat phase is often time-consuming and is likely to be difficult to accurately identify the correspondence. Currently, the main analysis methods for estimating the cardiac cycle phase based on the vascular image sequence itself are mainly divided into the following types: using the method of calculating the difference between different frames (such as adjacent frames) to find the time window with the slowest phase change, so as to analyze several image frames within the identified time window. This method does not estimate the phase of each frame, nor can it distinguish between the diastolic and systolic phases, and is particularly sensitive to background noise in the image; it is also possible to use the method of comparing with a reference image to determine the phase. This method requires manually selecting the reference image of the desired phase or requires a high degree of coincidence between the reference image in the database and the current image sequence. When applying the above analysis methods to image sequences of periodic physiological activities other than the cardiac cycle, such as the pulmonary activities during the respiratory cycle, similar problems will also occur.

[0007] Therefore, the typical methods for analyzing the periodic phase have the following problems: these methods either rely on additional periodic monitoring devices or on manually selected reference images. These aspects of the typical methods increase the complexity of the system and the labor and time burdens on clinicians. In addition, these typical methods usually require calculations for all individual pixels in each frame of the image sequence and estimate the phase of each frame of the image in the periodic motion based on the calculation results. Such methods increase the computational burden. In addition, the periodic motion is highly sensitive to background noise in the image or may be masked and weakened by other motions and noises. Therefore, the analysis results of the typical methods are prone to lack of accuracy.

[0008] This application is proposed to solve the above problems and achieve other beneficial effects. Summary of the Invention

[0009] Some embodiments can provide a method and system for efficiently and accurately estimating the phase by using the dynamic changes of the image sequence itself based on periodic physiological activities, which can get rid of the inefficient operation of all pixels and can also effectively reduce the interference of noise and other motions on the periodic physiological activities.

[0010] To at least solve the above technical problems, the embodiments of the present disclosure adopt the following technical solutions.

[0011] According to a first aspect of the present disclosure, there is provided a computer-implemented method for analyzing an image sequence of periodic physiological activities. The computer-implemented method begins with receiving the image sequence acquired by an imaging device, the image sequence including a plurality of images. Then, one or more feature portions are identified in a selected image of the image sequence, the one or more feature portions moving in response to the periodic physiological activities. Next, the following are performed by a processor: detecting corresponding feature portions in other images of the image sequence; and determining the phase of the selected image in the image sequence based on the movement of the feature portions.

[0012] According to a second aspect of the present disclosure, there is provided a system for analyzing an image sequence of periodic physiological activities. The system includes an interface and a processor. In the system, the interface is configured to receive an image sequence acquired by an imaging device, the image sequence including a plurality of images. The processor is configured to perform the following steps. One or more feature portions are identified in an image of the image sequence, the one or more feature portions moving in response to the periodic physiological activities. Subsequently, corresponding feature portions can be detected in other images of the image sequence. The processor determines the phase of the selected image in the image sequence based on the movement of the feature portions.

[0013] According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to execute a method for analyzing an image sequence of periodic physiological activities corresponding to the above method according to various embodiments of the present disclosure.

[0014] Each embodiment of the present disclosure has at least the following beneficial effects: The embodiment calculates the movement of a specific feature part in the image sequence of the periodic physiological activity image. Such an embodiment does not need to calculate the movement of all pixels (including each individual pixel) in each image of the image sequence. Therefore, the amount of calculation is reduced and the processing efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In the drawings, which are not necessarily to scale, the same reference numerals may describe similar components in different views. Similar reference numerals with alphabetic suffixes or different alphabetic suffixes may represent different instances of similar components. The drawings generally illustrate, by way of example and not limitation, various embodiments and are used in conjunction with the description and the claims to explain the disclosed embodiments. Such embodiments are illustrative and are not intended to be an exhaustive or exclusive embodiment of the method, apparatus, system, or non-transitory computer-readable medium having instructions for implementing the method.

[0016] Figure 1 A flowchart of a method for analyzing an image sequence of periodic physiological activities according to an embodiment of the present disclosure is shown;

[0017] Figure 2A and Figure 2B A schematic diagram of a selected feature part according to an embodiment of the present disclosure is shown;

[0018] Figure 3 is a schematic diagram showing tracking of a feature part by template matching at different resolutions according to an embodiment of the present disclosure;

[0019] Figure 4A A graph of the global motion metric of a coronary angiography image sequence obtained according to an embodiment of the present disclosure is shown;

[0020] Figure 4B A graph of the global displacement of a coronary angiography image sequence obtained according to another embodiment of the present disclosure is shown;

[0021] Figure 5 A system for analyzing an image sequence of periodic physiological activities according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Figure 1A flowchart of a method for analyzing an image sequence of a periodic physiological activity according to an embodiment of the present disclosure is shown. The image sequence may include multiple 2D images or 3D images (e.g., 3D images obtained from 4D cardiac CT or other 4D scans). As Figure 1 shown, the method may start with receiving an image sequence acquired by an imaging device, where the image sequence has multiple frames (operation 101). Next, the following operations may be performed by a processor. One or more feature parts may be identified in a frame, and the feature parts move in response to the periodic physiological activity (operation 102). There is no particular limitation on this frame, as long as it is suitable for identifying the feature parts. For example, in an angiography image, a frame with good filling effect and clear image development may be used as the frame for identifying the feature parts. The corresponding feature parts may be tracked in other frames of the image sequence (operation 103). Subsequently, based on the movement of the identified and tracked feature parts, the phase of the frames in the image sequence may be determined. For example, the movement characteristic information presented by the feature parts across the image sequence may be tracked to determine the phase of each frame in the entire image sequence. The movement characteristic information includes but is not limited to movement curves, movement parameters such as speed, acceleration, etc.

[0023] Hereinafter, operations 104 - 106 are taken as examples of the process for determining the phase of the frames in the image sequence based on the movement of the feature parts, but it should be noted that the implementation manner of this process is not limited thereto. Therefore, the process of determining the phase of each frame may include all or part of operations 104, 105, and 106, or may include other operations. In operation 104, the local motion vectors of each frame of the feature parts may be determined. In operation 105, based on the determined local motion vectors of each frame of the feature parts, a motion metric curve may be determined. In operation 106, based on the motion metric curve, the phase of the frames in the image sequence may be determined.

[0024] Hereinafter, embodiments of each operation performed in this method are described in further detail.

[0025] Selection of Feature Parts

[0026] First, an image sequence acquired by an imaging device is received, where the image sequence has multiple frames. Then, one or more feature parts are identified in a frame selected from the received image sequence, and the one or more feature parts move in response to the periodic physiological activity. For example, these parts of the method are performed in operation 101 and operation 102.

[0027] The feature part includes, but is not limited to, one or more points or one or more regions in the region of interest in the image. For example, the feature part can be a region corresponding to the following anatomical structure that moves synchronously with the periodic physiological activity. For example, the feature part can be the heart of the subject, one or two lungs, or even a part or a point of these organs. Therefore, the feature part can exhibit a periodic motion pattern during the periodic physiological activity, such as the cardiac cycle, where the feature part is all or part of the heart. Similarly, when the feature part is one or two lungs, or a part of one or two lungs, the feature part can exhibit a periodic motion during breathing.

[0028] On a frame selected from the image sequence, points or regions where the image features are easy to distinguish, detect, and track can be selected as the feature part. The selected frame here refers to the frame in which the feature part is identified. The relevant image features can include, but are not limited to, brightness, texture, curved edges, etc. In some embodiments, more than one feature part can include multiple key regions. In addition, specific regions of other objects in a frame of the image sequence can be identified as the feature part. For example, the specific region can include, but is not limited to, implanted devices, etc.

[0029] Figure 2A and Figure 2B The schematic diagram shows the feature parts selected according to various embodiments of the present disclosure.

[0030] In some embodiments, for example, in Figure 2A the angiography shown, anatomical structures such as vascular bifurcations can be selected as the feature part. The vascular bifurcation can represent the part of the position where the main blood vessel branches into several sub-blood vessels on the blood vessel. Since the coronary artery only contains a few bifurcation points, and given the unique geometric structure of the vascular bifurcation, it is easier to distinguish it from other parts of the blood vessel. Therefore, selecting the vascular bifurcation as the feature part is easier to detect and track.

[0031] In other embodiments, as Figure 2B shown, specific regions of intervention components such as the end of a catheter can also be selected as the feature part. Here, the catheter is an instrument for inserting into the entrance of a blood vessel to inject a contrast agent. Given that the catheter generally uses a fixed model and shape, it can be easily distinguished from other parts of the blood vessel. Specifically, for example, when the catheter is inserted into the blood vessel, the end of the catheter abuts against the blood vessel and is easy to distinguish from the blood vessel. For example, the catheter has image features different from those of the blood vessel, including but not limited to brightness, texture, curved edges, etc. And, the end of the catheter can follow the motion such as the periodic motion of the blood vessel according to a similar time schedule, and the end of the catheter can thus exhibit the periodic motion pattern of the blood vessel.

[0032] It should be noted that in different applications, different feature parts can be selected according to the actual situation, not limited to the examples listed above.

[0033] The number of feature parts can be one, or multiple feature parts can be selected as needed. In addition, the selection of feature parts can be carried out in an automatic recognition manner, or manually selected by the user on a frame image selected from the image sequence.

[0034] By selecting the feature parts, it is only necessary to calculate the movement of the specific feature parts in the image sequence of the periodic physiological activity image, and it is not necessary to calculate the movement of all (i.e., each pixel) pixels in the image in the embodiment, thereby reducing the calculation amount and improving the processing efficiency.

[0035] Tracking of Feature Parts

[0036] In the following, taking the CAG image as an example of the image, taking more than one feature point as an example of the feature part, and taking the cardiac cycle as an example of the periodic physiological activity, it should be noted that the present disclosure is not limited thereto, and can also be widely applied to other types of images, feature parts of other structures (such as multiple points, key regions, etc.), and other types of periodic physiological activities (such as respiratory activities, etc.). The discussion of embodiments similar to the above discussion will not be repeated here.

[0037] After identifying the feature points in a frame selected from the image sequence, next, track the corresponding feature points in other frames of the image sequence. That is, determine the position where the feature points appear in other image frames. Here, i represents the serial number of the feature point, and t represents the frame serial number. This operation can be called the tracking of the feature point. After determining the positions where the feature points appear in each frame of the image sequence. After that, calculate the motion vector of the previous frame compared with the next frame of the feature point in adjacent frames as the local motion vector of the feature point in each frame. Note that the term "local motion vector" in this article is only used to distinguish from the main motion vector after performing principal component analysis, and it actually represents the motion vector of the feature point between each frame.

[0038] Tracking of feature points can be achieved in a variety of ways. As an example, the following two methods can be adopted: The first method includes establishing a statistical model of the feature points and then independently tracking the corresponding feature points in each frame based on the corresponding statistical model. This tracking method based on the statistical model is stable because it does not accumulate drift errors along the image sequence and does not cause processing failure if the feature points to be tracked disappear from the field of view. However, this training method based on the statistical model usually requires an offline training phase and relevant labeled training data. The second method includes tracking feature points between consecutive or adjacent frames. When this method is adopted, the statistical model of the feature points can be dynamically updated according to the current tracking information, so as to adaptively handle situations including shape deformation and intensity change. This tracking-based method has a fast processing speed and can generate smooth trajectories. However, this tracking-based method will accumulate drift errors along the image sequence and, if the feature points to be tracked disappear from the field of view, usually causes processing failure. These two processing methods can also be combined to create a comprehensive tracking method. All of the above methods can be used to track feature parts in the application of phase detection of physiological activities.

[0039] In some embodiments, in the case where the similarity between a feature point and other parts in the frame where it is located is lower than a first threshold, the corresponding feature point can be tracked by matching image patches. As an example, when the grayscale or color features of the feature point are obvious (that is, the similarity between the feature point and other points on the image is relatively low), a template matching method can be adopted. In template matching, a region around the feature point is selected as the description of the feature point, and then it is matched with other regions of the image in another frame. There are various matching criteria, and commonly used ones include but are not limited to cross-correlation, mutual information, etc.

[0040] As an example, when the cross-correlation coefficient criterion is adopted as the matching criterion, this example method can be implemented as follows: An independent image patch centered on the feature point is established in the previous frame and the position of the independent image patch in the previous frame is determined, such as (u, v). In the subsequent frame, an image patch of the same size is selected within the search region (such as (u + Δu, v + Δv)) of the position (u, v) of the independent image patch in the previous frame, and the convolution of the two is calculated as the cross-correlation coefficient. The image patch with the largest correlation coefficient is taken as the image patch matching the independent image patch. Subsequently, the difference in their positions is calculated as the local motion vector of the feature point. By moving the independent image patch centered on the feature point, the local motion vectors of each feature point between two adjacent frames can be calculated.

[0041] When tracking feature points in adjacent frames by template matching, the template of the feature points can be updated according to the current tracking information. This method can be used to handle the situation where the morphology of the feature points changes greatly over time.

[0042] In some embodiments, template matching can be performed using image patches of different scales. For each scale, the template matching generates a map with metric score values at each sliding position. Then, the metric score maps of different scales can be combined by summation, which can be a weighted sum. The best tracking position can be finally determined by finding the maximum value in the combined score map.

[0043] In some embodiments, template matching can be performed at different resolutions. Specifically, as an initial aspect of template matching, the optimal matching region (i.e., the first or initial candidate region) of the feature points on another frame image can be found at a low resolution; then, the position of the optimal matching region can be further optimized near the found optimal matching region at a higher resolution. This optimal matching process is repeated until the original resolution of the image is restored.

[0044] Figure 3 is a schematic diagram showing an example of tracking a feature part by performing template matching at different resolutions according to an embodiment of the present disclosure. Figure 3 In, ρ represents a function that measures the matching degree between two image patches with the two image patches as inputs. For example, the ρ function can be square difference, normalized square difference, cross correlation, normalized cross correlation, cross-correlation coefficient, normalized cross-correlation coefficient, etc. In other embodiments, other ρ functions can also be used. In this embodiment, as Figure 3As shown in the figure, the specific steps for tracking the feature part by performing template matching at different resolutions are as follows: First, a series of template image patches 310 at different scales (corresponding to resolutions) can be obtained in the template image 320. Next, the template image patch 330 at a larger scale (corresponding to a lower resolution) and the target image patch 330 at the same scale selected in the target image can be input into the ρ function for calculation, so as to obtain the matching score between the two. Subsequently, the optimal matching region of the feature point on the target image at the current scale can be determined by finding the maximum value in the combined score map 340. Next, the position of the optimal matching region can be further optimized near the found optimal matching region at a smaller scale (corresponding to a higher resolution). Repeat this process until the position of the optimal matching region of the feature point on the target image is determined at the original scale (corresponding to the original resolution).

[0045] By performing template matching at different resolutions, on the one hand, the matching speed can be improved, and on the other hand, more accurate matching results can be obtained

[0046] In some other embodiments, the tracking of feature points can also be achieved in the following manner: when the similarity between the feature point and other parts in the frame where it is located is higher than the second threshold, the corresponding feature point can be independently tracked in each frame based on the statistical model of the feature point

[0047] In practice, there are various statistical models of feature points that can be adopted. For example, an image patch with a fixed size in the area near the feature point can be used as a description of the feature point. Generally speaking, the feature description model of the feature point is a transformation on the image patch in the feature point area. In machine vision, common feature descriptions include but are not limited to SIFT, Histogram of Oriented Gradients (HOG), etc. These feature descriptions can also be obtained by training on a convolutional network with training data in another way

[0048] In addition, the statistical model of the feature point can be combined with template matching to better track the feature point

[0049] In still some other embodiments, machine learning methods can be used to implement the tracking of feature points. For example, machine learning methods can be used to train a classifier or a metric learning model to detect feature points. For example, support vector machines and Bayesian methods, such as Bayesian networks, are all commonly used classification methods. In addition, the feature description model and the classification (or metric) model of the feature point can be obtained by training with deep learning methods in an end-to-end manner. Additionally, deep learning-based methods can also be used to track feature points

[0050] Background Elimination

[0051] In an actual situation, the movement of feature points in an image is mainly affected by the superposition of the following aspects: First, the periodic movement of blood vessels according to the heartbeat; second, the periodic movement associated with breathing; third, the background movement of non-physiological activities, such as when the doctor moves the imaging device acquisition table during image acquisition to observe blood vessels more clearly, resulting in global movement. Background movement is an aspect that must be considered.

[0052] Therefore, there is the following vector relationship:

[0053] V FeaturePort i on =V heart +V respiratory +V table

[0054] In the above formula, V FeaturePortion is the (total) movement of the feature point, V heart is the movement of the feature point caused by the heartbeat, V respiratory is the movement of the feature point caused by breathing, V table is the movement caused by non-physiological activities, as discussed in detail above. As an example, in the case of intending to determine the phase of each image frame in the cardiac cycle, the cardiac movement can be regarded as the target periodic physiological activity (that is, the periodic physiological activity to be analyzed), and the respiratory movement is regarded as other (interfering) periodic physiological activities. Therefore, it is important to determine which type of periodic movement is being monitored, as different types of movements may interact and / or interfere with each other.

[0055] As an example, when the respiratory movement and non-physiological movement are negligible, the movement of the feature point can be considered to correspond to the movement of the blood vessel. However, depending on the imaging angle and specific circumstances, when the movement of the feature point caused by breathing and non-physiological movement is significant, the respiratory movement and non-physiological movement need to be taken into account. In such examples, if the principal component analysis is directly performed on the local movement vectors of each feature point in each frame, the first principal component calculated is often not only the principal movement vector corresponding to the periodic movement of the heartbeat, but may also be affected by the global movement and offset caused by breathing and device movement. In such examples, it is not possible to directly and accurately estimate the cardiac phase.

[0056] In view of the above problems, more than one tracking background marker can be added as more than one reference marker. The movement of the reference marker on the image represents the sum of the movement of the feature point on the image caused by breathing and the movement caused by non-physiological activities. Such movement is intended to represent the movement that interferes with the identification of the target movement from the overall movement. That is, there is the following relationship:

[0057] V fiducial =V respiratory +Vtable

[0058] In the above formula, V fiducial is the movement of the reference marker on the image. Thus, the following relational expression can be obtained from the above formula:

[0059] V heart = V FeaturePortion – V fiducial

[0060] Specifically, in some embodiments, the local motion vector of the feature portion can be determined by the following operations. First, the displacement between the respective positions of the corresponding feature points in two adjacent frames is calculated. Next, more than one background marker can be identified in a frame. The periodic motion of the background marker caused by the target periodic physiological activity is less than the periodic motion of the feature points caused by the target periodic physiological activity. In some embodiments, the background marker can exhibit non-periodic motion (for example, a doctor may move the imaging device acquisition table), or can exhibit periodic motion caused by a periodic physiological activity other than the periodic physiological activity being observed (for example, movement caused by respiratory motion). Although it may be useful to consider the deformation of the lungs during respiration when respiration is the focus of the embodiment, this deformation is not helpful when analyzing the beating heart.

[0061] More than one background marker can include at least a part of an anatomical structure, at least a part of an implanted device, or at least a part of other objects or external objects, etc. Specifically, in the example where the cardiac motion is the target periodic physiological activity, the background marker can include anatomical sites outside the lungs. The lungs usually move synchronously with the cardiac motion, and the anatomical sites outside the lungs are less affected by the cardiac motion, so that respiratory motion and non-periodic device movement can be presented. In another example, the background marker can be a position on the human body with obvious image features, such as the diaphragm muscle. Alternatively, the background marker can also include a part of an inherent external object in the image frame. For example, for a lung CT, the background marker can be a part of the acquisition table in the image frame. It should be noted that different background markers can be selected according to the actual situation in different applications, and are not limited to the examples listed above. Then, the background marker can be tracked in other frames, and the displacement of the background marker can be calculated accordingly. More specifically, the displacement between the respective positions of the corresponding background markers in two adjacent frames can be calculated. Such a combination can obtain the local motion vector of the feature points by combining the displacement of the background marker to offset the influence of non-target periodic physiological activities and other interfering movements. For example, the local motion vector of the feature points can be obtained by subtracting the displacement of the corresponding background marker from the displacement of the corresponding feature points.

[0062] By adding more than one background marker as a reference marker to offset the effects of respiration and device movement, the interference of noise and other movements on periodic physiological activities can be effectively reduced, enabling more accurate subsequent phase analysis. At the same time, this method using background markers can avoid performing principal component analysis on the local motion vectors of motion information introducing respiratory motion and non-physiological motion, which cannot be used for subsequent phase analysis. Therefore, this method avoids wasting computing resources.

[0063] In the above calculations, the positions of each feature point in each frame are obtained. (Here, i represents the serial number of the feature point, and t represents the frame serial number), thus obtaining the local motion vectors of each frame of each feature point.

[0064] Next, the process of determining the motion metric curve based on the local motion vectors of each frame of the determined feature part will be further described in detail.

[0065] In some embodiments, the motion metric curve can be determined based on the local motion vectors of each frame of the determined feature part through the following operations: First, based on the principal component analysis of the local motion vectors, the principal motion vector of each feature part in each frame can be extracted. Then, for each frame, the global motion metric of the frame can be calculated based on the local motion vector and the principal motion vector of the frame. Finally, based on the global motion metrics of each frame, the motion metric curve can be determined.

[0066] In addition, for simplicity of calculation, the principal motion vector can be predefined without performing principal component analysis on the local motion vectors. For example, the principal motion vector can be defined in the head-to-toe direction or a specific direction in the image (taking the y-axis direction as an example).

[0067] Analysis of Main Motion Vectors

[0068] In the following, taking the principal motion vector analysis method as an example, how to determine the phase of the frames in the image sequence based on the motion of the feature part will be described. However, it should be noted that this is only an example and not a limitation. For example, other methods than extracting the principal motion vector can also be used to track the motion characteristic information presented by the feature part across the image sequence (including but not limited to motion curves, motion parameters such as speed, acceleration, etc.). For example, alternative methods such as averaging or integrating the motion characteristic information of each feature part can be used to determine the phase of each frame in the entire image sequence.

[0069] Due to the non-rigid motion of the cardiovascular system with the heartbeat, there are relative displacements between multiple feature points. To reduce the influence of relative displacements, after calculating the local motion vectors of feature points in each frame as described above, principal component analysis (PCA) can be used to extract the principal motion vectors of each frame image of the image sequence of periodic physiological activities. The principle of action of principal component analysis is to integrate the features in the direction of the largest variance change, highlighting the vectors that most significantly affect the motion of feature points. Since the motion trends of multiple feature points are highly correlated with each other, the result obtained based on this principal motion vector can be regarded as the global motion metric of each frame, and using such a result can characterize the global changes of the image sequence of periodic physiological activities.

[0070] Next, taking the coronary angiography (CAG) image sequence as an example, the specific process of principal component analysis will be described.

[0071] In the CAG image sequence, the coronary blood vessels and the catheter are the main moving objects. Under normal image acquisition conditions, the motion and deformation of the blood vessels and the catheter in the image are mainly caused by the heartbeat. Therefore, the first principal component in the principal component analysis result corresponds to the periodic motion of the heartbeat.

[0072] Specifically, M t can be used to represent the motion of feature points in the t-th frame, which is a set or group of the motions of each feature point in this frame. Here, n is the number of feature points, is the motion of the i-th feature point in the (x, y) direction. The value of is obtained by difference:

[0073] For the convenience of subsequent analysis, the elements in M t can be expanded in order into a one-dimensional vector with a length of 2n, denoted as EM t , for example:

[0074] The one-dimensional vectors EM of N frames of the image sequence are combined to obtain a high-dimensional motion vector, denoted as D = {EM t , t = 1,..., N}, where N is the frame ordinal number. D is a matrix with dimensions (2n) × N. In this way, by using principal component analysis, the eigenvector corresponding to the largest eigenvalue of this high-dimensional motion vector can be obtained according to Formula 1, which is also called the first principal component or the principal motion vector, denoted as X * .

[0075] X * = argmax ||x||＝1 var(D T X) Formula 1

[0076] This principal motion vector X* Characterize the direction and magnitude of the most significant motion changes of the feature points (also referred to as the principal motion direction and magnitude). The principal motion vector can be reversed as needed to align with the desired positive direction defined in that particular application setting.

[0077] By using the method of principal component analysis, the influence of a part of the background noise (especially static noise) can be filtered out. Compared with the existing methods that only calculate the image differences between different frames and the method based on reference image comparison, it is less susceptible to background noise.

[0078] Various methods can be used to perform principal component analysis. For example, linear decomposition or non - linear decomposition can be used to perform principal component analysis. In some embodiments, such as in the application of coronary angiography, at the normal temporal sampling rate (e.g., at a speed of 15 frames per second), principal component analysis based on linear decomposition can obtain the principal components of the motion. Therefore, PCA can better interpret the motion of the heartbeat. Compared with non - linear decomposition, using linear decomposition to implement principal component analysis consumes less computing resources, has a faster computing speed, and still provides useful results.

[0079] For example, PCA can be obtained through linear matrix decomposition, such as but not limited to being obtained through singular value decomposition (SVD) or eigenvalue decomposition (EVD).

[0080] For example, in singular value decomposition, according to Equation 2, matrix D can be decomposed as

[0081] D = U S V T Equation 2

[0082] In Equation 2, both U and V are orthogonal matrices, and S is a diagonal matrix. The principal component of PCA (also referred to as the principal motion vector) X* is equal to the first column vector of V.

[0083] Furthermore, for other applications, if PCA cannot well reduce the dimension of the superposition of different motions, non - linear dimensionality reduction methods can also be used, including but not limited to kernel principal component analysis, deep belief network, etc. In addition, for simplicity, the principal motion vector can be predefined without solving equations. For example, the principal motion vector can be defined by the direction from top to bottom or a specific direction in the image (such as the y - axis direction).

[0084] Global Motion Metric Reconstruction and Phase Estimation

[0085] For each frame, based on the local motion vector and the principal motion vector calculated for that frame as above, the global motion metric corresponding to the periodic heartbeat motion of that frame can be calculated.

[0086] In some embodiments, for each frame, it can be utilized asFigure 3 The dot product in Figure 3 is used to obtain the global motion metric m(t) corresponding to this frame by projecting the local motion vector of this frame onto the main motion vector.

[0087] m(t) = dot(X * , X t ) Formula 3

[0088] In Formula 3, the main motion vector X * is the main motion vector, and X t is the local motion vector of the t-th frame.

[0089] In some other embodiments, in order to exclude the influence of the magnitude of the main motion vector during projection, the above projection can be divided by the modulus or amplitude of the corresponding main motion vector to obtain the global motion metric m(t) corresponding to each frame, as shown in Formula 4:

[0090] m(t) = dot(X * , X t ) / |X * | Formula 4

[0091] Next, after obtaining the global motion metrics of each frame, the global motion metrics of each frame can be directly used to plot a curve as the global motion metric curve. Alternatively, the global motion metric curve can also be determined by integrating the global motion metrics of the frames before each frame. Furthermore, based on this global motion metric curve, the phase of the frames in the image sequence is determined. The specific processes of determining the global motion metric curve and phase estimation will be described in more detail below.

[0092] Figure 4A shows a graph of the global motion metrics of a coronary angiography image sequence obtained according to some embodiments of the present disclosure. In Figure 4A , the horizontal axis represents the frame serial number, and the vertical axis represents the global motion metric.

[0093] Based on the global motion metrics of each frame, the phase of each frame in the periodic physiological activity can be calculated. For example, different cardiac phases correspond to different motion characteristics, as shown in Table 1.

[0094] Table 1 Motion Descriptions of Cardiac Phases

[0095] Cardiac Phase Characteristics of Corresponding Global Motion Metrics Isovolumic Contraction Phase Contractile Motion, but with Smaller Velocity Rapid Ejection Phase Rapid Contractile Motion Reduced Ejection Phase Continuous Contractile Motion, but with Slower Velocity Isovolumic Relaxation Phase Transition from Contractile Motion to Relaxation Motion Rapid Filling Phase Rapid Relaxation Motion Reduced Filling Phase Continuous Relaxation Motion, but with Slower Velocity Atrial Contraction Phase Transition from Relaxation Motion to Contractile Motion

[0096] See Figure 4A, a positive global motion intensity on the vertical axis represents diastolic motion, and a negative global motion intensity on the vertical axis represents systolic motion. Here, diastolic motion refers to the motion during heart filling, and systolic motion refers to the motion during heart emptying. The magnitude of the global motion intensity represents the magnitude of the instantaneous velocity of the motion. For example, a zero-crossing point with a positive slope indicates that the global motion metric is changing from systolic motion to diastolic motion, and the vicinity thereof is the end-systolic phase of the ventricle; a zero-crossing point with a negative slope indicates that the global motion metric is changing from diastolic motion to systolic motion, and the vicinity thereof is the end-diastolic phase of the ventricle.

[0097] In some embodiments, after determining the end-systolic phase and the end-diastolic phase of the ventricle, the interval of a complete cardiac cycle can be defined. According to the distribution ratio of each phase in the cardiac cycle, the corresponding cardiac phase of each frame can be determined, and the physical phase of each frame within the interval of 0 to 2π can also be determined.

[0098] In other embodiments, in order to further improve the accuracy of estimating the cardiac phase, the global motion intensity m(x) can be integrated over time (see Equation 5 below) to obtain a curve graph of the global displacement d(t) of each frame, as Figure 4B shown.

[0099] d(t) = ∫0 t m(x)dx Equation 5

[0100] In Figure 4B , the horizontal axis is the serial number of the frame, and the vertical axis is the magnitude of the global displacement. From Figure 4B , the period of the motion can be estimated, and the corresponding cardiac phase of each frame can be found. For example, in Figure 4B , in addition to referring to the slope and zero-crossing point described above, it can be further obtained that the peak corresponds to the end-diastolic phase of the heart, and the trough corresponds to the end-systolic phase of the heart.

[0101] In some embodiments, after determining the end-systolic phase and the end-diastolic phase of the ventricle, or after determining adjacent end-systolic phases (or end-diastolic phases) of the ventricle, the interval of a complete cardiac cycle can be determined accordingly. Based on the distribution ratio of each phase in the cardiac cycle, the corresponding cardiac phase of each frame can be determined, and the physical phase of each frame within the interval of [0 to 2π] can also be determined. Such an interval can reflect a complete cardiac cycle.

[0102] The technology of the present disclosure is not limited to the clinical application of coronary angiography. Generally, the technology of the present disclosure can also be extended to the analysis of periodic time-series images in other computer vision fields. For example, the above entire analysis process can be applied to the image sequence of the periodic respiratory motion of a pulmonary 4D computed tomography (CT). Here, 4D means that the image sequence represents a 3D model that changes over time. Of course, it is not limited thereto.

[0103] Figure 5 It is a diagram of a system for analyzing an image sequence of periodic physiological activities according to another embodiment of the present disclosure. As those skilled in the art will understand, in some embodiments, the system 500 may be a dedicated intelligent device or a general intelligent device. For example, the system 500 may be a computer customized for a hospital to handle image data acquisition and image data processing tasks, or it may be a cloud server placed where such functions can be provided. For example, the system 500 may also be integrated in an imaging device that acquires an image sequence for a patient, such as being integrated in, including but not limited to, a digital subtraction angiography (DSA) imaging device, a pulmonary CT imaging device, etc.

[0104] The system 500 may include a processor 510 and an interface 520. Optionally, as Figure 5 shown, the system 500 may additionally include at least one of a memory 530, a medical database 540, an input / output 560, and an image display 550. In other embodiments, the system 500 may not include all of these listed elements and may include other elements.

[0105] The processor 510 may be a processing device including one or more general processing devices such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor 510 may be a processor including but not limited to a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor running other instruction sets, or a processor running a combination of instruction sets. The processor 510 may also be one or more dedicated processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc.

[0106] The interface 520 is configured to receive an image sequence acquired by an imaging device, and the image sequence has a plurality of frames. The interface 520 may include a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, high-speed data transfer adapters such as optical fibers, USB3.0 adapters, Thunderbolt, etc., wireless network adapters such as WiFi adapters, telecommunication (3G, 4G / LTE, etc.) adapters, etc. The system 500 may be connected to a network through the interface 520.

[0107] The processor 510 may be communicatively coupled to the memory 530 and configured to execute computer-executable instructions stored therein. The memory 530 may include, but is not limited to, read-only memory (ROM), flash memory, random access memory (RAM), static memory, etc. In some embodiments, the memory 530 may store computer-executable instructions such as one or more processing programs 531 and medical data 532 generated when the computer program is executed. The processor 510 may execute the processing program 531 to implement the various steps of the method for analyzing an image sequence of periodic physiological activities described above in connection with Figure 1 the method described above. Optionally, when the processor 510 executes the processing program 531, it may implement the method for determining the local motion vectors of the respective frames of the feature part described above. Optionally, when the processor 510 executes the processing program 531, it may implement the method for determining the motion metric curve based on the local motion vectors of the respective frames of the determined feature part described above. Optionally, when the processor 510 executes the processing program 531, it may implement the various steps of the phase estimation described in connection with Figure 4A and Figure 4B including the steps of projecting the local motion vectors of the respective pixels of each frame onto its principal motion vector to obtain a global motion metric, and determining the phase of each frame based on the global motion metric; in some embodiments, the processing program 531 further includes the steps of determining a global displacement based on the global motion metric and determining the phase of each frame based on the global displacement, etc.

[0108] The processor 510 may also send / receive medical data 532 to / from the memory 530. For example, the processor 510 may store the image sequence obtained through the interface in the memory 530 and, when processing, obtain the stored image sequence from the memory 53. In some embodiments, when the processor 510 executes the processing program 531, it may mark the estimated phase on the corresponding image frame and transmit the image frame marked with the phase to the memory 530. Optionally, the memory 530 can communicate with the medical database 540 to obtain an image sequence from the medical database 540 or transmit the image frame marked with the phase to the medical database 540 for authorized users accessing the medical database 540 to retrieve and use.

[0109] The medical database 540 is optional and may include multiple devices located in a centralized manner or in a distributed manner such as via a cloud network. The processor 510 may communicate with the medical database 540 to read an image sequence into the memory 530 or store the image sequence from the memory 530 into the medical database 540. Optionally, the medical database 540 may also store an image sequence to be phase-estimated, etc. In such examples, the processor 510 may communicate with the medical database 540 to transmit and store the phase-marked images into the memory 530 and display them on the display 550 for a doctor to annotate the phase-marked images through professional software using the input / output 560 and enable one or more processing programs to build a data set for three-dimensional reconstruction. For example, group two-dimensional images according to the phase, and the two-dimensional images in the same group are images with the same phase or different but similar or close projection angles, which can be used together for three-dimensional reconstruction.

[0110] The input / output 560 may be configured to allow the system 500 to receive and / or send data. The input / output 560 may include one or more digital and / or analog communication devices that allow the system 500 to communicate with a user or other machines and devices. For example, the input / output 560 may include a keyboard and a mouse for the user to provide input.

[0111] In addition to displaying medical images, the image display 550 may also display other useful information, such as the phase of the image in the physiological cycle activity, etc. For example, the image display 550 may include, but is not limited to, an LCD, a CRT, or an LED display.

[0112] The present disclosure describes various operations or functions that may be implemented or defined as software code or instructions. Such content may be in source code or differential code ("delta" or "patch" code) that is directly executable ("object" or "executable" form). The software implementation of the embodiments described in the present disclosure may be provided via an article of manufacture storing the code or instructions or via a method of operating a communication interface to send data via the communication interface. A machine or computer-readable storage medium may cause a machine to perform the described functions or operations. Such a medium may include any mechanism that stores information in a form accessible by a machine (e.g., a computing device, an electronic system, etc.). For example, a computer-readable storage medium may be a recordable / non-recordable medium (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). A communication interface includes any mechanism that engages to any one of a hardwired, wireless, optical, etc. medium to communicate with another device, such as but not limited to a memory bus interface, a processor bus interface, an Internet connection, a disk controller, etc. The communication interface may be configured to be ready to provide a data signal describing software content by providing configuration parameters and / or sending signals. The communication interface may be accessed via one or more commands or signals sent to the communication interface.

[0113] The present disclosure also relates to a system for performing the operations herein. The system may be specially constructed for the required purpose or the system may include a general-purpose computer selectively activated or reconfigured by executing or running a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium such as but not limited to any type of disk including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, where the computer-readable storage medium is coupled to the bus of the computer system.

[0114] The various aspects of the present invention have been described in detail, and it is apparent that modifications and variations can be made without departing from the scope of the present invention as defined by the appended claims. Since various changes can be made to the above structures, products, and methods without departing from the scope of the present invention, it is intended that all matter included in the above description and shown in the accompanying drawings be interpreted in an illustrative rather than a limiting sense.

Claims

1. A computer-implemented method for analyzing an image sequence of periodic physiological activities, characterized in that, The method includes: Receiving the image sequence acquired by the imaging device, the image sequence including a plurality of images; Identifying one or more feature parts in a selected image of the image sequence, the one or more feature parts moving in response to the periodic physiological activity; Detecting corresponding feature parts in other images of the image sequence by a processor; Determining local motion vectors of the feature parts in each image; Determining a motion metric curve based on the determined local motion vectors of the feature parts in each image; and Determining a phase of the selected image in the image sequence based on the determined motion metric curve.

2. The computer-implemented method according to claim 1, wherein Identifying the feature parts further includes: identifying one or both of a specific region of an anatomical structure and a specific region of other objects in the image as the feature parts.

3. The computer-implemented method according to claim 1, wherein Detecting the corresponding feature parts in other images of the image sequence includes at least one of the following: Detecting the corresponding feature parts by matching of image patches; Independently detecting the corresponding feature parts in each image based on a statistical model of the feature parts; or Detecting the corresponding feature parts using a machine learning method.

4. The computer-implemented method according to claim 1, wherein, The one or more feature parts include a plurality of key regions.

5. The computer-implemented method according to claim 1, wherein Determining a motion metric curve based on the local motion vectors of the feature parts in each image includes: Extracting a principal motion vector of each feature part in each image based on principal component analysis of the local motion vectors or using a predefined principal direction; For each image, calculating a global motion metric of the image based on the local motion vectors and the principal motion vectors of the corresponding image; and Determining a motion metric curve based on the global motion metrics of each image.

6. The computer-implemented method according to claim 5, wherein Determining a motion metric curve further includes one of the following: Determining a global motion metric curve of each image by integrating the global motion metrics of the images before each image; or Using the global motion metrics of the plurality of images as the global motion metric curve.

7. The computer-implemented method according to claim 1, wherein Determining local motion vectors of the feature parts further includes: Calculating a displacement between respective positions of the corresponding feature parts in two adjacent images; Identifying a background marker in the image, the background marker having less periodic motion caused by the periodic physiological activity than the periodic motion of the feature parts caused by the periodic physiological activity; Tracking the background marker in the other images; Calculating a displacement of each background marker; and Obtaining the local motion vectors of the feature parts by combining the displacements of the background markers.

8. The computer-implemented method according to claim 7, wherein The background marker further includes at least one of at least a part of an anatomical structure, at least a part of an implanted device, and at least a part of other objects.

9. The computer-implemented method according to claim 1, wherein The periodic physiological activity includes cardiac motion.

10. The computer-implemented method according to claim 1, wherein Further includes identifying a fiducial marker and using a position of the fiducial marker to identify and correct motion caused by at least one of unanalyzed periodic physiological activity or non-physiological motion.

11. A system for analyzing an image sequence of periodic physiological activities, characterized in that, Includes: An interface configured to receive an image sequence acquired by an imaging device, the image sequence including a plurality of images; A processor configured to: Identify more than one feature in a selected image of the image sequence, the more than one feature moving in response to the periodic physiological activity; Detect corresponding features in other images of the image sequence; Determine local motion vectors of the features in each image; Determine a motion metric curve based on the determined local motion vectors of the features in each image; And Determine the phase of the selected image in the image sequence based on the determined motion metric curve.

12. The system according to claim 11, wherein The processor is further configured to identify the feature by identifying one or both of a specific region of an anatomical structure and a specific region of other objects in the image as the feature.

13. The system according to claim 11, wherein The processor is further configured to detect the corresponding feature by performing at least one of the following steps: Detect the corresponding feature using matching of image patches; Independently detect the corresponding feature in each image based on a statistical model of the feature; And Detect the corresponding feature in other images of the image sequence using a machine learning method.

14. The system according to claim 11, wherein The processor is further configured to determine the local motion vector of the feature by: Calculating the displacement between the respective positions of the corresponding feature in two adjacent images; Identifying background markers in the image, the background markers having less periodic motion caused by the periodic physiological activity than the periodic motion of the feature caused by the periodic physiological activity; Tracking the background markers in the other images; Calculating the displacement of each background marker; And Obtaining the local motion vector of the feature by combining the displacements of the background markers.

15. The system according to claim 14, wherein The background markers further include at least one of at least a part of an anatomical structure, at least a part of an implanted device, and at least a part of other objects.

16. The system according to claim 11, characterized in that, The periodic physiological activity includes cardiac activity.

17. The system according to claim 11, wherein The processor is further configured to identify fiducial markers and use the positions of the fiducial markers to identify and correct motion caused by at least one of unanalyzed periodic physiological activity or non-physiological motion.

18. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, The instructions, when executed by a processor, cause the processor to perform a method for analyzing an image sequence of periodic physiological activity, the method including: Receiving the image sequence acquired by an imaging device, the image sequence including a plurality of images; Identifying more than one feature in a selected image of the image sequence, the more than one feature moving in response to the periodic physiological activity; Detecting corresponding features in other images of the image sequence; Determining local motion vectors of the features in each image; Determining a motion metric curve based on the determined local motion vectors of the features in each image; and Determining the phase of the selected image in the image sequence based on the determined motion metric curve.

Images