Method and device for automatic evaluation of ejection fraction by echocardiographic feature map filtering

Abstract

The application discloses an echocardiogram feature map filtering ejection fraction automatic evaluation method and device, and the method comprises the following steps: intercepting echocardiogram video data of any length of standard sections, the echocardiogram video data meets the quality standard; adopting a neural network segmentation model trained in advance to identify the data of the left ventricular long-axis four-chamber heart and two-chamber section in the echocardiogram data; filtering the feature map on the time axis by using the data of the left ventricular long-axis four-chamber heart and two-chamber section, taking the filtered feature map as the input of the segmentation prediction head and the key point prediction head, and calculating the myocardial segmentation map and the left ventricular basal segment key point; obtaining the left ventricular myocardial inner contour and the left ventricular myocardial outer contour; determining the key frame images of the diastolic end and the systolic end of the left ventricular long-axis four-chamber heart and two-chamber heart; and calculating the biplane ejection fraction of the key frame images of the diastolic end and the systolic end of the left ventricle long-axis four-chamber heart and two-chamber heart by using the Simpson method.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an automatic method and apparatus for evaluating ejection fraction by filtering echocardiographic feature maps. Background Technology

[0002] In a doctor's daily practice, ejection fraction is a very important medical indicator, used clinically to assess basic cardiac function and diagnose heart failure. External electrocardiogram (ECG) using ultrasound equipment can non-invasively and quickly diagnose a patient's ejection fraction. Therefore, in a doctor's daily practice, observing the left ventricle in a four-chamber or two-chamber view using an ultrasound probe can quickly assess a patient's ejection fraction.

[0003] Currently, the most common method doctors use on ultrasound equipment to calculate a patient's ejection fraction is in M-mode ultrasound. Based on the doctor's experience, they determine the position of the left ventricle and the location of the left ventricular septum endothelium and posterior ventricular endothelium in two frames within a cardiac cycle on a cardiac plane, and then estimate the patient's ejection fraction. This method relies on the doctor's experience and is less accurate than the biplane ejection fraction calculated using the Simpson method. However, its advantages include ease of use, low learning curve, and the ability to quickly assess a patient's approximate ejection fraction within one minute. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an automatic ejection fraction assessment method and apparatus based on echocardiographic feature map filtering, which automatically, quickly, and accurately calculates the ejection fraction of echocardiographic videos without the need for electrocardiograms, based on deep learning feature map filtering.

[0005] A first aspect of the present invention provides an automatic ejection fraction assessment method based on echocardiographic feature map filtering, comprising:

[0006] Echocardiographic video data of arbitrary duration extracted from a standard cross-section, wherein the image quality of the echocardiographic video data meets the standard;

[0007] A pre-trained neural network segmentation model was used to identify data from the left ventricular long-axis four-chamber and two-chamber sections in the echocardiographic data.

[0008] The feature map was filtered on the time axis using data from the long-axis four-chamber and two-chamber sections of the left ventricle, which served as input to the segmentation prediction head and the key point prediction head, and the myocardial segmentation map and key points of the left ventricular basal segment were calculated.

[0009] Based on the myocardial segmentation diagram and key points of the left ventricular basal segment, the inner and outer contours of the left ventricular myocardium are classified to obtain the inner and outer contours of the left ventricular myocardium.

[0010] The cardiac cycle was assessed based on the left ventricular contour area curve of echocardiographic video data, and key frame images of end-diastolic and end-systolic phases of the left ventricular long axis four-chamber and two-chamber hearts were determined.

[0011] The Simpson method was used to calculate the biplane ejection fraction of keyframe images at end-diastole and end-systole of the left ventricle in both four-chamber and two-chamber long-axis ventricles.

[0012] A second aspect of the present invention provides an automatic ejection fraction assessment device based on echocardiographic feature map filtering, comprising:

[0013] The acquisition module is used to extract echocardiographic video data of any duration from a standard cross-section, and the quality of the echocardiographic video data meets the standard.

[0014] The identification module is used to identify the data of the left ventricular long-axis four-chamber and two-chamber sections in the echocardiogram data using a pre-trained neural network segmentation model;

[0015] The calculation module is used to filter the feature map on the time axis using the data of the left ventricular long axis four-chamber and two-chamber sections, which is used as the input of the segmentation prediction head and the key point prediction head, and to calculate the myocardial segmentation map and the key points of the left ventricular basal segment.

[0016] The classification module is used to classify the inner and outer contours of the left ventricular myocardium based on the myocardial segmentation diagram and key points of the left ventricular basal segment, thereby obtaining the inner and outer contours of the left ventricular myocardium.

[0017] The keyframe module is used to assess the cardiac cycle based on the left ventricular contour area curve of echocardiographic video data, and to determine the keyframe images of the end-diastolic and end-systolic phases of the left ventricular long axis four-chamber and two-chamber hearts.

[0018] The evaluation module is used to calculate the biplane ejection fraction of keyframe images at end-diastole and end-systole of the left ventricle using the Simpson method.

[0019] A third aspect of the present invention provides an electronic device comprising:

[0020] At least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the automatic ejection fraction assessment method based on echocardiographic feature map filtering as described in the first aspect of the present invention.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a computer, performs the automatic ejection fraction assessment method based on echocardiographic feature map filtering as described in the first aspect of the present invention.

[0022] This invention alleviates the problem of poor inter-frame temporal consistency when predicting video sample data using neural networks pre-trained on single static images by applying temporal filtering to deep learning feature maps. It eliminates the need for network retraining and can be used as a plug-and-play module to extend any network from static image training results to temporally consistent video prediction.

[0023] This invention eliminates the need for an electrocardiogram (ECG) and automatically identifies the cardiac cycle based on the left ventricular endocardial area change curve in echocardiography, finding the maximum end-diastolic and maximum end-systolic phases. Combined with rapid calculation of twenty segments, it achieves a fast, automatic, and accurate calculation of Simpson's ejection fraction. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating an automatic ejection fraction assessment method based on echocardiographic feature map filtering in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the video segmentation method in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram illustrating the calculation of the area enclosed by the closed-loop curve using Green's formula in an embodiment of the present invention.

[0027] Figure 4 This is a curve showing the change in the endocardial contour area of ​​the left ventricle in an echocardiogram video in an embodiment of the present invention.

[0028] Figure 5 This is a curve showing the change in the endocardial contour area of ​​the left ventricle in a smoothed echocardiogram video, as described in an embodiment of the present invention.

[0029] Figure 6 This is a schematic diagram of the twenty-segment Simpson method structure in an embodiment of the present invention.

[0030] Figure 7 This is a schematic diagram of the n segments (n=20) of the endocardial contour in an embodiment of the present invention.

[0031] Figure 8 In this embodiment of the invention, the vector projection method is used to calculate the coordinates of the projection points of the endocardial contour points onto the PG ray.

[0032] Figure 9 This is a schematic diagram simulating an elliptical cylinder in an embodiment of the present invention.

[0033] Figure 10 A schematic diagram of a module for an automatic ejection fraction assessment device based on echocardiographic feature map filtering in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Please see Figure 1 As shown, the present invention, based on the Simpson method for dual-plane ejection fraction, requires recording two cardiac sections, namely the left ventricular long-axis four-chamber and two-chamber sections. Keyframes at end-diastole and end-systole within a cardiac cycle are selected, and the left ventricular endocardial contour is automatically determined in the keyframe images. Based on the two sets of left ventricular endocardial contours at end-diastole and end-systole in the four-chamber and two-chamber sections, the Simpson method dual-plane ejection fraction is calculated, which can obtain a more accurate ejection fraction.

[0036] This invention provides an automatic ejection fraction assessment method based on echocardiographic feature map filtering, comprising the following steps:

[0037] Step 100: Extract echocardiographic video data of any duration from a standard cross-section, wherein the quality of the echocardiographic video data meets the standard.

[0038] First, based on the results of the echocardiographic standard section quality control calculation, the standard section category of the current real-time video frame is identified. Here, the deep learning network used for the echocardiographic standard section quality control calculation includes, but is not limited to, models such as YOLOv3, YOLOv4, YOLOv5, YOLOv6, and YOLOv7 pre-trained using multiple standard echocardiographic section images. When the real-time video frame section is identified as a left ventricular long-axis four-chamber view, if n consecutive echocardiographic images are identified as left ventricular long-axis four-chamber views (n is the preset number of consecutive video frames meeting the same standard section), the algorithm automatically determines the current video segment as a left ventricular long-axis four-chamber view video and then prompts the user for final confirmation. The same method is also applicable to left ventricular long-axis four-chamber, three-chamber, and two-chamber standard sections, enabling automatic evaluation of single / biplane ejection fraction based on echocardiographic video.

[0039] Step 200: Use a pre-trained neural network segmentation model to identify the data of the left ventricular long-axis four-chamber and two-chamber sections in the echocardiogram data.

[0040] Step 300: Filter the feature map on the time axis using the data from the four-chamber and two-chamber sections of the left ventricle along the long axis. Use this data as input to the segmentation prediction head and the key point prediction head, and calculate the myocardial segmentation map and the key points of the left ventricular basal segment.

[0041] After obtaining standard echocardiographic videos of the left ventricle in long-axis four-chamber and two-chamber views, a deep learning network was used to segment the left ventricular myocardial region frame by frame. The deep learning network structure mainly consists of an encoder and a decoder. The encoder is responsible for encoding the data, reducing dimensionality, and extracting features, while the decoder decodes the extracted feature maps into header inputs for different tasks, such as left ventricular segmentation and left ventricular basal segment location. Because cardiac motion is continuous, the header input feature maps for left ventricular segmentation should also exhibit continuous motion characteristics; therefore, feature filtering is performed on the decoder's feature maps along the time axis.

[0042] Specifically: Feature image filtering is performed on the pre-trained deep convolutional neural network for left ventricular myocardial segmentation. First, the video is batch-processed according to FPS for deep learning segmentation of the left ventricular myocardium. Each batch contains n frames, with 8 frames recommended. A smaller n provides less acceleration for left ventricular myocardial segmentation inference, while a larger n consumes more GPU resources. After each batch of echocardiogram images passes through the encoder and decoder parts of the pre-trained deep convolutional neural network for left ventricular myocardial segmentation, feature maps for that batch of echocardiograms are obtained. Then, the feature maps with a frame width of m are filtered. Filtering methods include, but are not limited to, moving average filtering and Kalman filtering. By default, m = round(0.2 * FPS).

[0043] During the filtering process, n images in each batch are first fed into the left ventricular myocardium deep learning model for feature map inference. Then, the inferred feature maps are processed in batches along the time axis, and filtering is performed along the time axis with a width of m. Filtering methods include, but are not limited to, Kalman filtering, Gaussian filtering, median filtering, and mean filtering. The filtered feature maps are used as the prediction head for left ventricular basal segment keypoint detection and left ventricular myocardium segmentation tasks. These are then processed through a shallow convolutional neural network to achieve left ventricular basal segment keypoint regression and left ventricular myocardium segmentation.

[0044] In this embodiment, in order to achieve a unified and stable fast Simpson ejection fraction calculation for echocardiogram videos of different resolutions, the single-frame image size of the left ventricular myocardial segmentation task is kept at a fixed size n*n, which is set to 224 in this embodiment.

[0045] Step 400: Classify the inner and outer contours of the left ventricular myocardium according to the myocardial segmentation diagram and key points of the left ventricular basal segment to obtain the inner contour and outer contour of the left ventricular myocardium.

[0046] For the left and right keypoints of the current frame, calculate the distance from the keypoints to each contour point in turn, and find the contour point with the closest distance, which is the contour of the left ventricular myocardium.

[0047] Specifically, such as Figure 2 As shown, this is achieved in the following way:

[0048] Step 410: Based on the left ventricular myocardial segmentation results of the echocardiogram video data, extract the set of contour points in the same direction for each frame of the image;

[0049] Step 420: Calculate the distance from the key points of the left ventricular basal segment to each contour point of the myocardial segmentation map in sequence, and determine the contour point with the closest distance;

[0050] Step 430: Determine the left ventricular myocardial contour based on the nearest contour point; wherein the key point of the left ventricular basal segment is located at the left and right ends of the left ventricular myocardial segment.

[0051] In one embodiment, the echocardiogram video is processed in batches by extracting frames and then the left ventricular myocardium is segmented. Each batch of echocardiogram image frames is processed by a pre-trained deep convolutional neural network for left ventricular myocardial segmentation to obtain the feature map of that batch of echocardiograms. The feature maps of the echocardiograms are processed in batches according to the time axis, and the feature maps of the echocardiograms are filtered with a width m in the time axis direction.

[0052] Specifically, based on the left ventricular myocardial segmentation results and key points of the left ventricular basal segment of each frame of echocardiographic image, the inner and outer contours of the left ventricular myocardium are classified. The key points of the left ventricular basal segment are divided into left and right points, located at the left and right ends of the left ventricular myocardial segmentation. Therefore, by finding the contour points closest to the left and right key points on the contour points of the left ventricular myocardial segmentation, the inner and outer contour segments of the left ventricular myocardium can be distinguished. Specifically: First, based on the left ventricular myocardial segmentation results of the entire video obtained in the above steps, a set of contour points in the same direction is extracted for each frame of image using the OpenCV findContours method.

[0053] Excessive displacement deviations can cause large errors in the segmentation of the inner and outer contours of the left ventricular myocardium. To avoid excessive displacement deviations in a single frame for the left and right key points of the left ventricular basal segment obtained by deep learning regression, a key point anti-shaking algorithm is used to determine the left and right key points of the left ventricular basal segment.

[0054] For example, suppose the coordinate sequence of key points on the left and right sides of the left ventricular basal segment as a function of time (frame number) is {pl i} and {pr i}, i = 1, 2, ..., n, and the same algorithm to prevent significant keypoint jitter is applied sequentially to the left and right keypoint sequences. Let {pl i For example, specifically:

[0055] Step 1. Given the coordinate sequence {pl i Since most points in the sequence exhibit relatively small displacement changes with frame rate, calculating the median of the coordinate points in the x and y directions will ensure that the median falls within the region of minimal displacement changes in the x and y directions. For {pl} i The base point p is obtained by using the median in both the x and y directions of the coordinate system. base The formula is as follows:

[0056] p base =Median{pli}=(Median{x i},Median{y i})=(x median ,y median );

[0057] The Median function refers to the function that transforms the array {x} into a single array. i} Sort by numerical value from smallest to largest {x_sorted i When the array length n is even, Median{x} i}=x_sorted n / 2 When the array length n is odd, Median{x i}=(x_sorted (n-1) / 2+ x_sorted (n+1) / 2 ) / 2. Similarly, we can obtain the array {y} i} Sort by numerical value from smallest to largest {y_sorted i When the array length n is even, Median{y} i}=y_sorted n / 2 When the array length n is odd, Median{y i}=(y_sorted (n-1) / 2+y _sorted (n+1) / 2 ) / 2.

[0058] Step 2: Calculate the coordinate sequence {pl} i Distance from base point p base The straight-line distance is calculated using methods including, but not limited to, Norm 1 and Norm 2. Taking Norm 2 (Euclidean distance) as an example, the distance is calculated for each coordinate point pl. i Distance from base point pbase The set of line segment distances {L i The formula is as follows:

[0059]

[0060] Step 3: Calculate the adaptive displacement value threshold_deviation for large offsets. First calculate {L i Standard deviation (std) L Then, the threshold_deviation value is set using quantiles, where the quantile is preset to 0.25.

[0061] α is the quantile;

[0062]

[0063] Step 4: Anti-key point jitter algorithm detects offset points and calculates the coordinates of the detected points pl in each frame. i Distance from base point p base The distance of the straight line segment is used to determine if the distance is greater than the threshold_deviation value. If it is greater, it indicates that the coordinate point pl i If a large offset occurs, then pl will be... i Replace with p base Similarly, for pr i Execute judgments to prevent significant shaking at key points.

[0064] After completing the anti-large-amplitude jitter detection of the left and right coordinate points respectively, the left and right boundary contour points of the inner and outer contours of the left ventricular myocardium are determined.

[0065] Step 500: Evaluate the cardiac cycle based on the left ventricular contour area curve of the echocardiogram video data, and determine the key frame images of the end-diastolic and end-systolic phases of the left ventricular long axis four-chamber and two-chamber hearts.

[0066] Since the left ventricular myocardial contour area curve can reflect the heart rhythm, when Simpson's method is used to calculate the biplane ejection fraction in medicine, doctors will look for the end-diastolic and end-systolic images of the left ventricle based on the electrocardiogram. If there is no electrocardiogram, doctors will observe the end-diastolic and end-systolic image frames of the left ventricle on the echocardiogram based on experience.

[0067] This invention can automatically identify the end-diastolic and end-systolic images of the left ventricle's heartbeat, without relying on an electrocardiogram, saving doctors' time and completing keyframe image recognition.

[0068] In one embodiment, the contour area of ​​the left ventricular myocardium is calculated using Green's formula; the area curve enclosed by the left ventricular endocardial contour of the echocardiogram video frame is obtained based on the contour area of ​​the left ventricular myocardium; the peak value of the area curve enclosed by the left ventricular endocardial contour is calculated, and the peak value is used to determine the end-diastolic and end-systolic phases of the left ventricular long-axis four-chamber and two-chamber hearts.

[0069] Specifically, based on prior anatomical knowledge, the left ventricle of the human heart is a hollow cavity. Therefore, the left ventricle cannot undergo self-intersection. Green's formula is based on the premise that the shape cannot self-intersect, so it can be determined that the outline of the left ventricular endocardium will not self-intersect, making the calculation of the outline area using Green's formula accurate. (See below.) Figure 3 As shown, suppose the closed region D is bounded by a piecewise smooth curve L, and the functions P(x, y) and Q(x, y) have continuous first-order partial derivatives on D. The area is calculated using the following formula:

[0070]

[0071] Where L is the positively oriented boundary curve of D. Let P = -y, Q = x, then the area A can be obtained as...

[0072]

[0073] The area curve enclosed by the contour of the left ventricular endocardium in the echocardiographic video frame is shown below. Figure 4 As shown, noise can be observed in the area curve.

[0074] This invention provides two methods for automatically finding peak points:

[0075] The first method involves smoothing and filtering the area curve enclosed by the left ventricular endocardium contour to determine the extreme points; then calculating the index of these extreme points, and assigning an index t to any current extreme point. max_i or t min_i Find the local maximum value twice.

[0076] Specifically, the area curve signal is smoothed by filtering to find its extreme points. Filtering methods include, but are not limited to, median filtering, mean filtering, Gaussian filtering, low-pass filtering, convolution filtering, and polynomial fitting. Here, we first perform median filtering on the area curve signal, then mean filtering to obtain the smoothed curve. The default window width for median filtering is 3, and the default convolution kernel length for mean filtering is 5. The smoothed curve is shown below. Figure 5 As shown.

[0077] Then, calculate the sign change function of the first-order difference function of the smooth curve. Mathematically, the first-order difference represents the magnitude of the gradient change, and the sign change function of the first-order difference function can reflect the trend of the function at that point, as shown in the following formula.

[0078]

[0079] D(t)=G(t+1)*G(t),t=0,1,2,...,n-1;

[0080] extrema_indice(t)=where{D(t)<0};

[0081] Here, function f represents the smoothed left ventricular endocardial contour area curve signal, t represents the time frame of the echocardiogram video, and function G represents the sign change function of the first-order difference of function f, reflecting the trend of the function at point t. Function D represents the product of the sign change of the difference at the next time step after t and the sign change of the difference at time t. The index of the extreme point must appear at the position where function D is less than 0. The reason is that when two consecutive points show monotonic changes, the signs of functions G(t+1) and G(t) are the same, so D(t) must be greater than 0. Therefore, when an extreme value appears, the signs of G(t+1) and G(t) must be opposite, so D(t) must be less than 0. After obtaining the index of the extreme point, comparing the f values ​​of the first two extreme points can confirm the order of the maximum and minimum values ​​of the extreme points, because the maximum and minimum values ​​must alternate.

[0082]

[0083] Since the extreme point indices are obtained for a smooth function, the extreme point indices before smoothing need to be fine-tuned.

[0084] Therefore, for any current extreme point index t max_i or t min_i The formula for finding the local maximum value in the second quadratic method is as follows.

[0085] t max_i =argmax(F[t min_i-1 :t min_i ]), i = 0, 1, 2, ...;

[0086] t min_i =argmin(F[t max_i-1 :t max_i ]), i = 0, 1, 2, ...;

[0087] Here, the function F represents the curve of left ventricular endocardial area change before smoothing, and argmax represents the curve from the range t. min_i-1 to t min_i The index of the maximum value of the function F between t min_i-1 Represents t max_i The previous minimum index, t min_i Represents t max_iSubsequent minimum index. When i = 0, t min_i-1 =0; when t max_i When there is no local minimum, t min_i =len(F), which is the maximum index of the signal.

[0088] argmin represents the range t max_i-1 to t max_i The minimum index of the function F between t max_i-1 Represents t min_i The previous maximum index, t max_i Represents t min_i The subsequent maximum index. When i = 0, t max_i-1 =0; when t min_i When there is no local minimum, t max_i =len(F), which is the maximum index of the signal.

[0089] The second method: Use the AMPD algorithm to calculate the indices t of the maximum and minimum values ​​of the area curve function enclosed by the left ventricular endocardial contour. max_i and t min_i Calculate the maximum difference between the function F and the minimum values, and select the point pair with the maximum difference in the i-th cycle as the end-diastolic keyframe t for calculating the ejection fraction using the Simpson method. max_i and the key frame t at the end of the contraction period min_i .

[0090] Specifically, the AMPD algorithm, i.e., the automatic peak detection algorithm, is used to obtain the indices t of the maximum and minimum values ​​of the left ventricular endocardial area change curve function F. max_i and t min_i .

[0091] Finally, calculate the extreme point pairs (t) max_i , t min_i The maximum difference of the function F is used to select the point pair with the maximum difference in the i-th cycle as the end-diastolic keyframe t for calculating the ejection fraction using the Simpson method. max_i and the key frame t at the end of the contraction period min_i ,like Figure 4 As shown, the formula is as follows.

[0092] Opitmal(i) = argmax(|F(t) max_i )-F(t min_i )|).

[0093] Step 600: Calculate the biplane ejection fraction of keyframe images at end-diastole and end-systole of the left ventricle using the Simpson method.

[0094] This step is based on the Simpson method for calculating single / dual-plane ejection fraction. Step 4 obtains the end-diastolic and end-systolic keyframes from the left ventricular long-axis four-chamber and two-chamber views of the echocardiographic video used to calculate the single / dual-plane ejection fraction using the Simpson method. Therefore, there are four echocardiographic keyframe images that need to be calculated. Since the Simpson calculation method for the left ventricular endocardial contour is consistent across these four images, the left ventricular endocardial contour from the left ventricular long-axis four-chamber end-diastolic keyframe image is used as an example to illustrate the mathematical implementation principle of the rapid calculation of the Simpson method.

[0095] First, the single-frame image size of the left ventricular myocardial segmentation mentioned in step 2 is fixed at n*n, which means that the left ventricular endocardial contour point set {ci} extracted in step 3 must be on a scale of 224*224. Therefore, it is necessary to transform the n*n scale point set to the original echocardiographic scale. Assuming the original echocardiographic scale is h*w, the coordinate transformation formula for the left ventricular endocardial contour point set {Ci} at the original echocardiographic scale is as follows.

[0096]

[0097] like Figure 6 As shown, the set of path points {Ci} from point E to point P and then to point F represents the left ventricular endocardial contour of the left ventricular long-axis four-chamber end-diastolic keyframe image. Points E and F represent the two endpoints of the basal straight line segment of the endocardial contour, point M represents the midpoint of the basal straight line segment EF, point P represents the point farthest from point M in the point set on the endocardial contour, and point G represents the intersection point perpendicular to point F on the extension of ray PM.

[0098] The prerequisite for Simpson's method to calculate single / dual-plane ejection fraction is to first locate pairs of points on the 20 parallel line segments along the contour line. For ease of understanding, we will... Figure 6 The endocardium is temporarily positioned to observe the expression of segment n, such as... Figure 7 As shown, the principle of n segments is as follows: Divide the line segment PG into n equal segments parallel to the line segment GF, with each end of segment PG having a reserved distance of 0.5*L / n. Then, select n points at equal intervals L / n on the remaining segments. Next, find pairs of contour points on the endocardial contour parallel to the line segment GF, passing through these n points sequentially. This yields a segmented image with n nodes and n segments (the two reserved portions can be considered as one segment, with a combined length of L / n). In summary, we can... The position of each segment on the PG line segment, such as the position of the first node being 0.5*L / n, and the position of the i-th node being n. i The position is (i-0.5)*L / n, and when n=20, it is the 20th segment.

[0099] like Figure 8As shown, to quickly calculate the coordinates of the two endpoints of the 20 segments on the endocardial contour line, the vector projection method is first used to calculate the coordinates from point Ci to C′ on the endocardial contour line within the coordinate system with point P as the origin. i The transformation formula is as follows:

[0100]

[0101] in, Let represent the unit vector along the direction of PG ray, since points P and M are known, then the unit vector along the direction of PG ray is... It is known. The current coordinate system has P as its origin, and adding the offset of P's coordinates, then C' i The coordinate formula is as follows:

[0102]

[0103] C' i =C i ·u T ·u+P;

[0104] Obtain the set of PG ray projection points {C'} of the point set {Ci} on the contour line. i Afterwards, using point P as the boundary, the set of projected points is divided into segments PE and PF, denoted as the set of projected points {CE'}. i} and {CF' i Given the coordinates of point P, the formula for the n-segment point Ni on the PG ray is as follows:

[0105]

[0106] To quickly calculate the 20 segment point pairs on the contour line, we broadcast the Ni point to {CE'. i} and {CF' i On the set, compute the distance from point Ni to each {CE'}. i} and {CF' i The distance between {CE'}, where the distance includes but is not limited to Euclidean distance, is assumed to be {CE'}. i} and {CF' i The quantities of {CE'} are p and q respectively. Then, {CE'} i The distance broadcast matrix size is p*n, {CF' i The distance broadcast matrix size is q*n, where n = 20.

[0107] By finding the minimum value of the row vectors of the two broadcast matrices, we can quickly obtain the value in {CE' i} and {CF' i The set of the 20 points closest to point Ni, {CEN'} i} and {CFN' i}

[0108] It is worth noting that since the base segment EF is not guaranteed to be perpendicular to the base segment PM, a set of 20 points {CEN'} is obtained. i} and {CFN' i After that, it is necessary to determine whether there are any non-parallel line segments. Specifically:

[0109] Step 1: When 20 pairs of {CEN' are obtained i} and {CFN' i The set of endpoints of parallel line segments, i.e., each pair of CEN' i and CFN' i To represent a parallel line segment, first calculate the unit direction vector for each parallel line segment. The formula is as follows:

[0110]

[0111]

[0112] Then, respectively The median is calculated in the x and y directions because, since the base segment EF is not guaranteed to be perpendicular to the base segment PM, there may be a small number of parallel line segments that are not parallel, meaning that most parallel line segments are parallel. Therefore, the vector represented by the median in the x and y directions must be the standard parallel unit vector of the 20 parallel line segments.

[0113] Step 2: For 20 Calculation and The angle between them is calculated using the vector cosine formula, since... and Since they are all unit vectors, therefore:

[0114]

[0115] In theory, each and The cosine values ​​should all be equal to 1, meaning the included angle is 0 degrees. However, in reality, discrete mathematics introduces errors; therefore, we set the error angle to no more than 1 degree. The formula is as follows:

[0116]

[0117] when and When the cosine value is greater than cos(1°), it means that the line segment is a parallel line segment; otherwise, it means that the current point is parallel to CEN'. i and CFN' i They are non-parallel line segments.

[0118] Step 4: To accelerate the detection of parallel line segments, the detection direction is changed from 20 pairs {CEN' i} and {CFN' i The detection process begins by checking parallel line segments one by one from the base segment to the top parallel line. The rationale is that non-parallel line segments can only exist in the base segment, not in the top direction. Therefore, when the cosine value of the i-th parallel line segment is detected to be greater than cos(1°), the detection of parallel line segments stops.

[0119] If there are non-parallel line segments, then it represents the point CEN' i Or click CFN' i The correct position should be on the line segment EF. The correct point coordinates CEN' need to be calculated again using the following formula. i or CFN' i Since the line segment EF can only be tilted to one side relative to point P, therefore, if CEN' exists... i If there is an error, then the corresponding CFN' i This is correct, and vice versa. Therefore, for example, when CEN' i Error, given points P and M, correct CFN' i A straight line can be represented using point-normal form. From the point-normal form, the general equation of the straight line, A0x + B0y + C0 = 0, can be derived. Similarly, given points E and F, the general equation of the straight line, A1x + B1y + C1 = 0, can be constructed. Using these two equations, the formula for the intersection point of two intersecting straight lines can be calculated as follows:

[0120]

[0121] At this point, 20 pairs of endocardial contour points parallel to the GF straight line segment are obtained. The distance ai or bi can be calculated for each pair, using formulas including, but not limited to, Euclidean distance. Here, ai represents the distance of the 20 segment pairs of points at end-diastole or end-systole in the left ventricular four-chamber view, and bi represents the distance of the 20 segment pairs of points at end-diastole or end-systole in the left ventricular two-chamber view. Next, the single / dual-plane ejection fraction is calculated.

[0122] Case 1: Simpson's method for calculating biplane ejection fraction, i.e., the dataset contains cross-sectional images of the left ventricle in both four-chamber and two-chamber views.

[0123] like Figure 9 As shown, the portion between each segment of the left ventricle of the heart is simulated as an elliptical cylinder. The ai, bi, and L obtained in the above steps form three axes, where ai is the major axis of the elliptical cylinder, bi is the minor axis, and the length L / n of each segment is the height of the elliptical cylinder. The sum of n segments is the shape of the left ventricle of the heart.

[0124] Each segment of the left ventricle is simulated using an elliptical cylinder;

[0125] We obtain the total left ventricular volume (EDV) and ESV by first differentiating and then integrating, that is, by first calculating the estimated volume of each segment and then summing them up.

[0126] The volume of the simulated elliptical cylinder in the i-th segment is:

[0127]

[0128] The total volume of the left ventricle is:

[0129]

[0130] In our current algorithm, we set n=20. Based on the above formula, the left ventricular volume (EDV) and end-diastolic volume (ESV) can be calculated from the ED and ES frames of the heart at 4ch and 2ch.

[0131] Case 2: Single-plane ejection fraction, the dataset contains only one cross-sectional image of the left ventricle (four-chamber or two-chamber).

[0132] Based on the L and A calculated using the above steps, where A can be obtained using Green's formula to determine the left ventricular contour area value of the current end-diastolic or end-systolic frame, the left ventricular volume can be calculated:

[0133]

[0134] According to the above formula, the left ventricular volume EDV and ESV can be calculated from the ED and ES frames of any section in the heart 4ch and 2ch.

[0135] Finally, calculate the left ventricular ejection fraction (LVEF) and heart rate (HR).

[0136] 1) Calculate the left ventricular ejection fraction (LVEF):

[0137]

[0138] 2) Calculate heart rate (HR);

[0139] The algorithm's central tendency (HR) is calculated as follows:

[0140] Scenario 1: When only one cardiac cycle is detected in the video stream, the formula is as follows:

[0141]

[0142] In the formula, S and D represent the positions of the heart at the S and D frames during the adjacent end of systole and end of diastole, respectively. fps is the number of video frames captured by the device per second. If there is no specific data, fps is usually set to 30. The denominator of the intermediate formula represents the time required for one cardiac cycle in the video. The heart rate HR can be obtained by the final calculation.

[0143] Scenario 2: When multiple cardiac cycles are detected in the video stream, the formula is as follows:

[0144]

[0145] In the formula, Count(D) represents the number of D frames detected in the video stream, D_indexlast and D_indexlast represent the index of the last D frame and the index of the first D frame, fps is the number of video frames captured by the device per second. If there is no specific data, fps is usually set to 30. The denominator of the intermediate formula means the time required for one cardiac cycle in the video. The heart rate HR can be obtained by calculation.

[0146] like Figure 10 As shown, an automatic ejection fraction assessment device based on echocardiographic feature map filtering includes:

[0147] The acquisition module 11 is used to extract echocardiographic video data of any duration from a standard cross-section, wherein the quality of the echocardiographic video data meets the standard.

[0148] The identification module 12 is used to identify the data of the left ventricular long-axis four-chamber and two-chamber sections in the echocardiogram data using a pre-trained neural network segmentation model;

[0149] The calculation module 13 is used to filter the feature map on the time axis using the data of the left ventricular long axis four-chamber and two-chamber sections, as input to the segmentation prediction head and key point prediction head, and to calculate the myocardial segmentation map and the key points of the left ventricular basal segment.

[0150] Classification module 14 is used to classify the inner and outer contours of the left ventricular myocardium according to the myocardial segmentation diagram and key points of the left ventricular basal segment, and obtain the inner contour and outer contour of the left ventricular myocardium.

[0151] For example, based on the left ventricular myocardial segmentation results of the echocardiogram video data, a set of contour points in the same direction is extracted for each frame of the image; the distance from the key point of the left ventricular basal segment to each contour point of the myocardial segmentation is calculated sequentially to determine the closest contour point; the left ventricular myocardial contour is determined based on the closest contour point; wherein the key point of the left ventricular basal segment is located at the left and right ends of the left ventricular myocardial segmentation.

[0152] Keyframe module 15 is used to assess the cardiac cycle based on the left ventricular contour area curve of echocardiographic video data, and to determine keyframe images of the end-diastolic and end-systolic phases of the left ventricular long-axis four-chamber and two-chamber configurations; for example:

[0153] The contour area of ​​the left ventricular myocardium was calculated using Green's formula.

[0154] The area curve enclosed by the left ventricular endocardial contour of the echocardiogram video frame is obtained based on the contour area of ​​the left ventricular myocardial contour.

[0155] The peak value of the area curve enclosed by the endocardial contour of the left ventricle is calculated, and the peak value is used to determine the end-diastolic and end-systolic phases of the left ventricle in a four-chamber and a two-chamber configuration.

[0156] In one embodiment, the AMPD algorithm is used to calculate the indices t of the maximum and minimum values ​​of the area curve function enclosed by the left ventricular endocardial contour. max_i and t min_i ;

[0157] Calculate the maximum difference between the function F and the minimum values, and select the point pair with the maximum difference in the i-th cycle as the end-diastolic keyframe t for calculating the ejection fraction using the Simpson method. max_i and the key frame t at the end of the contraction period min_i .

[0158] In another embodiment, the extreme points are determined by smoothing and filtering the area curve enclosed by the contour of the left ventricular endocardium.

[0159] Calculate the index of the extreme point, for any current extreme point index t max_i or t min_i Find the local maximum value twice.

[0160] Evaluation module 16 is used to calculate the biplane ejection fraction of keyframe images of the end-diastolic and end-systolic phases of the left ventricle in a four-chamber and two-chamber long axis using the Simpson method.

[0161] The calculation module 13 includes a calculation submodule, which is used to process the echocardiogram video frames in batches and then segment the left ventricular myocardium. Each batch of echocardiogram image frames is processed by a pre-trained deep convolutional neural network for left ventricular myocardium segmentation to obtain the feature map of that batch of echocardiograms. The feature maps of the echocardiograms are processed in batches according to the time axis, and the feature maps of the echocardiograms are filtered with a width m in the time axis direction.

[0162] The present invention also provides an electronic device, comprising:

[0163] At least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the above-described automatic ejection fraction assessment method based on echocardiographic feature map filtering.

[0164] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described automatic ejection fraction assessment method based on echocardiographic feature map filtering.

[0165] It is understood that computer-readable storage media can include: any entity or device capable of carrying computer programs, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. Computer programs include computer program code. Computer program code can be in the form of source code, object code, executable files, or certain intermediate forms, etc. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0166] In some embodiments of the present invention, the device may include a controller, which is a microcontroller chip integrating a processor, memory, communication module, etc. The processor may refer to the processor included in the controller. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0167] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0169] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automatic ejection fraction assessment method based on echocardiographic feature map filtering, characterized in that, include: Echocardiographic video data of arbitrary duration extracted from a standard cross-section, wherein the image quality of the echocardiographic video data meets the standard; A pre-trained neural network segmentation model was used to identify data from the left ventricular long-axis four-chamber and two-chamber sections in the echocardiographic video data. After the echocardiogram video is processed in batches by frame extraction, the left ventricular myocardium is segmented. Each batch of echocardiogram image frames is processed by encoding and decoding of a pre-trained deep convolutional neural network for left ventricular myocardial segmentation to obtain the feature map of that batch of echocardiograms. The echocardiogram feature maps are batch-processed according to the time axis. The echocardiogram feature maps are filtered with a width m in the time axis direction. The filtered feature maps are used as inputs to the segmentation prediction head and the key point prediction head, and the myocardial segmentation map and the key points of the left ventricular basal segment are calculated. Based on the myocardial segmentation diagram and key points of the left ventricular basal segment, the inner and outer contours of the left ventricular myocardium are classified to obtain the inner and outer contours of the left ventricular myocardium. The area curve enclosed by the left ventricular endocardial contour of the echocardiogram video data is obtained based on the contour area of ​​the left ventricular myocardial contour. Calculate the peak value of the area curve enclosed by the left ventricular endocardial contour, determine the cardiac cycle through the peak value, and determine the key frame images of the end-diastolic and end-systolic phases of the left ventricular long axis four-chamber and two-chamber hearts. The biplane ejection fraction of keyframe images at end-diastole and end-systole of the left ventricle in a four-chamber and two-chamber configuration along the long axis was calculated using the Simpson method. The peak value of the area curve enclosed by the contour of the left ventricular endocardium includes: The AMPD algorithm is used to calculate the indices of the maxima and minima of the area curve function bounded by the left ventricular endocardial contour. and ; Calculate the maximum difference between the function F that has the maximum and minimum values, and select the point pair with the maximum difference in the i-th cycle as the end-diastolic keyframe and end-systolic keyframe for calculating the ejection fraction using the Simpson method. Or, The extreme points are determined by smoothing the area curve enclosed by the contour of the left ventricular endocardium. Calculate the index of the extreme point, and find the local maximum value twice for any current extreme point index.

2. The automatic ejection fraction assessment method based on echocardiographic feature map filtering according to claim 1, characterized in that, The classification of the inner and outer contours of the left ventricular myocardium based on the myocardial segmentation map and key points of the left ventricular basal segment to obtain the inner and outer contours of the left ventricular myocardium includes: Based on the left ventricular myocardial segmentation results of the echocardiogram video data, a set of contour points in the same direction is extracted for each frame of the image; Calculate the distances from the key points of the left ventricular basal segment to each contour point of the myocardial segmentation map in sequence, and determine the contour point with the closest distance; The left ventricular myocardial contour is determined based on the nearest contour point; wherein the key points of the left ventricular basal segment are located at the left and right ends of the left ventricular myocardial segment.

3. An automatic ejection fraction assessment device based on echocardiographic feature map filtering, characterized in that, include: The acquisition module is used to extract echocardiographic video data of any duration from a standard cross-section, and the quality of the echocardiographic video data meets the standard. The identification module is used to identify the data of the left ventricular long-axis four-chamber and two-chamber sections in the echocardiogram video data using a pre-trained neural network segmentation model; The calculation module is used to process the echocardiogram video frames in batches and then segment the left ventricular myocardium. Each batch of echocardiogram image frames is encoded and decoded by a pre-trained deep convolutional neural network for left ventricular myocardial segmentation to obtain the feature map of that batch of echocardiograms. The feature maps of the echocardiograms are processed in batches according to the time axis. The feature maps of the echocardiograms are filtered in the time axis direction with a width m. The filtered feature maps are used as inputs to the segmentation prediction head and the key point prediction head, and the myocardial segmentation map and the key points of the left ventricular basal segment are calculated. The classification module is used to classify the inner and outer contours of the left ventricular myocardium based on the myocardial segmentation diagram and key points of the left ventricular basal segment, thereby obtaining the inner and outer contours of the left ventricular myocardium. The keyframe module is used to obtain the area curve enclosed by the left ventricular endocardial contour of the echocardiogram video data based on the contour area of ​​the left ventricular myocardial contour; calculate the peak value of the area curve enclosed by the left ventricular endocardial contour, determine the cardiac cycle through the peak value, and determine the keyframe images of the end-diastolic and end-systolic phases of the left ventricular long-axis four-chamber and two-chamber hearts. The peak value of the area curve enclosed by the contour of the left ventricular endocardium includes: The AMPD algorithm is used to calculate the indices of the maxima and minima of the area curve function bounded by the left ventricular endocardial contour. and ; Calculate the maximum difference between the function F with maximum and minimum values, and select the point pair with the maximum difference in the i-th cycle as the end-diastolic keyframe and end-systolic keyframe for calculating the ejection fraction using the Simpson method. Or, The area curve enclosed by the contour of the left ventricular endocardium is smoothed and filtered to determine the extreme points; the index of the extreme points is calculated, and the local maximum value is obtained for any current extreme point index or by quadratic calculation. The evaluation module is used to calculate the biplane ejection fraction of keyframe images at end-diastole and end-systole of the left ventricle using the Simpson method.

4. An electronic device, characterized in that, include: At least one processor; And at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the automatic ejection fraction assessment method based on echocardiographic feature map filtering as described in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a computer, performs the automatic ejection fraction assessment method based on echocardiographic feature map filtering as described in any one of claims 1 to 2.

Images