Method, system and equipment for automatically measuring cardiac muscle full-segment thickness in echocardiogram

Abstract

The invention relates to the technical field of medical imaging, and provides an automatic measurement method, system and equipment for the cardiac muscle full segment thickness of an echocardiogram, and the method comprises the steps: determining a sector rectangular frame in an ultrasonic video stream through employing a deep learning target detection model, so as to determine a cutting region; cutting the ultrasonic video stream based on the cutting area, and scaling the cut ultrasonic cardiac image to a target size in an equal proportion; performing feature extraction on the echocardiography image through a feature extraction module and generating a feature map; cardiac muscle segmentation is completed through the cardiac muscle segmentation module, and mitral valve endpoint recognition is completed through the mitral valve endpoint recognition module; based on the data obtained by the myocardial segmentation module and the mitral valve endpoint identification module, a post-processing module is adopted to calculate and visually output a myocardial full segment thickness result at the end of diastole or end of systole; the cardiac muscle full segment thickness can be effectively measured, the accuracy and repeatability of cardiac function parameter measurement can be improved, and the medical examination process is optimized.

Description

Technical Field

[0001] The present application relates to the field of medical imaging technology, and more specifically, to an automatic measurement method, system and device for the thickness of the entire myocardial segment using echocardiography. Background Art

[0002] Echocardiography is one of the key means to evaluate cardiac function. The measurement of the thickness of the ventricular septum and the posterior wall of the left ventricle based on the long axis section of the left ventricle in echocardiography is of vital importance for examining the heart, especially hypertrophic myocardium, and is an indispensable routine indicator in echocardiography. Traditional measurements of cardiac function parameters such as the thickness of the ventricular septum and the posterior wall of the left ventricle are completely dependent on the operation of ultrasound doctors, which has the limitations of strong subjectivity, low efficiency, and susceptibility to the influence of operator experience. The current myocardial thickness measurement method is mostly "key point prediction and distance calculation". This technology is generally based on the assumption that the myocardial thickness is relatively uniform. It is necessary to predict two key points on both sides of the myocardium in the echocardiogram, and then determine the myocardial thickness by calculating the distance between the two points. However, in clinical practice, myocardial thickening often presents an uneven state, which makes the myocardial thickness measurement based on key points only reflect the thickness of a specific myocardial segment, and it is difficult to achieve a comprehensive quantification of the overall myocardial thickness.

[0003] In the prior art, the method for measuring myocardial thickness based on key points is, for example, a patent application document with publication number CN118429269A, entitled "Method for measuring the wall thickness of the left ventricle, ultrasonic measurement device and its measurement method". This method mainly measures the wall thickness of the left ventricle on three short-axis sections of the echocardiogram, but does not cover the measurement of the ventricular septum on the parasternal long-axis section of the left ventricle, and does not have the ability to comprehensively measure the thickness of the entire myocardium. Another example is a patent application document with publication number CN115775233A, entitled "A processing method and device for measuring characteristic dimensions based on cardiac ultrasonic video". This method mainly achieves measurement by predicting the positions of key points on both sides of the myocardium, but there is still obvious uncertainty in the positioning standard of the key points.

[0004] In addition, in practical applications, in order to obtain the myocardial thickness data of the entire cardiac cycle, especially the end-systole and end-diastole, it is also necessary to clarify the principles for determining the cardiac cycle, but these existing methods do not provide relevant effective guidance.

[0005] Application Contents

[0006] The technical problem to be solved by the present application is to provide a method, system and device for automatically measuring the thickness of the entire segment of the myocardium by echocardiography in view of the above-mentioned defects of the prior art.

[0007] The technical solution adopted by this application to solve its technical problem is:

[0008] In one aspect, the present application provides an automatic method for measuring the thickness of the entire myocardial segment using echocardiography, the method comprising:

[0009] Determine the ultrasound sector area: obtain an ultrasound video stream containing cardiac structure information, and use a deep learning target detection model to automatically identify and determine a sector rectangular frame in the ultrasound video stream to determine a cropping area;

[0010] Preprocessing the ultrasound video signal: based on the determined clipping area of ​​the sector rectangular frame, clipping the ultrasound video stream, and scaling the clipped ultrasound cardiac image to a target size in proportion;

[0011] Training the deep learning target detection model: the deep learning target detection model includes a feature extraction module, a myocardial segmentation module and a mitral valve endpoint recognition module; the feature extraction module performs feature extraction on the preprocessed ultrasound cardiac image and generates a feature map; based on the feature map, the myocardial segmentation module performs myocardial segmentation, and the mitral valve endpoint recognition module performs mitral valve endpoint recognition;

[0012] Model output post-processing: Based on the data obtained by training the myocardial segmentation module and the mitral valve endpoint recognition module, a post-processing module is used to calculate and visualize the output of the myocardial full segment thickness results at the end of diastole or end of systole.

[0013] In some embodiments, the method of automatically identifying and determining a sector rectangular frame in the ultrasound video stream using a deep learning target detection model to determine a clipping area includes:

[0014] Obtaining the vertex coordinates of the sector rectangular frame of the first 10 frames of the ultrasound video stream;

[0015] An average value is calculated for the vertex coordinates of each of the fan-shaped rectangular frames, and a clipping area is determined based on the calculation result of the average value.

[0016] In some embodiments, the step of cropping the ultrasound video stream based on the determined cropping area of ​​the sector rectangular frame and scaling the cropped ultrasound cardiogram to a target size includes:

[0017] The ultrasound video stream is cropped according to the coordinates of the cropping area, and the cropped ultrasound image is proportionally scaled to a target size using a bilinear interpolation method, and during the scaling process, a narrow edge of the ultrasound image is filled with a black background;

[0018] The pixel value of the ultrasound cardiology image of the target size is divided by the maximum pixel value to normalize the pixel value to a 32-bit floating point number between 0 and 1.

[0019] In some embodiments, extracting features from the preprocessed ultrasound cardiac image and generating a feature map by the feature extraction module includes:

[0020] The feature extraction module is used to extract features from the preprocessed ultrasound cardiac image and generate multi-layer feature maps at different sampling rates;

[0021] The feature extraction module adopts a three-layer feature pyramid architecture;

[0022] Multiple layers of feature maps are merged based on the three-layer feature pyramid architecture to extract multi-scale image features and form an output feature map.

[0023] In some embodiments, the myocardial segmentation is performed by the myocardial segmentation module based on the feature map, including:

[0024] enabling the myocardial segmentation module to couple the first target detection task and the segmentation task;

[0025] The output feature map provided by the feature extraction module is used as input to perform multi-layer deconvolution and bilinear interpolation upsampling calculations to obtain a myocardial contour image;

[0026] Based on the myocardial contour image, locating the myocardial contour and generating a bounding box according to the first target detection task;

[0027] According to the segmentation task, a segmentation mask is generated by a segmentation head based on upsampling and convolution operations to segment pixels inside the bounding box for detection.

[0028] In some embodiments, the step of completing mitral valve endpoint identification by the mitral valve endpoint identification module includes:

[0029] The mitral valve endpoint recognition module is coupled with a second target detection task and a key point recognition task;

[0030] Using the output feature map provided by the feature extraction module as input, positioning the entire mitral valve according to the second target detection task to determine its location;

[0031] When the mitral valve leaflet region is successfully identified, the focus is placed on the interior of the mitral valve leaflet region, and the mitral valve endpoint is identified and determined according to the key point identification task.

[0032] In some embodiments, the post-processing module is used to calculate and visualize the end-diastolic or end-systolic myocardial full segment thickness results, including:

[0033] The segmentation result provided by the myocardial segmentation module is used as input to calculate the myocardial center curve and the myocardial outer contour line, and obtain the coordinates of the myocardial center curve and the coordinates of the myocardial segmentation outer contour points;

[0034] Based on the myocardial center curve, multiple segments of myocardial center straight lines are fitted by piecewise linear fitting;

[0035] Determine the vertical line based on the slope and coordinates of each segment of the myocardial center straight line, calculate the distance between the vertical line and the two intersection points of the upper and lower contour lines of the myocardium, and the distance is the myocardial thickness of the corresponding straight line segment; arrange the segment thicknesses corresponding to all the myocardial center straight lines in sequence to obtain the thickness distribution of the entire myocardial segment;

[0036] Obtaining the vertical distance between the mitral valve endpoint and the myocardial center straight line in the single frame of the ultrasonic cardiogram by calculating the corresponding vertical distance between the mitral valve endpoint and the myocardial center straight line;

[0037] Draw a distance curve that changes with time according to the vertical distance between the mitral valve endpoint and the myocardial center straight line in each frame of the entire ultrasound video stream, and use a Kalman filter method to smooth the distance curve;

[0038] Based on the smoothed distance curve, the end diastole or end systole of the cardiac cycle is determined by analyzing its maximum or minimum value, and the myocardial full segment thickness result at the end diastole or end systole is visually output.

[0039] In some embodiments, the feature extraction module includes multiple convolutional neural network layers, linear neural network layers and convolutional attention layers; wherein the convolutional attention layer uses a PSA module to process the input feature map in parallel through multiple branches, and each of the branches is composed of convolutional layers with convolution kernels of different sizes.

[0040] On the other hand, the present application also provides an automatic measurement system for the whole segment thickness of myocardium using echocardiography, which adopts the automatic measurement method for the whole segment thickness of myocardium using echocardiography as described in any of the above items.

[0041] On the other hand, the present application also provides an automatic measurement device for the whole segment thickness of myocardium by echocardiography, including the automatic measurement system for the whole segment thickness of myocardium by echocardiography as described above.

[0042] The beneficial effects of the present application are as follows: Different from the prior art, the present application's method for automatically measuring the thickness of the whole segment of myocardium in an ultrasound cardiogram obtains an ultrasound video stream containing cardiac structure information, and uses a deep learning target detection model to automatically identify and determine a fan-shaped rectangular frame in the ultrasound video stream to determine a clipping area; based on the determined clipping area of ​​the fan-shaped rectangular frame, the ultrasound video stream is clipped, and the clipped ultrasound cardiogram is proportionally scaled to a target size; the deep learning target detection model includes a feature extraction module, a myocardial segmentation module, and a mitral valve endpoint recognition module; the feature extraction module extracts features from the preprocessed ultrasound cardiogram and generates a feature map; based on the feature map, the myocardial segmentation module is used to complete myocardial segmentation, and the mitral valve endpoint recognition module is used to complete mitral valve endpoint recognition; based on the data obtained by training the myocardial segmentation module and the mitral valve endpoint recognition module, a post-processing module is used to calculate and visualize the output of the whole segment thickness of the myocardium at the end of diastole or end of systole; the effective measurement of the whole segment thickness of the myocardium can be achieved, which helps to improve the accuracy and repeatability of the measurement of cardiac function parameters and optimize the medical examination process. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of the automatic measurement method of the whole segment thickness of the myocardium by echocardiography in the embodiment of the present application;

[0044] Figure 2 Schematic diagram of the application of the three-layer feature pyramid architecture of the feature extraction module in the embodiment of the present application;

[0045] Figure 3 is a schematic diagram of the target detection function realized by the myocardial segmentation module and the mitral valve endpoint recognition module in the embodiment of the present application;

[0046] Figure 4 is a schematic diagram of the connection relationship between the feature extraction module, the myocardial segmentation module and the mitral valve endpoint recognition module in the embodiment of the present application;

[0047] Figure 5 It is a schematic diagram of the overall architecture for realizing the method for automatically measuring the thickness of the entire segment of the myocardium by echocardiography in an embodiment of the present application;

[0048] Figure 6 It is a schematic diagram of the hardware relationship for realizing the automatic measurement method of the whole segment thickness of the myocardium by echocardiography in the embodiment of the present application. DETAILED DESCRIPTION

[0049] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] "Multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0052] Moreover, the terms "up, down, front, back, left, right, upper end, lower end" etc. indicating directions are all based on the posture and position of the device or equipment described in this solution during normal use.

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present application.

[0054] Embodiment 1: The embodiment of the present application provides an automatic measurement method for the thickness of the entire segment of myocardium by ultrasound. The automatic measurement method for the thickness of the entire segment of myocardium by ultrasound is based on deep learning model technology. It obtains the thickness of the entire segment and displays the thickened area by accurately segmenting the target myocardial segment in the ultrasound cardiogram. In addition, the automatic measurement method for the thickness of the entire segment of myocardium by ultrasound also accurately determines whether the cardiac cycle is in the end-diastole or end-systole by identifying the mitral valve and the mitral valve endpoint in the ultrasound cardiogram, calculating the vertical distance between the mitral valve endpoint and the myocardial segment, and obtaining the results of the thickness of the entire segment of myocardium corresponding to the end-diastole and end-systole. The automatic measurement of the thickness of the entire segment of myocardium by ultrasound can be accurately realized without the assumption that the thickness of the myocardium is relatively uniform, and is not affected by whether the thickness of the myocardium is uniform. It has significant advantages in applications involving myocardial thickness measurement.

[0055] The implementation process of the method for automatically measuring the thickness of the entire segment of myocardium by echocardiography in the embodiment of the present application is as follows: Figure 1 For details, please refer to the following steps S1-S4:

[0056] S1: Determine the ultrasound sector area: Obtain an ultrasound video stream containing cardiac structure information, and use a deep learning target detection model to automatically identify and determine the sector rectangular frame in the ultrasound video stream to determine the cropping area;

[0057] In this step, real-time ultrasound video data is mainly obtained from a data source, such as a data transmission port connected to an ultrasound system or ultrasound equipment. The sector rectangular frame is the smallest circumscribed rectangular frame containing the ultrasound sector; in order to achieve desensitization processing of the image / video data and provide ultrasound sector information containing only the cardiac structure for subsequent calculations, the embodiment of the present application adopts a deep learning target detection model to automatically identify and determine the smallest circumscribed rectangular frame containing the ultrasound sector.

[0058] In this step, the deep learning target detection model is used to automatically identify and determine the sector rectangle frame in the ultrasound video stream to determine the specific steps of the cropping area as follows, see steps S10 and S11:

[0059] S10: Obtaining the vertex coordinates of the sector rectangle frame of the first 10 frames of the ultrasound video stream;

[0060] S11: Calculate the average value of the vertex coordinates of each sector rectangular frame, and determine the clipping area based on the calculation result of the average value.

[0061] To ensure computational efficiency and accuracy, the embodiment of the present application only detects and calculates the sector rectangle frame for the first 10 frames of the ultrasound video stream. Then the vertex coordinates of the sector rectangle frame obtained in these 10 frames are averaged to determine the final cropping area, and subsequent ultrasound videos are cropped based on this area. In this way, the heart area in the ultrasound cardiogram can be accurately determined, thereby providing high-quality data support for subsequent analysis steps.

[0062] In this step, the available deep learning target detection models include but are not limited to YOLO v6-11, YOLO-X series models, and FastCNN.

[0063] S2: preprocessing the ultrasound video signal: based on the determined clipping area of ​​the sector rectangular frame, clipping the ultrasound video stream, and scaling the clipped ultrasound cardiac image to a target size in proportion;

[0064] In this step, based on the determined clipping area of ​​the sector rectangular frame, the ultrasound video stream is clipped, and the clipped ultrasound cardiogram is proportionally scaled to the target size. The specific steps are as follows, see steps S20 and S21:

[0065] S20: cropping the ultrasound video stream according to the coordinates of the cropping area, and scaling the cropped ultrasound image to a target size in proportion using a bilinear interpolation method. During the scaling process, a black background is used to fill the narrow edge of the ultrasound image.

[0066] S21: Divide the pixel value of the ultrasound cardiology image of the target size by the maximum pixel value to normalize it to a 32-bit floating point number between 0 and 1.

[0067] Based on the coordinates of the sector rectangle determined in step S1, the ultrasound video stream is precisely cropped, and then the cropped ultrasound image is proportionally scaled to the target size, specifically 448×448 or 512×512 pixels, using the bilinear interpolation method. To ensure that the ultrasound image maintains the original aspect ratio and does not lose important information during the scaling process, the narrow edge of the ultrasound image is filled with a black background with a pixel value of (0,0,0), and finally the pixel value of the ultrasound image of the target size is divided by the maximum pixel value of 255 to normalize it to a 32-bit floating point number between 0 and 1 for subsequent processing and analysis.

[0068] S3: Training a deep learning target detection model: The deep learning target detection model includes a feature extraction module, a myocardial segmentation module, and a mitral valve endpoint recognition module; the feature extraction module extracts features from the preprocessed ultrasound cardiac image and generates a feature map; based on the feature map, the myocardial segmentation module performs myocardial segmentation, and the mitral valve endpoint recognition module performs mitral valve endpoint recognition;

[0069] Among them, the feature extraction module, myocardial segmentation module and mitral valve endpoint recognition module are deep learning models that are trained simultaneously. During the deep learning target detection model training process, these three modules work together to perform deep learning training to achieve efficient multi-task processing.

[0070] In this step, the specific steps of extracting features from the preprocessed ultrasound cardiac image and generating a feature map by the feature extraction module are as follows, see steps S30-S32:

[0071] S30: extracting features from the preprocessed ultrasound cardiac image through a feature extraction module, and generating multi-layer feature maps at different sampling rates;

[0072] S31: Make the feature extraction module adopt a three-layer feature pyramid architecture;

[0073] S32: Merge multiple layers of feature maps based on a three-layer feature pyramid architecture to extract multi-scale image features and form an output feature map.

[0074] The feature extraction module is an image feature extraction structure that includes multiple convolutional neural network layers, linear neural network layers, and convolutional attention layers. It is responsible for extracting effective features from ultrasound images. It should be noted that the convolutional attention layer uses the Pyramid Squeeze Attention (PSA) module to process the input feature map in parallel through multiple branches. Each branch is composed of convolutional layers with different sizes of convolution kernels (such as 3×3, 5×5, 7×7, etc.). The feature map processed by each branch is weighted, the attention weight is calculated, and finally the weighted, multi-scale feature maps are merged to form the final output feature map. The convolutional attention layer can adaptively adjust the attention to different features, thereby improving the adaptability of the model to multi-scale targets.

[0075] In order to extract multi-scale image features, the present embodiment uses a three-layer feature pyramid architecture for the feature extraction module, such as Figure 2 As shown in the figure, the feature maps at different sampling rates (such as 8x, 16x, and 32x sampling rates) are merged through a three-layer feature pyramid architecture, providing a basis for subsequent module processing. The three-layer feature pyramid architecture allows the model to capture details at different resolutions, which is particularly important for the subsequent collaborative training of different tasks such as myocardial segmentation and mitral valve endpoint recognition.

[0076] In this step, the specific steps of completing myocardial segmentation by the myocardial segmentation module based on the feature map are as follows, see steps S33-S36:

[0077] S33: enabling the myocardial segmentation module to couple the first target detection task and the segmentation task;

[0078] S34: taking the output feature map provided by the feature extraction module as input, performing multi-layer deconvolution and bilinear interpolation upsampling calculation to obtain a myocardial contour image;

[0079] S35: Based on the myocardial contour image, locate the myocardial contour according to the first target detection task and generate a bounding box;

[0080] S36: According to the segmentation task, a segmentation mask is generated through the segmentation head based on upsampling and convolution operations to segment the pixels inside the bounding box for detection.

[0081] The myocardial segmentation module is used to segment the myocardial area in the ultrasound cardiac image, such as the interventricular septum, the posterior wall of the left ventricle, etc. The myocardial segmentation module extends the segmentation function based on target detection by coupling the target detection and segmentation tasks. That is, the myocardial segmentation module can locate the target of the myocardial contour according to the first target detection task and generate a bounding box; at the same time, the myocardial segmentation module also generates segmentation masks through the segmentation head (Segmentation Head), and these segmentation masks are generated through upsampling and convolution operations. The myocardial segmentation module used in the embodiment of the present application only segments the pixels inside the bounding box (i.e., the detection box) used for detection, such as Figure 3 As shown in Figure A, Figure 3 Figure A in the middle is a schematic diagram of the target detection function implemented by the myocardial segmentation module. In this way, the mis-segmentation of background or irrelevant areas is avoided, the accuracy of target detection is effectively utilized, the generation of segmentation masks is ensured to be focused on the target object, and the false positive problem that may occur in traditional segmentation models (such as U-Net) is effectively avoided.

[0082] The loss function for myocardial segmentation module training is composed of the target detection box loss (Box loss), segmentation loss (Segmentation loss), category loss (Class loss) and distributed focal loss (Distributed Focalloss). Among them, the target detection box loss (Box loss) is used to calculate the overlapping area between the target detection box and the true box to measure the degree of alignment. The segmentation loss (Segmentation loss) is used to calculate the difference between the probability distribution of the pixel point and the true label to ensure that the model can accurately assign the correct category to each pixel in the target detection box. The class loss (Class loss) is used to measure the difference between the target detection box and the true box to ensure that the model can correctly predict the category of the target. The distributed focal loss (Distributed Focal loss) is used to evaluate the regression accuracy of the coordinates between the target detection box and the true box, and further improve the model's ability to predict the target position by optimizing the positioning of the bounding box.

[0083] In this step, the specific steps of completing the mitral valve endpoint recognition by the mitral valve endpoint recognition module are as follows, see steps S37-S39:

[0084] S37: coupling the mitral valve endpoint recognition module with the second target detection task and the key point recognition task;

[0085] S38: using the output feature map provided by the feature extraction module as input, positioning the entire mitral valve according to the second target detection task to determine its location;

[0086] S39: When the mitral valve leaflet region is successfully identified, focus on the inside of the mitral valve leaflet region, and identify and determine the mitral valve endpoint according to the key point recognition task.

[0087] In this step, the mitral valve leaflet area, such as the mitral valve anterior leaflet area, when the mitral valve anterior leaflet area is successfully identified, the focus is on the interior of the mitral valve anterior leaflet area, and the mitral valve anterior leaflet endpoint in the mitral valve anterior leaflet area is identified and determined according to the key point recognition task, so as to accurately complete the thickness calculation of the corresponding segment in the subsequent process.

[0088] The mitral valve endpoint recognition module adopts a strategy of coupling target detection with key point recognition, that is, after detecting the target of the mitral valve leaflet region according to the second target detection task, the key point recognition task is then used to perform key point recognition only inside the detection frame, which helps ensure that the module can focus on processing the target object and avoid misidentification of the background or irrelevant areas, such as Figure 3 As shown in Figure B, Figure 3 Figure B is a schematic diagram of the mitral valve endpoint recognition module to achieve the target detection function. In this way, the mitral valve endpoint recognition module can effectively reduce the interference of background noise and target occlusion, and significantly improve the accuracy and robustness of key point recognition. Finally, the mitral valve endpoint recognition module processes the feature map through a multi-layer convolutional neural network and a fully connected layer to complete the regression of the key point coordinates.

[0089] The loss function for training the mitral valve endpoint recognition module is composed of the target detection box loss (Box loss), class loss (Class loss), distributed focal loss (Distributed Focal loss) and keypoint similarity loss (Keypoint Object Similarity loss). Among them, the functions of the target detection box loss (Box loss), class loss (Class loss), and distributed focal loss (Distributed Focal loss) can be referred to the above description and will not be repeated here; the keypoint similarity loss (Keypoint Object Similarity loss) is calculated based on ObjectKeypoint Similarity (OKS), which is an indicator to measure the similarity between the predicted keypoints and the real keypoints. OKS comprehensively evaluates the matching degree of the keypoints by considering the Euclidean distance of the keypoints, the target scale, and the importance weight of the keypoints. Its calculation formula is:

[0090]

[0091] Among them, di is the Euclidean distance between the predicted keypoint and the true keypoint; s is the scale parameter of the target; ki is the weight of the keypoint; Nkpts is the total number of keypoints.

[0092] By adopting OKS as the loss function, the model is able to regress the locations of keypoints more accurately while adapting to different object scales and differences in the importance of keypoints.

[0093] It should be noted that the implementation order of steps S33-S36 and steps S37-S39 is not limited by the order of the step numbers, and the two tasks of myocardial segmentation and mitral valve endpoint identification can be implemented simultaneously.

[0094] Specifically, in this embodiment, the feature extraction module first extracts features from the preprocessed ultrasound images to generate feature maps. These feature maps are then shared by the myocardial segmentation module and the mitral valve endpoint recognition module to complete the myocardial segmentation and mitral valve endpoint recognition tasks, respectively. The relationship between these three modules and the data flow are as follows: Figure 4 As shown in Figure 2. Through this design, the two different tasks of myocardial segmentation and mitral valve endpoint recognition can share the same feature map, which significantly reduces the number of model parameters and effectively speeds up the training process.

[0095] It is understandable that since each deep learning model is composed of many layers of networks, the feature extraction module, myocardial segmentation module and mitral valve endpoint recognition module also together form a large deep learning model.

[0096] S4: Model output post-processing: Based on the data obtained from the training of the myocardial segmentation module and the mitral valve endpoint recognition module, the post-processing module is used to calculate and visualize the output of the myocardial full-segment thickness results at the end of diastole or end of systole.

[0097] Among them, the post-processing module is a non-deep learning module, which takes the output of the myocardial segmentation module and the mitral valve endpoint recognition module as input. The post-processing module further performs subsequent calculations and data processing based on the precise information provided by the myocardial segmentation module and the mitral valve endpoint recognition module. Through a series of complex calculation steps, the post-processing module can accurately output the full segment thickness of the myocardium at the end of diastole or end of systole, and display the results in a visual form.

[0098] In this step, the specific steps of using the post-processing module to calculate and visualize the results of the myocardial full segment thickness at the end of diastole or end of systole are as follows, see steps S40-S45:

[0099] S40: taking the segmentation result provided by the myocardial segmentation module as input, calculating the myocardial center curve and the myocardial outer contour line, and obtaining the coordinates of the myocardial center curve and the coordinates of the myocardial segmentation outer contour points;

[0100] The calculation of the myocardial center curve uses an image thinning algorithm, such as the "Zhang-Suen thinning algorithm" and the "Lee-TC skeleton algorithm", so as to calculate the coordinates of the myocardial center curve according to the myocardial segmentation contour. The myocardial outer contour line is determined by a contour extraction algorithm, such as the findContours function in the OpenCV image processing library, to obtain the coordinates of the myocardial segmentation outer contour points.

[0101] This step can effectively ensure that the baseline and boundary definitions for subsequent measurements are accurate, providing a basis for subsequent thickness calculations.

[0102] S41: Based on the myocardial center curve, multiple segments of myocardial center straight lines are fitted by piecewise linear fitting;

[0103] In this step, the curve-to-straight line conversion is performed by using a "piecewise linear fitting" method; available algorithms include piecewise least squares fitting, thereby simplifying a complex curve into multiple straight line segments to facilitate subsequent calculations of myocardial thickness and vertical distance.

[0104] S42: determining a perpendicular line based on the slope and coordinates of each myocardial center straight line, calculating the distance between the perpendicular line and the two intersection points of the upper and lower contour lines of the myocardium, the distance being the myocardial thickness of the corresponding straight line segment; arranging the segment thicknesses corresponding to all myocardial center straight lines in sequence to obtain the thickness distribution of the entire myocardial segment;

[0105] Through this calculation, preliminary data of myocardial thickness can be obtained, providing a basis for subsequent cardiac cycle analysis.

[0106] S43: Obtaining the vertical distance between the mitral valve endpoint and the myocardial center straight line in the single-frame ultrasound cardiac image by calculating the vertical distance between the corresponding mitral valve endpoint and the myocardial center straight line;

[0107] Calculation of this distance helps to effectively distinguish between end-diastole and end-systole when determining the phase of the cardiac cycle.

[0108] S44: drawing a distance curve that changes with time according to the vertical distance between the mitral valve endpoint and the myocardial center straight line in each frame of the entire ultrasound video stream, and smoothing the distance curve using a Kalman filter method;

[0109] This curve reflects the dynamic change of the distance between the mitral valve endpoint and the target myocardium during the cardiac cycle. Kalman filtering is an efficient self-recursive filter that can effectively remove noise from the curve, making the curve smoother and easier to analyze later.

[0110] S45: Based on the distance curve after smoothing, the end diastole or end systole of the cardiac cycle is determined by analyzing the maximum value or the minimum value thereof, and the result of the whole segment thickness of the myocardium at the end diastole or the end systole is output visually.

[0111] This step determines the cardiac cycle and ensures the accuracy of the myocardial thickness measurement results. For example, the frame with the closest distance between the mitral valve endpoint and the ventricular septum is the end-diastole, and the frame with the closest distance to the left ventricular posterior wall is the end-systole. The myocardial full-segment thickness results at the end-diastole and end-systole are visualized. Through intuitive image display, the operator can more clearly understand the changes in myocardial thickness.

[0112] Through the above steps, the post-processing module not only realizes the accurate measurement of myocardial thickness, but also provides comprehensive and efficient support for clinical applications through the judgment and visualization of the cardiac cycle.

[0113] The overall architecture design adopted in the embodiment of the present application is as follows Figure 5 As shown, the embodiment of the present application adopts an analysis architecture that integrates a feature extraction module, a myocardial segmentation module, a mitral valve endpoint recognition module and a post-processing module, and jointly realizes the two major functions of myocardial full-stage measurement and cardiac cycle determination based on an automatic measurement method of full-segment thickness of the myocardium by echocardiography.

[0114] The innovative features of the method for automatically measuring the thickness of the entire myocardial segment using echocardiography in the embodiment of the present application are as follows:

[0115] 1) Comprehensive myocardial thickness measurement: Unlike traditional methods that only measure the thickness of specific myocardial segments, this method can measure the thickness of the entire myocardial segment, which helps to provide more comprehensive myocardial thickness information;

[0116] 2) Accurate determination of the cardiac cycle: By calculating the vertical distance between the mitral valve endpoint and the myocardial segment, the stage of the cardiac cycle can be accurately determined, thereby clarifying the principle of determining the cardiac cycle, which is crucial for obtaining myocardial thickness data at different time points in the cardiac cycle;

[0117] 3) Multi-parameter comprehensive analysis: This method combines the determination of myocardial thickness and cardiac cycle, providing richer information for clinical applications and helping to more accurately evaluate cardiac function and status.

[0118] Embodiment 2: The embodiment of the present application also provides an automatic measurement system for the thickness of myocardial segments by ultrasound, which adopts the automatic measurement method for the thickness of myocardial segments by ultrasound provided in Embodiment 1. For the description of the automatic measurement method for the thickness of myocardial segments by ultrasound, please refer to Embodiment 1, and this embodiment will not be repeated.

[0119] The hardware involved includes an ultrasound cardiogram instrument 10, a video capture card 20, an artificial intelligence computer 30, and a display 40. In the process of measuring myocardial thickness, the ultrasound doctor uses the ultrasound cardiogram instrument 10 to collect relevant cardiac sections, the video capture card 20 obtains the video stream data on the screen of the ultrasound cardiogram instrument 10, and the artificial intelligence computer 30 performs cardiac structure segmentation, detection and recognition, thereby calculating myocardial thickness. Finally, the measurement results of each frame of the video can be displayed in real time on the display 40. The hardware links and corresponding data flows can be referred to Figure 6 .

[0120] Embodiment 3: The embodiment of the present application also provides an automatic measurement device for the whole segment thickness of myocardium by echocardiography. The automatic measurement device for the whole segment thickness of myocardium by echocardiography includes the automatic measurement system for the whole segment thickness of myocardium by echocardiography provided in Embodiment 2.

[0121] It should be understood that ordinary technical workers in the field can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the claims attached to this application.

Claims

1. An automatic method for measuring the thickness of the whole segment of myocardium by echocardiography, characterized in that: The method comprises: Determine the ultrasound sector area: obtain an ultrasound video stream containing cardiac structure information, and use a deep learning target detection model to automatically identify and determine a sector rectangular frame in the ultrasound video stream to determine a cropping area; Preprocessing the ultrasound video signal: based on the determined clipping area of ​​the sector rectangular frame, clipping the ultrasound video stream, and scaling the clipped ultrasound cardiac image to a target size in proportion; Training the deep learning target detection model: the deep learning target detection model includes a feature extraction module, a myocardial segmentation module and a mitral valve endpoint recognition module; the feature extraction module performs feature extraction on the preprocessed ultrasound cardiac image and generates a feature map; based on the feature map, the myocardial segmentation module performs myocardial segmentation, and the mitral valve endpoint recognition module performs mitral valve endpoint recognition; Model output post-processing: Based on the data obtained by training the myocardial segmentation module and the mitral valve endpoint recognition module, a post-processing module is used to calculate and visualize the output of the myocardial full segment thickness results at the end of diastole or end of systole.

2. The method for automatically measuring the thickness of the whole segment of myocardium by echocardiography according to claim 1, characterized in that: The method of automatically identifying and determining a sector rectangular frame in the ultrasound video stream using a deep learning target detection model to determine a clipping area includes: Obtaining the vertex coordinates of the sector rectangular frame of the first 10 frames of the ultrasound video stream; An average value is calculated for the vertex coordinates of each of the fan-shaped rectangular frames, and a clipping area is determined based on the calculation result of the average value.

3. The method for automatically measuring the thickness of the whole segment of myocardium by echocardiography according to claim 2, characterized in that: The step of clipping the ultrasound video stream based on the determined clipping area of ​​the sector rectangular frame and scaling the clipped ultrasound cardiogram to a target size in proportion includes: The ultrasound video stream is cropped according to the coordinates of the cropping area, and the cropped ultrasound image is proportionally scaled to a target size using a bilinear interpolation method, and during the scaling process, a narrow edge of the ultrasound image is filled with a black background; The pixel value of the ultrasound cardiology image of the target size is divided by the maximum pixel value to normalize the pixel value to a 32-bit floating point number between 0 and 1.

4. The method for automatically measuring the thickness of the whole segment of myocardium by echocardiography according to claim 1, characterized in that: The feature extraction module is used to extract features from the preprocessed ultrasound cardiac image and generate a feature map, including: The feature extraction module is used to extract features from the preprocessed ultrasound cardiac image and generate multi-layer feature maps at different sampling rates; The feature extraction module adopts a three-layer feature pyramid architecture; Multiple layers of feature maps are merged based on the three-layer feature pyramid architecture to extract multi-scale image features and form an output feature map.

5. The method for automatically measuring the thickness of the whole segment of myocardium by echocardiography according to claim 4, characterized in that: The step of completing myocardial segmentation by the myocardial segmentation module based on the feature map includes: enabling the myocardial segmentation module to couple the first target detection task and the segmentation task; The output feature map provided by the feature extraction module is used as input to perform multi-layer deconvolution and bilinear interpolation upsampling calculations to obtain a myocardial contour image; Based on the myocardial contour image, locating the myocardial contour and generating a bounding box according to the first target detection task; According to the segmentation task, a segmentation mask is generated by a segmentation head based on upsampling and convolution operations to segment pixels inside the bounding box for detection.

6. The method for automatically measuring the thickness of the whole segment of myocardium by echocardiography according to claim 4, characterized in that: The method of completing mitral valve endpoint recognition by the mitral valve endpoint recognition module includes: The mitral valve endpoint recognition module is coupled with a second target detection task and a key point recognition task; Using the output feature map provided by the feature extraction module as input, positioning the entire mitral valve according to the second target detection task to determine its location; When the mitral valve leaflet region is successfully identified, the focus is placed on the interior of the mitral valve leaflet region, and the mitral valve endpoint is identified and determined according to the key point identification task.

7. The method for automatically measuring the thickness of the whole segment of myocardium by echocardiography according to claim 1, characterized in that: The post-processing module is used to calculate and visualize the end-diastolic or end-systolic myocardial full segment thickness results, including: The segmentation result provided by the myocardial segmentation module is used as input to calculate the myocardial center curve and the myocardial outer contour line, and obtain the coordinates of the myocardial center curve and the coordinates of the myocardial segmentation outer contour points; Based on the myocardial center curve, multiple segments of myocardial center straight lines are fitted by piecewise linear fitting; Determine the vertical line based on the slope and coordinates of each segment of the myocardial center straight line, calculate the distance between the vertical line and the two intersection points of the upper and lower contour lines of the myocardium, and the distance is the myocardial thickness of the corresponding straight line segment; arrange the segment thicknesses corresponding to all the myocardial center straight lines in sequence to obtain the thickness distribution of the entire myocardial segment; Obtaining the vertical distance between the mitral valve endpoint and the myocardial center straight line in the single frame of the ultrasonic cardiogram by calculating the corresponding vertical distance between the mitral valve endpoint and the myocardial center straight line; Draw a distance curve that changes with time according to the vertical distance between the mitral valve endpoint and the myocardial center straight line in each frame of the entire ultrasound video stream, and use a Kalman filter method to smooth the distance curve; Based on the distance curve after smoothing, the end diastole or end systole of the cardiac cycle is determined by analyzing its maximum value or minimum value, and the myocardial full segment thickness result at the end diastole or end systole is output visually.

8. The method for automatically measuring the thickness of the whole segment of myocardium by echocardiography according to any one of claims 1 to 7, characterized in that: The feature extraction module includes multiple convolutional neural network layers, linear neural network layers and convolutional attention layers; wherein the convolutional attention layer uses a PSA module to process the input feature map in parallel through multiple branches, and each branch is composed of convolutional layers with convolution kernels of different sizes.

9. An automatic measurement system for myocardial full segment thickness by echocardiography, characterized in that: The method for automatically measuring the thickness of the entire segment of myocardium by echocardiography is adopted as described in any one of claims 1 to 8.

10. An automatic measurement device for the thickness of the whole segment of myocardium in echocardiography, characterized in that: The invention comprises the automatic measurement system of full segment thickness of myocardium by echocardiography as claimed in claim 9.

Images