Heart valve image segmentation method, electronic device and storage medium

Abstract

The application provides a heart valve image segmentation method, an electronic device and a storage medium. The segmentation method comprises: extracting a target heart valve region of interest from a heart image in an acquired heart video to obtain a corresponding target heart valve region of interest image; and inputting the target heart valve region of interest image into a heart valve segmentation model to obtain a target heart valve image. The heart valve segmentation model comprises a cascaded segmentation network and a shape completion network. An output layer of the segmentation network is connected to an input layer of the shape completion network. The segmentation network is used for segmenting the target heart valve region of interest image to obtain an initial target heart valve image. The shape completion network is used for shape completion of the initial target heart valve image to obtain a target heart valve image with complete shape. The application can improve the integrity and continuity of the final target heart valve image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for segmenting heart valve images, an electronic device, and a storage medium. Background Technology

[0002] In modern medical imaging, ultrasound images have advantages such as low intensity, low cost, and harmlessness to the human body, and are particularly effective for detecting soft tissues and observing blood flow in cardiovascular organs. With improved living standards and an aging population, common heart valve diseases such as aortic valve malformations are becoming increasingly prevalent. Clinically, the main diagnostic method for these diseases is to observe the shape and movement of the valve using ultrasound equipment. Echocardiography is an excellent tool for detecting heart valve diseases, and the first step in its analysis is echocardiographic image segmentation. Because ultrasound images contain a lot of speckle noise, complex target motion, and low grayscale contrast between the target and the background, segmentation is very difficult. In actual ultrasound image processing and analysis, the identification, localization, and quantitative analysis of targets and lesions mainly rely on manual segmentation based on the physician's experience. Therefore, to segment the aortic valve from cardiac ultrasound images mixed with a large amount of speckle noise and artifacts, physicians need extensive clinical medical knowledge and a keen sense of spatial location. A typical ultrasound sequence consists of dozens or even hundreds of images; if entirely segmented manually by physicians, it would be an enormous workload.

[0003] With the development of image processing technology, aortic valve segmentation based on neural network algorithms has made some progress. However, due to the motion characteristics of the aortic valve, it may appear blurry or even invisible during ultrasound acquisition. Therefore, aortic valve images segmented using traditional neural network algorithms may show missing leaflets, which can affect the doctor's diagnosis.

[0004] It should be noted that the information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method, electronic device, and storage medium for segmenting heart valve images, which can complete the missing shape of heart valves (e.g., missing leaflets of the aortic valve) caused by low signal-to-noise ratio, thereby obtaining a heart valve image with a complete shape.

[0006] To achieve the above objectives, the present invention provides a method for segmenting heart valve images, comprising:

[0007] The region of interest of the target heart valve is extracted from the heart images in the acquired heart video to obtain the corresponding region of interest image of the target heart valve;

[0008] The target heart valve region of interest image is input into the heart valve segmentation model to obtain the target heart valve image;

[0009] The heart valve segmentation model includes a cascaded segmentation network and a shape completion network. The output layer of the segmentation network is connected to the input layer of the shape completion network. The segmentation network is used to segment the region of interest image of the target heart valve to obtain an initial target heart valve image. The shape completion network is used to complete the shape of the initial target heart valve image to obtain the target heart valve image.

[0010] Optionally, the heart valve segmentation model is trained through the following process:

[0011] The shape completion network is trained using a first preset number of first training samples to update the model parameters of the shape completion network until the first preset training termination condition is met. The first training samples include training images of heart valves with missing shapes and corresponding label images of heart valves with complete shapes.

[0012] The model parameters of the trained shape completion network are loaded and frozen. The heart valve segmentation model is trained using a second preset number of second training samples to update the model parameters of the segmentation network until the second preset training termination condition is met. The second training samples include training images of the target heart valve region of interest and corresponding label images of the heart valve with complete shape.

[0013] Optionally, training images of heart valves with missing shapes can be obtained by randomly occluding the acquired images of heart valves with complete shapes.

[0014] Optionally, the random occlusion of the acquired intact heart valve label image includes:

[0015] A random number and size of rectangles are used to occlude the acquired, intact heart valve label image at random locations.

[0016] Optionally, the step of using a random number and size of rectangular boxes to occlude at random locations in the acquired, intact heart valve label image includes:

[0017] Based on the lower and upper similarity thresholds, a random number of rectangles of random size are used to occlude the obtained complete heart valve label image at random locations, so that the similarity between the obtained training image of the heart valve with missing shape and the complete heart valve label image is within the range defined by the lower and upper similarity thresholds.

[0018] Optionally, the step of extracting the region of interest (ROI) of the target heart valve from the acquired cardiac images in the cardiac video to obtain the corresponding ROI image of the target heart valve includes:

[0019] A target detection model was used to extract the region of interest (ROI) of the target heart valve in each frame of the acquired cardiac video to obtain the location information of the corresponding target heart valve ROI.

[0020] Curve fitting is performed based on the location information of the target heart valve region of interest in each frame of cardiac images, and the location information of the target heart valve region of interest in each frame of cardiac images is corrected based on the fitting results to obtain the corrected location information of the target heart valve region of interest in each frame of cardiac images.

[0021] Based on the corrected target heart valve region of interest location information of each frame of cardiac images and the preset magnification, the magnified target heart valve region of interest location information of each frame of cardiac images is calculated.

[0022] Based on the location information of the target heart valve region of interest after magnification of each frame of cardiac images, the corresponding target heart valve region of interest is cropped from each frame of cardiac images to obtain the corresponding target heart valve region of interest image.

[0023] Optionally, the step of performing curve fitting based on the location information of the target heart valve region of interest in each frame of echocardiogram, and correcting the location information of the target heart valve region of interest in each frame of echocardiogram based on the fitting result, includes:

[0024] Based on the target heart valve region of interest location information extracted from each frame of cardiac images by the target detection model, curve fitting is performed to obtain the correspondence between the fitted image frame and the target heart valve region of interest location information.

[0025] Based on the correspondence between the fitted image frames and the location information of the target heart valve region of interest, the location information of the target heart valve region of interest in each frame of the cardiac image is corrected to obtain the corrected location information of the target heart valve region of interest in each frame of the cardiac image.

[0026] Optionally, the step of correcting the target heart valve region of interest location information of each frame of echocardiogram based on the correspondence between the fitted image frames and the target heart valve region of interest location information to obtain the corrected target heart valve region of interest location information of each frame of echocardiogram includes:

[0027] For each frame of the heartbeat image:

[0028] Based on the correspondence between the fitted image frame and the location information of the target heart valve region of interest, the fitted location information of the target heart valve region of interest of the cardiac image frame is obtained.

[0029] The first positional deviation information of the cardiac image is obtained by using the absolute value of the difference between the target heart valve region of interest location information extracted by the target detection model and the fitted target heart valve region of interest location information of the cardiac image.

[0030] Based on the first positional deviation information of the cardiac image frame and the confidence probability value of the target heart valve region of interest extracted by the target detection model, the second positional deviation information of the cardiac image frame is obtained.

[0031] Based on the second positional deviation information of the cardiac image frame, it is determined whether the positional information of the target heart valve region of interest extracted by the target detection model in the cardiac image frame is accurate.

[0032] If so, the target heart valve region of interest location information extracted by the target detection model for that frame of cardiac image is used as the corrected target heart valve region of interest location information for that frame of cardiac image.

[0033] If not, then based on the accurate location information of the target heart valve region of interest in the previous frame of the cardiac image adjacent to this frame, and the accurate location information of the target heart valve region of interest in the next frame of the cardiac image, the corrected location information of the target heart valve region of interest in the cardiac image is obtained.

[0034] Optionally, before inputting the target heart valve region of interest image into the heart valve segmentation model, the segmentation method further includes:

[0035] The larger of the width and height dimensions of the target heart valve region of interest image is used as the target side length dimension;

[0036] The target heart valve region of interest image is filled to adjust the smaller of the width and height dimensions of the target heart valve region of interest image to the target side length dimension;

[0037] The target heart valve region of interest, adjusted to the target side length size, is magnified or reduced to adjust the size of the target heart valve region of interest image to a preset size.

[0038] To achieve the above objectives, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the heart valve image segmentation method described above.

[0039] To achieve the above objectives, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the heart valve image segmentation method described above.

[0040] Compared with existing technologies, the heart valve image segmentation method, electronic device, and storage medium provided by this invention have the following advantages: This invention first extracts the region of interest (ROI) of the target heart valve from the acquired cardiac video image to obtain the corresponding ROI image, laying a solid foundation for the subsequent rapid and accurate acquisition of the corresponding target heart valve image. Since the heart valve segmentation model in this invention includes a cascaded segmentation network and a shape completion network, the segmentation network can segment the ROI image of the target heart valve input to the heart valve segmentation model to obtain an initial target heart valve image. Then, by inputting the initial target heart valve image into the shape completion network, the shape completion network can complete the missing parts of the target heart valve shape (e.g., missing aortic valve leaflets) in the initial target heart valve image, thereby obtaining a target heart valve image with a complete shape. Therefore, this invention not only improves the integrity and continuity of the final target heart valve image (e.g., aortic valve segmentation image) but also reduces the potential for variations caused by human factors. Furthermore, this invention enables an end-to-end algorithm process with strong versatility, thereby better assisting doctors in improving diagnostic efficiency and reducing the risks caused by errors in the analysis of heart valve abnormalities using echocardiography. Attached Figure Description

[0041] Figure 1 A schematic flowchart of a heart valve image segmentation method provided in one embodiment of the present invention;

[0042] Figure 2a A heart rate image provided according to an embodiment of the present invention;

[0043] Figure 2b From Figure 2aThe image shown is a region of interest cropped from the cardiac image of the target heart valve (aortic valve);

[0044] Figure 2c To Figure 2b The target heart valve (aortic valve) region of interest image obtained after filling the region of interest image shown;

[0045] Figure 3 This is a schematic diagram of the training process of a heart valve segmentation model provided in one embodiment of the present invention.

[0046] Figure 4a An image of a fully shaped heart valve in an open state, provided as a specific example of the present invention;

[0047] Figure 4b To Figure 4a The training image of a heart valve with missing shape is obtained by randomly occluding the labeled image of a heart valve with a complete shape.

[0048] Figure 4c This is a specific example of the present invention, showing a fully shaped heart valve label in its closed state;

[0049] Figure 4d To Figure 4c The training image of a heart valve with missing shape is obtained by randomly occluding the labeled image of a heart valve with a complete shape.

[0050] Figure 4e To use the shape completion network in this invention Figure 4c The diagram shows the prediction results obtained after shape completion of a training image of a heart valve with missing shape.

[0051] Figure 4f To use the shape completion network provided by this invention to... Figure 4d The diagram shows the prediction results obtained after shape completion of a training image of a heart valve with missing shape.

[0052] Figure 5a This is a specific example of the present invention, showing the region of interest of the aortic valve in an open state.

[0053] Figure 5b To use the heart valve segmentation model provided by this invention for Figure 5a The aortic valve image obtained by segmenting the region of interest image of the aortic valve shown;

[0054] Figure 5c This is a specific example of the present invention, showing the region of interest of the aortic valve in a closed state;

[0055] Figure 5d To use the heart valve segmentation model provided by this invention for Figure 5c The aortic valve image obtained by segmenting the region of interest image of the aortic valve shown;

[0056] Figure 6 This is a schematic diagram showing the segmentation results provided in a specific example of the present invention;

[0057] Figure 7 This is a schematic diagram of the structure of a segmentation network provided as a specific example of the present invention;

[0058] Figure 8 A schematic diagram of the bottleneck layer provided as a specific example of the present invention;

[0059] Figure 9 This is a schematic diagram of the structure of a transition block provided in a specific example of the present invention;

[0060] Figure 10 This is a schematic diagram of the structure of an upward transition block provided in a specific example of the present invention;

[0061] Figure 11 A schematic diagram of the network structure of a shape completion network provided as a specific example of the present invention;

[0062] Figure 12 This is a block diagram of an electronic device provided according to an embodiment of the present invention.

[0063] The reference numerals in the attached figures are as follows:

[0064] Processor-101; Communication interface-102; Memory-103; Communication bus-104. Detailed Implementation

[0065] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the heart valve image segmentation method, electronic device, and storage medium proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clearly illustrate the embodiments of this invention. Please refer to the drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes and to enable those skilled in the art to understand and read them, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, provided they produce the same or similar effects and achieve the same objectives as this invention, should still fall within the scope of the technical content disclosed in this invention.

[0066] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0067] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0068] The core idea of ​​this invention is to provide a heart valve image segmentation method, electronic device, and storage medium that can complete the missing shape of the heart valve (e.g., the missing leaflet of the aortic valve) caused by low signal-to-noise ratio, thereby obtaining a heart valve image with a complete shape.

[0069] It should be noted that the heart valve image segmentation method of this invention can be applied to the electronic device described in this invention. This electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, tablet computer, or other hardware device with various operating systems. Furthermore, although this document uses echocardiography as an example, as those skilled in the art will understand, the echocardiogram can also be acquired by other devices besides ultrasound (e.g., a cardiac endoscope), and this invention does not limit this. Additionally, although this document uses the aortic valve as the target heart valve, as those skilled in the art will understand, the target heart valve can also be the mitral valve, tricuspid valve, pulmonary valve, etc., and this invention does not limit this. It should also be noted that in this document, the long side direction of the image is defined as the length direction, and the short side direction of the image is defined as the width direction.

[0070] To achieve the above-mentioned goals, this invention provides a method for segmenting heart valve images. Please refer to [the relevant documentation]. Figure 1 The diagram illustrates a flowchart of a cardiac valve image segmentation method according to an embodiment of the present invention. Figure 1 As shown, the heart valve image segmentation method includes the following steps:

[0071] Step S100: Extract the region of interest of the target heart valve from the heart rate image in the acquired heart rate video to obtain the corresponding region of interest image of the target heart valve.

[0072] Step S200: Input the target heart valve region of interest image into the heart valve segmentation model to obtain the target heart valve image.

[0073] The heart valve segmentation model includes a cascaded segmentation network and a shape completion network. The output layer of the segmentation network is connected to the input layer of the shape completion network. The segmentation network is used to segment the region of interest image of the target heart valve to obtain an initial target heart valve image. The shape completion network is used to complete the shape of the initial target heart valve image to obtain the target heart valve image.

[0074] Therefore, the heart valve image segmentation method provided by this invention first extracts the region of interest (ROI) of the target heart valve from the acquired cardiac video image to obtain the corresponding ROI image, which lays a good foundation for the subsequent rapid and accurate acquisition of the corresponding target heart valve image. Since the heart valve segmentation model in this invention includes a cascaded segmentation network and a shape completion network, the segmentation network can segment the ROI image of the target heart valve input to the heart valve segmentation model to obtain an initial target heart valve image. Then, by inputting the initial target heart valve image into the shape completion network, the shape completion network can complete the missing parts of the target heart valve shape (e.g., missing aortic valve leaflets) in the initial target heart valve image, thereby obtaining a complete target heart valve image that is more anatomically accurate. It is evident that this invention not only improves the completeness and continuity of the final target heart valve image (e.g., aortic valve segmentation image) but also reduces the potential for variations caused by human factors. Furthermore, this invention enables an end-to-end algorithm flow, exhibiting strong versatility and thus better assisting doctors in improving diagnostic efficiency and reducing the risks caused by errors in the analysis of heart valve abnormalities using echocardiography. It should be noted that, as those skilled in the art will understand, the target heart valve region of interest can be extracted either frame-by-frame from each frame of the acquired cardiac video, or by skipping frames to select specific frames for extraction; this invention does not limit the scope of the extraction.

[0075] As an example, the cardiac video is an echocardiogram (each cardiac video contains multiple cardiac cycles), and the resolution of the echocardiogram can be set according to specific circumstances, such as 600×800. Specifically, the echocardiogram is a PSAX-AV cross-sectional image acquired by an ultrasound device.

[0076] In one exemplary embodiment, the segmentation method further includes, prior to inputting the target heart valve region of interest image into the heart valve segmentation model:

[0077] The larger of the width and height dimensions of the target heart valve region of interest image is used as the target side length dimension;

[0078] The target heart valve region of interest image is filled to adjust the smaller of the width and height dimensions of the target heart valve region of interest image to the target side length dimension;

[0079] The target heart valve region of interest, adjusted to the target side length size, is magnified or reduced to adjust the size of the target heart valve region of interest image to a preset size.

[0080] Since the heart valve segmentation model requires images of a uniform size as input, adjusting the size of the target heart valve region of interest image to a preset size can meet the input requirements of the heart valve segmentation model. Specifically, the preset size can be set according to specific circumstances. In the preset size, the length and width dimensions of the image are consistent, that is, the image after adjustment to the preset size is a square image, for example, the preset size is 320*320. Therefore, by setting the length and width dimensions in the preset size to be consistent, it is easier to adjust the size of the target heart valve region of interest image to the preset size.

[0081] For details, please refer to Figures 2a to 2c ,in Figure 2a A heartbeat image provided as a specific example of the present invention; Figure 2b From Figure 2a The image shown is a region of interest cropped from the cardiac image of the target heart valve (aortic valve); Figure 2c To Figure 2b The image shows the region of interest (ROI) of the target heart valve (aortic valve) after filling. Figures 2a to 2c As shown, the method described above allows for the accurate cropping of the target heart valve (aortic valve) region of interest image from the acquired ultrasound image. After cropping the target heart valve (aortic valve) region of interest image, black pixels (with a pixel value of 0) can be used to fill the target heart valve region of interest image along the width direction, adjusting the width dimension of the target heart valve region of interest image to match the length dimension, i.e., adjusting the target heart valve region of interest image to a square image. Then, the adjusted square target heart valve region of interest image can be enlarged or reduced by a certain factor to adjust the target heart valve region of interest image to a preset size.

[0082] Please continue to refer to this. Figure 3 The diagram illustrates the training process of a heart valve segmentation model provided in one embodiment of the present invention. Figure 3 As shown, in one exemplary embodiment, the heart valve segmentation model is trained through the following process:

[0083] The shape completion network is trained using a first preset number of first training samples to update the model parameters of the shape completion network until the first preset training termination condition is met. The first training samples include training images of heart valves with missing shapes and corresponding label images of heart valves with complete shapes.

[0084] The model parameters of the trained shape completion network are loaded and frozen. The heart valve segmentation model is trained using a second preset number of second training samples to update the model parameters of the segmentation network until the second preset training termination condition is met. The second training samples include training images of the target heart valve region of interest and corresponding label images of the heart valve with complete shape.

[0085] Therefore, this invention first trains the shape completion network to obtain a trained shape completion network, then connects the input layer of the trained shape completion network to the output layer of the segmentation network, loads and freezes the model parameters of the trained shape completion network, and trains the heart valve segmentation model. During the training of the heart valve segmentation model, the model parameters of the shape completion network are not updated, but only the model parameters of the segmentation network are updated. This ensures that the model parameters of both the shape completion network and the segmentation network in the trained heart valve segmentation model reach their optimal levels, thereby effectively improving the training effect.

[0086] Specifically, the target heart valve region of interest can be cropped from the acquired cardiac images, and the size of the cropped region of interest can be adjusted to a preset size using the size adjustment method described above, thereby obtaining a training image of the target heart valve region of interest. Further, an OpenCV contour extraction algorithm can be used to find the contour of the target heart valve on the obtained training image of the target heart valve region of interest to obtain a label image of a heart valve with a complete shape. After obtaining the label image of a heart valve with a complete shape, by occluding certain positions of the target heart valve region in the label image, a training image of a heart valve with a missing shape can be obtained.

[0087] It should be noted that, as those skilled in the art will understand, the number of the first training samples and the number of the second training samples can be the same or different; that is, the first preset number and the second preset number can be the same value or different values, and this invention does not limit this. Furthermore, it should be noted that, as those skilled in the art will understand, the first preset training termination condition can be that the error value between the prediction result of the training image of the heart valve with missing shape and its corresponding label image of the heart valve with complete shape converges to a first preset error value. The first preset training termination condition can also be that the number of training iterations reaches a first preset number of iterations (e.g., 200 times). Similarly, the second preset training termination condition can be that the error value between the prediction result of the training image of the target heart valve region of interest and its corresponding label image of the heart valve with complete shape converges to a second preset error value. The second preset training termination condition can also be that the number of training iterations reaches a second preset number of iterations (e.g., 200 times).

[0088] In one exemplary implementation, training images of heart valves with missing shapes are obtained by randomly occluding the acquired images of heart valves with intact shapes.

[0089] Therefore, by randomly occluding the acquired intact heart valve label images, it is possible to simulate the situation where the target heart valve shape is missing due to the segmentation network caused by low signal-to-noise ratio (e.g., the aortic valve leaflet is missing).

[0090] In one exemplary implementation, the random occlusion of the acquired, intact heart valve tag image includes:

[0091] A random number and size of rectangles are used to occlude the acquired, intact heart valve label image at random locations.

[0092] Specifically, for each complete heart valve label image, 2-8 rectangles with sides of 5-16 pixels are randomly generated and placed at random locations on the image to occlude it. During occlusion, the pixel values ​​of pixels within the rectangles are set to 0 (i.e., pixels within the rectangles are set to black), while the pixel values ​​of pixels in the unoccluded areas remain unchanged. Please refer to [reference needed]. Figures 4a to 4d ,in Figure 4a An image of a fully shaped heart valve label in an open state, provided by a specific example of the present invention, is illustrated. Figure 4b The illustration shows the... Figure 4a The training image of a heart valve with missing shape is obtained by randomly occluding the labeled image of a heart valve with a complete shape. Figure 4cAn image of a fully shaped heart valve label in its closed state is schematically shown as a specific example of the present invention; Figure 4d The illustration shows the... Figure 4c The training image of a heart valve with missing shapes is obtained by randomly occluding a labeled image of a heart valve with a complete shape. Figures 4a to 4d As shown, by using a random number and size of rectangles to occlude at random locations in a fully formed heart valve label image, the scenario of missing target heart valve shape due to low signal-to-noise ratio segmentation network can be constructed more realistically.

[0093] Furthermore, the step of using a random number and size of rectangular boxes to occlude at random locations in the acquired, fully shaped heart valve label image includes:

[0094] Based on the lower and upper similarity thresholds, a random number of rectangles of random size are used to occlude the obtained complete heart valve label image at random locations, so that the similarity between the obtained training image of the heart valve with missing shape and the complete heart valve label image is within the range defined by the lower and upper similarity thresholds.

[0095] Therefore, by controlling the DICE coefficient (used as a similarity metric function to characterize the similarity between two samples) before and after occlusion, that is, controlling the similarity between the occluded image (training image of a heart valve with missing shape) and the image before occlusion (label image of a heart valve with complete shape), the occlusion ratio can be effectively controlled, thereby generating a corresponding random number and size of rectangles. It should be noted that, as those skilled in the art will understand, the lower and upper similarity thresholds can be set according to specific circumstances, and this invention does not limit this. For example, the lower similarity threshold can be set to 0.65, and the upper similarity threshold can be set to 0.95.

[0096] Please continue to refer to this. Figure 4e and Figure 4f ,in Figure 4e The shape completion network of this invention is illustrated schematically. Figure 4c The diagram shows the prediction results obtained after shape completion of a training image of a heart valve with missing shape. Figure 4f The shape completion network provided by this invention is illustrated schematically. Figure 4d The diagram illustrates the prediction result obtained after shape completion of a training image of a heart valve with missing shape. Figure 4e and Figure 4fAs shown, by using the shape completion network in this invention to complete the input shape-deficient target heart valve image, the missing parts in the shape-deficient target heart valve image can be accurately completed.

[0097] Please continue to refer to this. Figures 5a to 5d ,in Figure 5a An image of the region of interest of the aortic valve in an open state, provided by a specific example of the present invention, is illustrated. Figure 5b The diagram illustrates the use of the heart valve segmentation model provided by this invention for... Figure 5a The aortic valve image obtained by segmenting the region of interest image of the aortic valve shown; Figure 5c An image of the region of interest of the aortic valve in the closed state is schematically shown in a specific example of the present invention; Figure 5d The diagram illustrates the use of the heart valve segmentation model provided by this invention for... Figure 5c The aortic valve image is obtained by segmenting the region of interest image of the aortic valve shown. For example... Figures 5a to 5d As shown, by using the heart valve segmentation method provided by the present invention, a complete aortic valve image can be obtained, which improves the continuity of aortic valve segmentation and reduces the variability that may be caused by human factors.

[0098] In one exemplary embodiment, the step of extracting the region of interest (ROI) of the target heart valve from the acquired cardiac video image to obtain the corresponding ROI image of the target heart valve includes:

[0099] A target detection model is used to detect each frame of the acquired cardiac video to obtain the location information of the corresponding target heart valve region of interest.

[0100] Curve fitting is performed based on the location information of the target heart valve region of interest in each frame of cardiac images, and the location information of the target heart valve region of interest in each frame of cardiac images is corrected based on the fitting results to obtain the corrected location information of the target heart valve region of interest in each frame of cardiac images.

[0101] Based on the corrected target heart valve region of interest location information of each frame of cardiac images and the preset magnification, the magnified target heart valve region of interest location information of each frame of cardiac images is calculated.

[0102] Based on the location information of the target heart valve region of interest after magnification of each frame of cardiac images, the corresponding target heart valve region of interest is cropped from each frame of cardiac images to obtain the corresponding target heart valve region of interest image.

[0103] Therefore, this invention corrects the target heart valve region of interest location information of each frame of cardiac images detected by the target detection model, and crops the corresponding target heart valve region of interest image from the cardiac images according to the corrected target heart valve region of interest location information, thereby obtaining a more accurate target heart valve region of interest image. This lays a good foundation for obtaining a more accurate and complete target heart valve image, effectively improving the accuracy of the finally obtained complete target heart valve image. Furthermore, while object detection models can detect the target heart valve's region of interest (ROI) in each frame of echocardiogram, providing preliminary localization for subsequent heart valve segmentation models, they also result in the loss of detailed information such as the surrounding tissue. Therefore, this invention calculates a magnified version of the target heart valve's ROI location based on the corrected ROI location information. Specifically, the bounding box of the corrected ROI obtained from the object detection model is magnified by a predetermined factor, such as 1.3 times, to obtain the magnified bounding box. The area defined by this magnified bounding box is the final target heart valve ROI. Since this magnified bounding box includes detailed information such as the surrounding tissue, it further improves the segmentation accuracy of subsequent heart valve segmentation models. It should be noted that, as those skilled in the art will understand, the center position of the magnified bounding box is the same as the center position of the unmagnified bounding box.

[0104] Specifically, the object detection model is a pre-trained ResNet50 neural network model. Because ResNet uses skip connections (or shortcuts), it directly passes the activation values ​​of a network layer to deeper layers. Furthermore, skip connections only transmit data; during backpropagation, the signal can be transmitted without attenuation, without worrying about gradient changes, allowing effective gradient propagation to the next layer. Therefore, skip connections effectively alleviate the gradient vanishing problem caused by deepening network layers. Through the stacking of residual blocks, very deep network models can be constructed, enabling effective training even at deep network layers. It should be noted that the dataset used for training the object detection model includes a certain number of cardiac training images and corresponding labels. The labels represent the coordinate positions of the target heart valve region of interest in the cardiac training images, marked with a rectangle in the cardiac training images, and the labels are represented by the coordinates of the four corner points of the rectangle.

[0105] In one exemplary embodiment, the step of performing curve fitting based on the target heart valve region of interest location information of each frame of echocardiogram, and correcting the target heart valve region of interest location information of each frame of echocardiogram based on the fitting result, includes:

[0106] Based on the target heart valve region of interest location information extracted from each frame of cardiac images by the target detection model, curve fitting is performed to obtain the correspondence between the fitted image frame and the target heart valve region of interest location information.

[0107] Based on the correspondence between the fitted image frames and the location information of the target heart valve region of interest, the location information of the target heart valve region of interest in each frame of the cardiac image is corrected to obtain the corrected location information of the target heart valve region of interest in each frame of the cardiac image.

[0108] Therefore, by performing curve fitting on the target heart valve region of interest location information of each frame of cardiac images detected by the target detection model, the correspondence between the fitted image frame and the target heart valve region of interest location information is obtained (this correspondence is used to characterize the fitted target heart valve region of interest location information corresponding to each frame of cardiac images). Thus, the fitted target heart valve region of interest location information of each frame of cardiac images can be obtained based on the correspondence between the fitted image frame and the target heart valve region of interest location information.

[0109] In one exemplary embodiment, the step of correcting the target heart valve region of interest location information of each frame of echocardiogram based on the correspondence between the fitted image frames and the target heart valve region of interest location information, to obtain the corrected target heart valve region of interest location information of each frame of echocardiogram, includes:

[0110] For each frame of the heartbeat image:

[0111] Based on the correspondence between the fitted image frame and the location information of the target heart valve region of interest, the fitted location information of the target heart valve region of interest of the cardiac image frame is obtained.

[0112] The first positional deviation information of the cardiac image is obtained by using the absolute value of the difference between the target heart valve region of interest location information extracted by the target detection model and the fitted target heart valve region of interest location information of the cardiac image.

[0113] Based on the first positional deviation information of the cardiac image frame and the confidence probability value of the target heart valve region of interest extracted by the target detection model, the second positional deviation information of the cardiac image frame is obtained.

[0114] Based on the second positional deviation information of the cardiac image frame, it is determined whether the positional information of the target heart valve region of interest extracted by the target detection model in the cardiac image frame is accurate.

[0115] If so, the target heart valve region of interest location information extracted by the target detection model for that frame of cardiac image is used as the corrected target heart valve region of interest location information for that frame of cardiac image.

[0116] If not, then based on the accurate location information of the target heart valve region of interest in the previous frame of the cardiac image adjacent to this frame, and the accurate location information of the target heart valve region of interest in the next frame of the cardiac image, the corrected location information of the target heart valve region of interest in the cardiac image is obtained.

[0117] Therefore, for each frame of echocardiogram, if the determination result is that the target heart valve region of interest location information extracted by the target detection model for that frame of echocardiogram is accurate, then the target heart valve region of interest location information extracted by the target detection model for that frame of echocardiogram is directly used as the corrected target heart valve region of interest location information for that frame of echocardiogram; if the determination result is that the target heart valve region of interest location information extracted by the target detection model for that frame of echocardiogram is inaccurate, then the corrected target heart valve region of interest location information for that frame of echocardiogram is obtained based on the target heart valve region of interest location information of the previous frame of echocardiogram with accurate location information and the target heart valve region of interest location information of the next frame of echocardiogram with accurate location information, thereby making the corrected target heart valve region of interest location information of each frame of echocardiogram more accurate in the end.

[0118] Further, the step of performing curve fitting based on the target heart valve region of interest location information extracted from each frame of the cardiac image by the target detection model to obtain the correspondence between the fitted image frame and the target heart valve region of interest location information includes:

[0119] Based on the x-coordinate and y-coordinate information of the first corner point and the second corner point of the target heart valve region of interest extracted from each frame of cardiac images by the target detection model, curve fitting is performed on the x-coordinate, y-coordinate, and y-coordinate of the first corner point of the target heart valve region of interest, respectively. This is to obtain the correspondence between the fitted image frame and the x-coordinate, y-coordinate, and y-coordinate of the first corner point of the target heart valve region of interest, respectively.

[0120] Specifically, the first corner point can be the upper left corner of the target heart valve region of interest extracted by the target detection model in each frame of the cardiac image, and the second corner point can be the lower right corner of the target heart valve region of interest extracted by the target detection model in each frame of the cardiac image. Thus, the position information of the first corner point and the position information of the second corner point can represent the position information of the target heart valve region of interest. By curve fitting the abscissa of the first corner point of the target heart valve region of interest in each frame of cardiac images extracted by the target detection model, a curve representing the correspondence between the fitted image frame and the abscissa of the first corner point of the target heart valve region of interest can be obtained; by curve fitting the ordinate of the first corner point of the target heart valve region of interest in each frame of cardiac images extracted by the target detection model, a curve representing the correspondence between the fitted image frame and the ordinate of the first corner point of the target heart valve region of interest can be obtained; by curve fitting the abscissa of the second corner point of the target heart valve region of interest in each frame of cardiac images extracted by the target detection model, a curve representing the correspondence between the fitted image frame and the second corner point of the target heart valve region of interest can be obtained; by curve fitting the ordinate of the second corner point of the target heart valve region of interest in each frame of cardiac images extracted by the target detection model, a curve representing the correspondence between the fitted image frame and the ordinate of the second corner point of the target heart valve region of interest can be obtained. Therefore, based on these four fitted curves, we can obtain the x-coordinate and y-coordinate information of the first corner point and the second corner point of the fitted target heart valve region of interest for any frame of cardiac image, and thus obtain the location information of the fitted target heart valve region of interest for any frame of cardiac image.

[0121] In one exemplary embodiment, the step of correcting the target heart valve region of interest location information of each frame of echocardiogram based on the correspondence between the fitted image frames and the target heart valve region of interest location information, to obtain the corrected target heart valve region of interest location information of each frame of echocardiogram, includes:

[0122] For each frame of the heartbeat image:

[0123] Based on the correspondence between the fitted image frame and the location information of the target heart valve region of interest, the fitted location information of the target heart valve region of interest of the cardiac image frame is obtained.

[0124] The first positional deviation information of the cardiac image is obtained by using the absolute value of the difference between the target heart valve region of interest location information extracted by the target detection model and the fitted target heart valve region of interest location information of the cardiac image.

[0125] Based on the first positional deviation information of the cardiac image frame and the confidence probability value of the target heart valve region of interest extracted by the target detection model, the second positional deviation information of the cardiac image frame is obtained.

[0126] Based on the second positional deviation information of the cardiac image frame, it is determined whether the positional information of the target heart valve region of interest extracted by the target detection model in the cardiac image frame is accurate.

[0127] If so, the target heart valve region of interest location information extracted by the target detection model for that frame of cardiac image is used as the corrected target heart valve region of interest location information for that frame of cardiac image.

[0128] If not, then based on the accurate location information of the target heart valve region of interest in the previous frame of the cardiac image adjacent to this frame, and the accurate location information of the target heart valve region of interest in the next frame of the cardiac image, the corrected location information of the target heart valve region of interest in the cardiac image is obtained.

[0129] Therefore, for each frame of cardiac image, based on the target heart valve region of interest location information extracted by the target detection model and the fitted target heart valve region of interest location information, the first positional deviation information of the cardiac image frame is obtained. Then, based on the first positional deviation information of the cardiac image frame and the confidence probability value of the target heart valve region of interest extracted by the target detection model, the second positional deviation information of the cardiac image frame is obtained. Furthermore, based on the second positional deviation information, it is determined whether the target heart valve region of interest location information extracted by the target detection model for the cardiac image frame is accurate. This allows for a more accurate determination of the accuracy of the target heart valve region of interest location information extracted by the target detection model for the cardiac image frame, effectively avoiding misjudgments and thus effectively improving the correction effect of the target heart valve region of interest location information, further laying a good foundation for obtaining high-precision target heart valve images.

[0130] Furthermore, the step of obtaining the second positional deviation information of the cardiac image frame based on the first positional deviation information of the cardiac image frame and the confidence probability value of the target heart valve region of interest extracted by the target detection model in the cardiac image frame includes:

[0131] The second positional deviation information of the cardiac image frame is obtained according to the following formula:

[0132] e i =E i *(1-p i )

[0133] In the formula, e i E represents the second positional deviation of the i-th frame of the heartbeat image. i p represents the first positional deviation of the i-th frame of the heartbeat image. i This represents the confidence probability value of the target heart valve region of interest extracted by the target detection model from the i-th frame of the cardiac image.

[0134] Specifically, taking the i-th frame of the echocardiogram as an example, assuming that the target detection model extracts the region of interest location information of the target heart valve in the i-th frame of the echocardiogram as (w 1i ,h 1i ,w 2i ,h 2i ), where w 1i h 1i w 2i h 2iThe coordinates of the first corner point, the coordinates of the second corner point, and the coordinates of the third corner point are respectively represented by the x-coordinate of the first corner point, the x-coordinate of the second corner point, and the y-coordinate of the third corner point of the target heart valve region of interest extracted from the i-th frame of the cardiac image by the target detection model. The fitted location information of the target heart valve region of interest in the i-th frame of the cardiac image obtained from the fitting result is (w' 1i ,h' 1i ,w' 2i ,h' 2i ), where w' 1i h' 1i w' 2i h' 2i Let |w| represent the x-coordinate of the first corner point, the y-coordinate of the first corner point, the x-coordinate of the second corner point, and the y-coordinate of the second corner point, respectively, of the fitted region of interest (ROI) of the target heart valve in the i-th frame of the echocardiogram. Then, the absolute value of the difference between the x-coordinate of the first corner point of the ROI extracted by the target detection model and the x-coordinate of the first corner point of the fitted ROI of the target heart valve is |w|. 1i -w' 1i The absolute value of the difference between the ordinate of the first corner point of the target heart valve region of interest extracted by the target detection model and the ordinate of the first corner point of the fitted target heart valve region of interest is |h 1i -h' 1i The absolute value of the difference between the x-coordinate of the second corner point of the target heart valve region of interest extracted by the target detection model and the x-coordinate of the fitted target heart valve region of interest is |w 2i -w' 2i The absolute value of the difference between the ordinate of the second corner point of the target heart valve region of interest extracted by the target detection model and the ordinate of the second corner point of the fitted target heart valve region of interest is |h 2i -h' 2i |;that is, the first positional deviation E of the i-th frame of the cardiac image i for:

[0135] E i =(|w 1i -w' 1i |,|h 1i -h' 1i |,|w 2i -w' 2i |,|h 2i -h' 2i |)

[0136] Then the second positional deviation e of the i-th frame of the heart image i for:

[0137] ei =(|w 1i -w' 1i |*(1-p i ),|h 1i -h' 1i |*(1-p i ),|w 2i -w' 2i |*(1-p i ),|h 2i -h' 2i |*(1-p i ))

[0138] In one exemplary embodiment, determining whether the location information of the target heart valve region of interest extracted by the target detection model from the cardiac image frame is accurate based on the second positional deviation information of the cardiac image frame includes:

[0139] Based on the first position deviation information of each frame of the heartbeat image, the mean first position deviation information of the heartbeat video is obtained;

[0140] The average confidence probability of the target heart valve region of interest in each frame of the heart rate image is extracted based on the target detection model, and the average confidence probability of the heart rate video is obtained.

[0141] Based on the mean first positional deviation information of the heartbeat video and the mean confidence probability of the heartbeat video, the mean second positional deviation information of the heartbeat video is obtained.

[0142] The position judgment threshold is obtained based on the preset multiple threshold and the average second position deviation information of the heartbeat video;

[0143] For each frame of the cardiac motion video, based on the second positional deviation information of the cardiac motion image and the positional judgment threshold, it is determined whether the positional information of the target heart valve region of interest extracted by the target detection model for that frame of cardiac motion image is accurate.

[0144] Specifically, the average first positional deviation of the cardiac video is obtained by averaging the first positional deviation of each frame of cardiac images; the average confidence probability of the cardiac video is obtained by averaging the confidence probability values ​​of each frame of cardiac images; the product of the average first positional deviation and the average confidence probability of the cardiac video is the average second positional deviation of the cardiac video; the product of the average second positional deviation and a preset multiple threshold is the position judgment threshold. Since the position judgment threshold is based on the average second positional deviation of the cardiac video and the preset multiple threshold, the position judgment threshold is different for different cardiac videos, meaning the position judgment threshold in this invention is dynamically changing. This further improves the correction effect of the target heart valve region of interest location information, laying a good foundation for obtaining accurate target heart valve region of interest location information.

[0145] Assuming the heartbeat video comprises n frames of heartbeat images, the mean first positional deviation E of the heartbeat video can be expressed as:

[0146]

[0147] The mean confidence probability of the heartbeat video It can be represented as:

[0148]

[0149] The mean of the second position deviation of the heartbeat video It can be represented as:

[0150]

[0151] Assuming the preset magnification factor is m, then the position determination threshold T of the heartbeat video is... th It can be represented as:

[0152]

[0153] It should be noted that, as those skilled in the art will understand, during location determination, the x-coordinate, y-coordinate, and y-coordinate of the first corner point, the second corner point, and the third corner point of the target heart valve region of interest are determined respectively, and the x-coordinate, y-coordinate, and y-coordinate of the first corner point, the second corner point, and the third corner point of the target heart valve region of interest are corrected accordingly based on the determination results. Specifically, taking the i-th frame of the echocardiogram as an example, if the x-coordinate of the first corner point of the target heart valve region of interest in the i-th frame of the echocardiogram satisfies the following condition: |w 1i -w' 1i |*(1-p i (greater than) This indicates that the x-coordinate error of the first corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is large. Therefore, the average of the x-coordinates of the first corner point of the target heart valve region of interest in the previous frame (i.e., the x-coordinates of the first corner point of the region of interest extracted by the target detection model) and the x-coordinates of the first corner point of the target heart valve region of interest in the next frame (i.e., the x-coordinates of the first corner point of the region of interest extracted by the target detection model) is taken as the corrected x-coordinate of the first corner point of the target heart valve region of interest in the i-th frame of the cardiac image; if |w 1i -w' 1i |*(1-p i Less than or equal to This indicates that the x-coordinate of the first corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is accurate. Therefore, the x-coordinate of the first corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is directly used as the x-coordinate of the first corner point of the target heart valve region of interest in the i-th frame of the cardiac image after correction.

[0154] If the ordinate of the first corner point of the region of interest of the target heart valve in the i-th frame of the cardiac image satisfies the following condition: |h 1i -h' 1i |*(1-p i (greater than) This indicates that the error in the ordinate of the first corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is large. Therefore, the average of the ordinates of the first corner point of the target heart valve region of interest in the previous frame (where the ordinates are accurate, i.e., the ordinates of the first corner point of the region of interest extracted by the target detection model are accurate) and the ordinates of the first corner point of the target heart valve region of interest in the next frame (where the ordinates are accurate, i.e., the ordinates of the first corner point of the region of interest extracted by the target detection model are accurate) is taken as the corrected ordinate of the first corner point of the target heart valve region of interest in the i-th frame of the cardiac image; if |h 1i -h' 1i |*(1-p i Less than or equal to This indicates that the ordinate of the first corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is accurate. Therefore, the ordinate of the first corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is directly used as the ordinate of the first corner point of the target heart valve region of interest in the i-th frame of the cardiac image after correction.

[0155] If the x-coordinate of the second corner point of the region of interest of the target heart valve in the i-th frame of the cardiac image satisfies the following condition: |w 2i -w' 2i |*(1-p i (greater than) This indicates that the x-coordinate error of the second corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is large. Therefore, the average of the x-coordinates of the second corner point of the target heart valve region of interest in the previous frame (i.e., the x-coordinates of the second corner point of the region of interest extracted by the target detection model are accurate) and the x-coordinates of the second corner point of the target heart valve region of interest in the next frame (i.e., the x-coordinates of the second corner point of the region of interest extracted by the target detection model are accurate) is taken as the corrected x-coordinate of the second corner point of the target heart valve region of interest in the i-th frame of the cardiac image; if |w 2i -w' 2i |*(1-p i Less than or equal to This indicates that the x-coordinate of the second corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is accurate. Therefore, the x-coordinate of the second corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is directly used as the x-coordinate of the second corner point of the target heart valve region of interest in the i-th frame of the cardiac image after correction.

[0156] If the ordinate of the second corner point of the region of interest of the target heart valve in the i-th frame of the cardiac image satisfies the following condition: |h 2i -h' 2i |*(1-p i (greater than) This indicates that the error in the ordinate of the second corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is large. Therefore, the average of the ordinates of the second corner point of the target heart valve region of interest in the previous frame (where the ordinates of the second corner point are accurate, i.e., the ordinates of the second corner point of the region of interest extracted by the target detection model are accurate) and the ordinates of the second corner point of the target heart valve region of interest in the next frame (where the ordinates of the second corner point are accurate, i.e., the ordinates of the second corner point of the region of interest extracted by the target detection model are accurate) is taken as the corrected ordinate of the second corner point of the target heart valve region of interest in the i-th frame of the cardiac image; if |h 2i -h' 2i |*(1-p i Less than or equal to This indicates that the ordinate of the second corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is accurate. Therefore, the ordinate of the second corner point of the target heart valve region of interest extracted by the target detection model in the i-th frame of the cardiac image is directly used as the ordinate of the second corner point of the target heart valve region of interest in the i-th frame of the cardiac image after correction.

[0157] In one exemplary embodiment, the cardiac valve image segmentation method provided by the present invention further includes:

[0158] The region of interest and the outline of the target heart valve are marked on the cardiac image.

[0159] Therefore, by marking the region of interest and the outline of the target heart valve on the cardiac image, doctors can more intuitively view the test results, which is beneficial to improving the accuracy of diagnosis. Please refer to... Figure 6 The diagram illustrates a specific example of the segmentation result display provided by the present invention. Figure 6 As shown in the figure, the area defined by the rectangle is the region of interest of the target heart valve, and the curved outline in the figure is the outline of the segmented target heart valve.

[0160] In one exemplary embodiment, after marking the region of interest and the outline of the target heart valve on the echocardiogram, median filtering can be used to set the gray value of each pixel in the echocardiogram to the median of the gray values ​​of all pixels within its neighborhood window. The size parameter of the filtering kernel can be set according to specific circumstances, for example, to 5×5. Thus, median filtering can effectively remove salt-and-pepper noise from the echocardiogram. It should be noted that, as those skilled in the art will understand, in other embodiments, other filtering methods besides median filtering can be used to filter the echocardiogram, and this invention does not limit this to such methods.

[0161] The specific structures of the segmentation network and shape completion network in the heart segmentation model provided by this invention are described below. For ease of distinction, the input layer of the segmentation network is represented by the first input layer, the output layer of the segmentation network is represented by the first output layer, the input layer of the shape completion network is represented by the second input layer, and the output layer of the shape completion network is represented by the second output layer.

[0162] In one exemplary embodiment, the segmentation network is a DenseNet neural network model. Since the DenseNet neural network model is a densely connected convolutional neural network, where the input of each layer comes from the outputs of all preceding layers, this neural network structure enhances feature transfer and utilizes features more effectively. Furthermore, the DenseNet neural network model has good anti-overfitting performance, making it particularly suitable for applications with relatively scarce training data. Therefore, using the DenseNet neural network model as the segmentation network in this invention can effectively improve the segmentation efficiency and accuracy of the inner and outer contours of the target heart valve (e.g., the aortic valve). Specifically, the DenseNet neural network model consists of multiple densely connected blocks connected by transition blocks; that is, any two adjacent densely connected blocks are connected by a transition block, and the number of convolutional output channels within each densely connected block is consistent, facilitating the superposition of feature information from each layer.

[0163] One layer in a densely connected block is called a bottleneck layer. Dense connections in DenseNet connect each layer in a densely connected block to all subsequent layers, enabling feature reuse.

[0164] Please continue to refer to this. Figure 7 The diagram illustrates the structure of a segmentation network provided in a specific example of the present invention. Figure 7As shown in this example, the segmentation network includes a first input layer, a first convolutional layer, a first pooling layer (preferably a max pooling layer), a first dense connection block, a first transition block, a second dense connection block, a second transition block, a third dense connection block, a third transition block, a fourth dense connection block, a first upward transition block, a second upward transition block, a second convolutional layer (with a kernel size of 1×1), and a first output layer, connected in sequence. In this configuration, the first convolutional layer extracts features such as the boundary and texture of the target heart valve (e.g., the aortic valve) from the region of interest image received by the first input layer. The first pooling layer performs pooling operations on the output of the first convolutional layer to remove unnecessary redundant information, such as background noise, from the image. The first dense connection block extracts features of the target heart valve (e.g., the aortic valve) from the output of the first pooling layer. The first transition block compresses the output of the first dense connection block to reduce the size of the feature map output by the first dense connection block. The second dense connection block extracts features of the target heart valve (e.g., the aortic valve) from the output of the first transition block. The second transition block compresses the output of the second dense connection block to reduce the size of the feature map output by the second dense connection block. The third dense connection block... The connecting block is used to extract target heart valve (e.g., aortic valve) features from the output of the second transition block. The third transition block is used to compress the output of the third dense connecting block to reduce the size of the feature map output by the third dense connecting block. The fourth dense connecting block is used to extract target heart valve (e.g., aortic valve) features from the output of the third transition block. The first upward transition block is used to deconvolve the output of the fourth dense connecting block to increase the size of the feature map output by the fourth dense connecting block. The second upward transition block is used to deconvolve the output of the first upward transition block to increase the size of the feature map output by the first upward transition block. The second convolutional layer is used to perform nonlinear mapping regression on the output of the second upward transition block to obtain an initial target heart valve image. The first output layer is used to output the initial target heart valve image.

[0165] Specifically, the second convolutional layer can perform a non-linear mapping regression on the output of the second upward transition block using the sigmoid function, the formula of which is shown below:

[0166]

[0167] As shown in the above equation, the Sigmoid function can map any input real number to the real number mapping interval (0,1). When the input value x is large, the output value g tends to 1, and when the input value x is small, the output value g tends to 0.

[0168] It should be noted that, as those skilled in the art will understand, the first dense connection block, the second dense connection block, the third dense connection block, and the fourth dense connection block all include multiple bottleneck layers, and the number of bottleneck layers in the first dense connection block, the second dense connection block, the third dense connection block, and the fourth dense connection block can be the same or different. The specific number can be set according to actual needs, and the present invention does not limit this. For example, the first dense connection block may have 6 bottleneck layers, the second dense connection block may have 12 bottleneck layers, the third dense connection block may have 24 bottleneck layers, and the fourth dense connection block may have 16 bottleneck layers.

[0169] Please continue to refer to this. Figure 8 The diagram illustrates the structure of the bottleneck layer provided in a specific example of the present invention. Figure 8 As shown, the bottleneck layer comprises a first batch normalization layer A, a first activation layer A, a third convolutional layer A, a first batch normalization layer B, a first activation layer B, and a third convolutional layer B connected in sequence. The kernel size of the third convolutional layer A is 1×1, and the kernel size of the third convolutional layer B is 3×3. Therefore, by adding a 1×1 convolution before the 3×3 convolution in the bottleneck layer, this invention reduces the number of feature maps and the dimensionality of each feature map, thereby reducing computational cost and fusing features from various channels. Furthermore, since the bottleneck layer performs batch normalization (BN) and ReLU activation operations before both the 1×1 and 3×3 convolution operations, training speed and convergence efficiency can be further improved.

[0170] Please continue to refer to this. Figure 9 The diagram illustrates the structure of a transition block provided in a specific example of the present invention. Figure 9 As shown, the first transition block, the second transition block, and the third transition block each include a second batch normalization layer, a second activation layer, a fourth convolutional layer, and a second pooling layer (preferably an average pooling layer) connected in sequence. The kernel size of the fourth convolutional layer is 1×1. Thus, the convolutional operation of the fourth convolutional layer can reduce the dimensionality of the feature map, and the average pooling operation of the second pooling layer can solve the problem of excessive channels in the feature map, preventing model complexity caused by too many densely connected blocks. Furthermore, since each transition block performs batch normalization (BN) and ReLU activation operations before the 1×1 convolutional operation, the number of parameters can be further compressed.

[0171] Please continue to refer to this. Figure 10 The diagram illustrates the structure of an upward transition block provided in a specific example of the present invention. Figure 10 As shown, both the first upward transition block and the second upward transition block include a third batch normalization layer A, a third activation layer A, a fifth convolutional layer A, a third batch normalization layer B, a third activation layer B, a fifth convolutional layer B, a third batch normalization layer C, a third activation layer C, and a first deconvolutional layer connected in sequence. The size of the convolutional kernels of the fifth convolutional layer A and the fifth convolutional layer B is 3×3.

[0172] In one exemplary implementation, the segmentation network uses a binary cross-entropy loss function during training, the formula of which is shown below:

[0173]

[0174]

[0175] In the formula, y i For real labels, This is the predicted result.

[0176] Furthermore, after training the segmentation network, this invention also uses the Dice coefficient formula to evaluate the algorithm accuracy of the segmentation network, as shown below:

[0177]

[0178] In the formula, X represents the prediction result, and Y represents the true label.

[0179] The value of Dice ranges from 0 to 1. The closer the Dice value is to 1, the higher the segmentation accuracy of the segmentation network.

[0180] In one exemplary implementation, the shape completion network uses a UNet-based network architecture. Please refer to [link / reference needed]. Figure 11 The diagram illustrates the network structure of a shape completion network provided in a specific example of the present invention. Figure 11As shown, the shape completion network includes a decoding module and an encoding module; wherein, the decoding module includes a second input layer, multiple cascaded first neural network groups, and a sixth convolutional layer (with a kernel size of 3×3). The second input layer is used to receive the initial target heart valve image output by the first output layer of the segmentation network. The first neural network group includes a cascaded seventh convolutional layer (with a kernel size of 3×3) and a max pooling layer. The seventh convolutional layer is used to extract image feature information from the initial target heart valve image or the output image of the previous first neural network group. The max pooling layer is used to pool the output image of the seventh convolutional layer. The sixth convolutional layer is used to extract image feature information from the output image of the deepest first neural network group. The encoding module includes multiple cascaded second neural network groups, an eighth convolutional layer (with a kernel size of 1×1), and a second output layer. The second neural network groups correspond one-to-one with the first neural network groups. Each second neural network group includes a cascaded second deconvolutional layer, a merging layer, and a ninth convolutional layer (with a kernel size of 3×3). The second deconvolutional layer is used to perform the opposite operation to the pooling operation of the corresponding max pooling layer in the decoding module. The merging layer is used to linearly add and merge the output image of the second deconvolutional layer with the output image of the corresponding seventh convolutional layer in the decoding module. The ninth convolutional layer is used to recover the image feature information lost during the pooling process of the corresponding max pooling layer in the decoding module. The eighth convolutional layer is used to perform logistic regression on the output result of the deepest ninth convolutional layer to obtain a target heart valve image with a complete shape. The second output layer is used to output the target heart valve image with a complete shape.

[0181] It should be noted that, Figure 11 In the shape completion network structure shown, the number of first neural network groups in the decoding module and the number of second neural network groups in the encoding module are merely examples and should not be construed as limiting the implementation of this application. The number of first neural network groups in the decoding network and the number of second neural network groups in the encoding module can be set according to specific needs. It should be noted that, since encoding and decoding have a one-to-one correspondence, in the shape completion network structure provided in this application embodiment, the number of first neural network groups in the decoding module is equal to the number of second neural network groups in the encoding module. Furthermore, the number of seventh convolutional layers in the first neural network group and the number of ninth convolutional layers in the second neural network group are not limited to two; they can also be three or more, and this invention does not impose any limitations on them.

[0182] Based on the same inventive concept, the present invention also provides an electronic device, please refer to... Figure 12A block diagram illustrating an embodiment of the electronic device provided by the present invention is shown. Figure 12 As shown, the electronic device includes a processor 101 and a memory 103. The memory 103 stores a computer program, which, when executed by the processor 101, implements the heart valve image segmentation method described above. Since the electronic device provided by this invention and the heart valve image segmentation method described above belong to the same inventive concept, the electronic device provided by this invention possesses all the advantages of the heart valve image segmentation method described above. Therefore, the beneficial effects of the electronic device provided by this invention will not be elaborated upon here.

[0183] like Figure 12 As shown, the electronic device also includes a communication interface 102 and a communication bus 104, wherein the processor 101, the communication interface 102, and the memory 103 communicate with each other via the communication bus 104. The communication bus 104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 104 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 102 is used for communication between the aforementioned electronic device and other devices.

[0184] The processor 101 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 101 is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines. The memory 103 can be used to store the computer program. The processor 101 implements various functions of the electronic device by running or executing the computer program stored in the memory 103 and by calling data stored in the memory 103.

[0185] The memory 103 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0186] This invention also provides a readable storage medium storing a computer program that, when executed by a processor, can implement the heart valve image segmentation method described above. Since the readable storage medium provided by this invention and the heart valve image segmentation method described above belong to the same inventive concept, the readable storage medium provided by this invention possesses all the advantages of the heart valve image segmentation method described above. Therefore, the beneficial effects of the readable storage medium provided by this invention will not be elaborated further here.

[0187] The readable storage medium of embodiments of the present invention can be any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer hard disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0188] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0189] In summary, compared with existing technologies, the heart valve image segmentation method, electronic device, and storage medium provided by this invention have the following advantages: This invention first extracts the region of interest (ROI) of the target heart valve from the acquired cardiac video image to obtain the corresponding ROI image, laying a solid foundation for the subsequent rapid and accurate acquisition of the corresponding target heart valve image. Since the heart valve segmentation model in this invention includes a cascaded segmentation network and a shape completion network, the segmentation network can segment the ROI image of the target heart valve input to the heart valve segmentation model to obtain an initial target heart valve image. Then, by inputting the initial target heart valve image into the shape completion network, the shape completion network can complete the missing parts of the target heart valve shape (e.g., missing aortic valve leaflets) in the initial target heart valve image, thereby obtaining a target heart valve image with a complete shape. Therefore, this invention not only improves the integrity and continuity of the final target heart valve image (e.g., aortic valve segmentation image) but also reduces the potential for variations caused by human factors. Furthermore, this invention enables an end-to-end algorithm process with strong versatility, thereby better assisting doctors in improving diagnostic efficiency and reducing the risks caused by errors in the analysis of heart valve abnormalities using echocardiography.

[0190] It should be noted that computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0191] It should be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0192] In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0193] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for segmenting heart valve images, characterized in that, include: The region of interest (ROI) of the target heart valve is extracted from the heart images in the acquired heart video to obtain the corresponding ROI image of the target heart valve. The target heart valve region of interest image is input into the heart valve segmentation model to obtain the target heart valve image; The heart valve segmentation model includes a cascaded segmentation network and a shape completion network. The output layer of the segmentation network is connected to the input layer of the shape completion network. The segmentation network is used to segment the region of interest image of the target heart valve to obtain an initial target heart valve image. The shape completion network is used to complete the shape of the initial target heart valve image to obtain the target heart valve image. The step of extracting the region of interest (ROI) of the target heart valve from the acquired cardiac images in the cardiac video to obtain the corresponding ROI image of the target heart valve includes: A target detection model was used to extract the region of interest (ROI) of the target heart valve in each frame of the acquired cardiac video to obtain the location information of the corresponding target heart valve ROI. Curve fitting is performed based on the location information of the target heart valve region of interest in each frame of cardiac images, and the location information of the target heart valve region of interest in each frame of cardiac images is corrected based on the fitting results to obtain the corrected location information of the target heart valve region of interest in each frame of cardiac images. Based on the corrected target heart valve region of interest location information of each frame of cardiac images and the preset magnification, the magnified target heart valve region of interest location information of each frame of cardiac images is calculated. Based on the location information of the target heart valve region of interest after magnification of each frame of cardiac images, the corresponding target heart valve region of interest is cropped from each frame of cardiac images to obtain the corresponding target heart valve region of interest image.

2. The method for segmenting heart valve images according to claim 1, characterized in that, The heart valve segmentation model was trained through the following process: The shape completion network is trained using a first preset number of first training samples to update the model parameters of the shape completion network until the first preset training termination condition is met. The first training samples include training images of heart valves with missing shapes and corresponding label images of heart valves with complete shapes. The model parameters of the trained shape completion network are loaded and frozen. The heart valve segmentation model is trained using a second preset number of second training samples to update the model parameters of the segmentation network until the second preset training termination condition is met. The second training samples include training images of the target heart valve region of interest and corresponding label images of the heart valve with complete shape.

3. The method for segmenting heart valve images according to claim 2, characterized in that, By randomly occluding the acquired complete heart valve label images, corresponding training images of heart valves with missing shapes are obtained.

4. The method for segmenting heart valve images according to claim 3, characterized in that, The random occlusion of the acquired intact heart valve label image includes: A random number and size of rectangles are used to occlude the acquired, intact heart valve label image at random locations.

5. The method for segmenting heart valve images according to claim 4, characterized in that, The step of using a random number and size of rectangles to occlude at random locations in the acquired, intact heart valve label image includes: Based on the lower and upper similarity thresholds, a random number of rectangles of random size are used to occlude the obtained complete heart valve label image at random locations, so that the similarity between the obtained training image of the heart valve with missing shape and the complete heart valve label image is within the range defined by the lower and upper similarity thresholds.

6. The method for segmenting heart valve images according to claim 1, characterized in that, The step of performing curve fitting based on the location information of the target heart valve region of interest in each frame of echocardiogram, and correcting the location information of the target heart valve region of interest in each frame of echocardiogram based on the fitting result, includes: Based on the target heart valve region of interest location information extracted from each frame of cardiac images by the target detection model, curve fitting is performed to obtain the correspondence between the fitted image frame and the target heart valve region of interest location information. Based on the correspondence between the fitted image frames and the location information of the target heart valve region of interest, the location information of the target heart valve region of interest in each frame of the cardiac image is corrected to obtain the corrected location information of the target heart valve region of interest in each frame of the cardiac image.

7. The method for segmenting heart valve images according to claim 6, characterized in that, The step of correcting the target heart valve region of interest location information of each frame of cardiac images based on the correspondence between the fitted image frames and the target heart valve region of interest location information, to obtain the corrected target heart valve region of interest location information of each frame of cardiac images, includes: For each frame of the heartbeat image: Based on the correspondence between the fitted image frame and the location information of the target heart valve region of interest, the fitted location information of the target heart valve region of interest of the cardiac image frame is obtained. The first positional deviation information of the cardiac image is obtained by using the absolute value of the difference between the target heart valve region of interest location information extracted by the target detection model and the fitted target heart valve region of interest location information of the cardiac image. Based on the first positional deviation information of the cardiac image frame and the confidence probability value of the target heart valve region of interest extracted by the target detection model, the second positional deviation information of the cardiac image frame is obtained. Based on the second positional deviation information of the cardiac image frame, it is determined whether the positional information of the target heart valve region of interest extracted by the target detection model in the cardiac image frame is accurate. If so, the target heart valve region of interest location information extracted by the target detection model for that frame of cardiac image is used as the corrected target heart valve region of interest location information for that frame of cardiac image. If not, then based on the accurate location information of the target heart valve region of interest in the previous frame of the cardiac image adjacent to this frame, and the accurate location information of the target heart valve region of interest in the next frame of the cardiac image, the corrected location information of the target heart valve region of interest in the cardiac image is obtained.

8. The method for segmenting heart valve images according to claim 1, characterized in that, Before inputting the target heart valve region of interest image into the heart valve segmentation model, the segmentation method further includes: The larger of the width and height dimensions of the target heart valve region of interest image is used as the target side length dimension; The target heart valve region of interest image is filled to adjust the smaller of the width and height dimensions of the target heart valve region of interest image to the target side length dimension; The target heart valve region of interest, adjusted to the target side length size, is magnified or reduced to adjust the size of the target heart valve region of interest image to a preset size.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the heart valve image segmentation method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the heart valve image segmentation method according to any one of claims 1 to 8.

Images