Real-time echocardiogram segmentation method and system based on hybrid vision and anatomical perception

Abstract

The application discloses a real-time echocardiogram segmentation method and system based on mixed vision and anatomical perception, and mainly solves the problems of poor modeling capability and low segmentation precision in the prior art. The implementation scheme comprises the following steps: collecting an ultrasonic image sector region image by using a non-invasive screen vision capture method, and generating a preprocessed image to be segmented; constructing an echocardiogram segmentation model comprising a double-flow mixed feature extraction network, a feature fusion enhancement network and an MLP decoding head; defining a loss function of the echocardiogram segmentation model, inputting an existing CAMUS data set into the model for iterative training until the loss function converges, and obtaining a trained echocardiogram segmentation model; and inputting the collected preprocessed image to be segmented into the trained echocardiogram segmentation model, and obtaining a segmentation result of the echocardiogram. The application can balance global semantic understanding and local texture sensitivity, and realizes accurate segmentation of weak boundary structures under complex imaging conditions, and can be used for evaluation of cardiac structure and function.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and specifically relates to a real-time echocardiogram segmentation method and system, which can be used for the assessment of cardiac structure and function. Background Technology

[0002] Echocardiography, with its advantages of being non-invasive, real-time, and cost-effective, has become the preferred imaging modality for clinical assessment of cardiac structure and function. However, due to limitations in its imaging principles, ultrasound images generally suffer from low signal-to-noise ratio, speckle noise interference, and acoustic occlusion. Furthermore, the heart, as a non-rigid organ, exhibits complex deformation with significant differences between different cross-sections. Traditional image processing methods, such as active contour models, often rely on manual initialization and are sensitive to noise, making them unsuitable for complex clinical scenarios. In recent years, deep learning technology has made breakthroughs in medical image segmentation, but it still faces many challenges in terms of the completeness of feature extraction, the rationality of anatomical structures, and the real-time nature of clinical deployment.

[0003] Patent document CN110475505A discloses a method for automatic heart segmentation using fully convolutional networks. It primarily relies on a convolutional neural network architecture with contraction and expansion paths, combined with image intensity thresholding to distinguish between blood and muscle tissue. The limitation of this method lies in its pure convolutional structure, which is constrained by a local receptive field, making it difficult to effectively capture long-range global semantic dependencies. This leads to segmentation fragmentation when ultrasound image boundaries are blurred or acoustic shadowing is present. Furthermore, this method heavily depends on intensity-based post-processing thresholding, making it extremely sensitive to common gray-level inhomogeneities and artifacts in ultrasound images, resulting in poor robustness.

[0004] Patent document CN118537317A discloses a deep learning-based algorithm for evaluating left ventricular ejection fraction in two-dimensional echocardiography. It introduces an attention mechanism on top of the U-Net architecture to enhance the model's focus on key features and suppress background noise. While this method improves feature extraction capabilities to some extent through the attention module, its core framework is still based on convolutional neural networks. When dealing with the heart, an anatomical structure with strong topological constraints, it lacks explicit modeling of geometric morphology and topological continuity. This often leads to topological errors in the model's output segmentation mask, such as myocardial discontinuities or chamber overlaps, which violate physiological common sense and thus affect the accuracy of clinical parameter measurements.

[0005] Patent document CN120635459A discloses a training and segmentation method for a cardiac structure segmentation model in fetal echocardiography. It attempts to apply the large visual model SAM2 to the field of medical ultrasound, fine-tuning it by introducing an adapter and cross-attention module into the image encoder to utilize the strong representational capabilities of the large model. However, because this method is based on a general large model, its parameter count is enormous and its computational complexity is extremely high, making it difficult to meet the stringent requirements of high frame rate real-time imaging in clinical ultrasound equipment. Furthermore, since the pre-training features of the general large model are mainly derived from natural images, direct transfer to the medical ultrasound field often results in wasted computational resources and excessively high inference latency if a lightweight architecture reconstruction specifically for ultrasound-specific speckle noise and anatomical priors is lacking, hindering portable deployment.

[0006] In summary, existing technologies mainly face challenges such as insufficient global modeling capabilities of convolutional networks, lack of explicit anatomical topological constraints, and computationally intensive large-scale models that are difficult to deploy in real time. Therefore, there is an urgent need for an echocardiogram segmentation method that can balance global and local features, possess anatomical perception capabilities, and achieve efficient real-time inference. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art by proposing a real-time echocardiogram segmentation method and system based on hybrid vision and anatomical perception, so as to completely extract the features of the left ventricle, left atrium and myocardium in echocardiograms, and achieve accurate segmentation that conforms to physiological topology while ensuring high frame rate and real-time operation.

[0008] The key technology of this invention is: to balance global and local features by constructing a dual-stream hybrid network architecture; and to improve the segmentation accuracy of echocardiograms in real time by combining explicit anatomical prior knowledge with a low-rank adaptive (LoRA) lightweight fine-tuning strategy. The implementation scheme includes:

[0009] 1. A real-time echocardiogram segmentation method based on hybrid visual and anatomical perception, including:

[0010] (1) Acquire the fan-shaped region image of ultrasound image using a non-invasive screen visual capture method, and preprocess it to generate a preprocessed image to be segmented;

[0011] (2) Construct an echocardiographic segmentation model including a dual-stream hybrid feature extraction network, a feature fusion enhancement network, and an MLP decoder:

[0012] The dual-stream hybrid feature extraction network includes a global stream branch Hiera backbone unit and a local stream branch MobileViT unit, which are used to extract global semantic features and local texture features of the preprocessed image and output multi-scale dual-stream features.

[0013] The feature fusion enhancement network is used to perform cross-scale attention fusion and adaptive feature pyramid processing on the extracted dual-stream multi-scale features to obtain fused features. By introducing an anatomically perceptual attention mechanism, the topology and adjacency relationship of the fused features are constrained to obtain enhanced fused features.

[0014] The MLP decoding head is used to perform channel mapping, feature refinement and upsampling on the enhanced fusion features to generate a segmentation probability map of the preprocessed image;

[0015] (3) Define the loss function of the echocardiogram segmentation model, input the existing CAMUS dataset into the model for iterative training until the loss function converges, and obtain the trained echocardiogram segmentation model;

[0016] (4) Input the acquired image into the trained echocardiography segmentation model to obtain the echocardiography segmentation result.

[0017] Furthermore, the preprocessing process is as follows: First, the fan-shaped region of interest (ROI) is selected by covering the window, and interface interference elements outside the ROI are automatically cropped and removed; then, the original grayscale single-channel data is copied and expanded into a three-channel tensor, and Z-score normalization is performed according to ImageNet statistics; finally, the normalized image is scaled to a uniform resolution by bilinear interpolation to obtain the preprocessed image to be segmented.

[0018] Furthermore, the global flow branch Hiera backbone unit and the local flow branch MobileViT unit in the dual-stream hybrid feature extraction network adopt an asymmetric architecture. The global flow branch Hiera backbone unit includes four stages. Each stage uses a masked unit attention mechanism to divide the feature map into non-overlapping local windows and calculate attention only within the windows to balance computational efficiency and receptive field expansion. Based on the pre-trained weights of Masked Autoencoders (MAE), it provides more stable initialization to improve feature generalization and convergence stability. The local flow branch MobileViT unit introduces a dual-branch transformation of local convolution and global transformation in parallel at each scale to enhance weak boundaries and high-frequency texture details. The two outputs of the local convolution branch and the global transformation branch are concatenated by channels and then subjected to 1×1 convolution for cross-domain feature fusion.

[0019] Furthermore, the feature fusion enhancement network performs cross-scale attention fusion and adaptive feature pyramid processing on the extracted dual-stream multi-scale features, including: aligning global and local stream features to the same number of channels through convolutional layers at each corresponding stage; using global stream features as query vectors and local stream features as key and value vectors, and calculating attention maps; multiplying the attention maps and value vectors and superimposing them back onto the global features to obtain fused features, thereby explicitly suppressing background noise and highlighting endocardial boundary-related responses; constructing an adaptive feature pyramid based on dynamic weights, performing global average pooling on the fused features, and then generating importance weights for the four stages of features through two convolutional layers, and obtaining dynamic weight vectors through Softmax normalization; weighting the fused features of each stage according to the dynamic weight vectors to obtain a multi-scale decoding feature sequence, thereby improving scale invariance adaptation to differences in heart size among different patients.

[0020] Furthermore, the feature fusion enhancement network introduces an anatomical awareness attention mechanism and an anatomical consistency loss function to constrain the topology and adjacency relationships of the fused features. This includes: before entering the MLP decoder, generating probability distribution maps of four anatomical regions—background, left ventricle, left atrium, and myocardium—from the decoded features, and then calculating and fusing region-specific attention and global contextual attention based on these probability distributions to enhance the stable response of blurred boundaries and low signal-to-noise ratio regions; constructing a 4×4 anatomical relationship matrix; calculating the spatial overlap intensity of any two classes in the predicted probability map output by the network; and defining an anatomical consistency loss, which suppresses chamber misalignment, boundary breaks, and isolated artifacts through gradient penalty when the prediction result violates topological common sense.

[0021] Furthermore, the MLP decoder first unifies the multi-scale features output by the adaptive feature pyramid to a fixed channel dimension through linear projection. Then, it upsamples the features to a uniform 1 / 4 resolution through bilinear interpolation and stitches them together. Finally, after spatial refinement of the fused features by a lightweight convolutional layer, the final segmentation probability map is predicted through the MLP layer to avoid the checkerboard effect caused by deconvolution and reduce the computational burden.

[0022] Furthermore, the loss function of the echocardiography segmentation model adopts Dice loss as the main loss function to address the imbalance between foreground and background pixels; Focal loss is used to mine difficult-to-separate samples such as blurred boundaries; anatomical consistency loss is used to constrain topological correctness; and an auxiliary prediction head is introduced in the middle layer of the feature pyramid to define deep supervision loss to alleviate the gradient vanishing problem in deep networks. The joint loss function is obtained by weighted summation of the above losses.

[0023] Furthermore, during the iterative training of the model, images from different subjects are used for training and testing to ensure data integrity. A LoRA structure is introduced into the attention layer of the global flow branch, freezing the original pre-trained weight matrix and training only the low-rank matrix. For the local flow branch, feature fusion enhancement network, and MLP decoder head, all network parameters are trained to enhance local texture correction, multi-scale interaction, and decoding capabilities. In each training iteration, images from the CAMUS dataset are input into the model for forward propagation, and the joint loss is calculated for backpropagation to update the trainable parameters in the network. This iterative process is repeated until the joint loss converges, resulting in a trained echocardiogram segmentation model. During the inference phase, only the main prediction output of the MLP decoder head is used; auxiliary prediction heads are discarded to avoid increasing inference computational overhead.

[0024] 2. A real-time echocardiographic segmentation system based on hybrid visual and anatomical perception, comprising:

[0025] The image acquisition module is used to acquire the pixel stream of ultrasound images in real time by screen capture and provides a semi-transparent, borderless overlay window to select a sector-shaped ROI.

[0026] The preprocessing module is used to perform cropping, channel duplication, Z-score normalization, and scale unification on the fan-shaped ROI image to obtain a preprocessed image;

[0027] The segmentation model construction module is used to construct an echocardiogram segmentation model consisting of a dual-stream hybrid feature extraction network, a feature fusion enhancement network, and an MLP decoder connected in series.

[0028] The training module is used to define the joint loss function of the echocardiogram segmentation model. The existing echocardiogram dataset is input into the segmentation model for forward propagation to output the prediction result, and backpropagation is performed based on the joint loss function to update the network parameters. The process is iterated until the joint loss function converges to obtain the trained echocardiogram segmentation model.

[0029] The inference engine module is used to load the trained echocardiogram segmentation model and perform forward inference on the preprocessed image, outputting a segmentation mask or segmentation probability map.

[0030] The display rendering module is used to restore the segmentation mask to the original resolution through nearest neighbor interpolation. It pre-calculates the mapping table from category index to transparency to RGB value using a lookup table method, simplifies the color mixing rendering operation to integer operations to reduce the rendering time of a single frame, and displays it on the ultrasound image as a semi-transparent color layer to achieve real-time visualization assistance.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] Firstly, this invention employs a dual-stream hybrid feature extraction network architecture combining Hiera and MobileViT. The global stream branch Hiera captures the overall macroscopic topological structure and long-range semantic dependencies of the heart, while the local stream branch MobileViT extracts texture edge details under high-frequency speckle noise. This approach balances global semantic understanding with local texture sensitivity, effectively solving the dual problems of limited receptive field in pure convolutional methods and computationally intensive and local texture-ignoring pure Transformer methods. This enables accurate segmentation of weak boundary structures under complex ultrasound imaging conditions.

[0033] Secondly, this invention introduces an anatomical perception attention mechanism and an anatomical consistency loss function. By explicitly embedding the prior knowledge of medical experts into the network decoding process through a predefined anatomical relationship matrix, the model is forced to learn the spatial adjacency relationships and topological constraints between different heart chambers and myocardium. This effectively solves the topological errors such as myocardial rupture, chamber misalignment, or isolated noise that often occur in pure pixel classification methods, and significantly improves the clinical interpretability and confidence of the segmentation results.

[0034] Third, this invention employs a cross-scale attention fusion and adaptive feature pyramid mechanism. By dynamically adjusting the fusion ratio of feature maps at different scales according to the input content through the dynamic weight vector, the model can automatically increase the weight of deep features for large heart samples to obtain a larger receptive field and automatically increase the weight of shallow features for small heart samples to retain more details. This effectively solves the generalization problem caused by large differences in heart size and multiple cross-sectional perspectives, and enhances the robustness of the model in multi-section and multi-disease scenarios.

[0035] Fourth, this invention addresses the problem that existing deep learning segmentation methods typically calculate a single loss function only at the final output layer, leading to the gradient vanishing problem during deep network training. This results in insufficient sensitivity of shallow and mid-layer features to anatomical structures and slow convergence. The invention employs a deep supervised training mechanism, applying direct supervision signals to multiple intermediate layers of the decoder. It also combines a joint loss optimization strategy using Dice loss, Focal loss, anatomical consistency loss, and deep supervision loss. This effectively alleviates the gradient vanishing problem, accelerates model convergence, and ensures the sensitivity of network features to anatomical structures, further enhancing the ability to capture weak signal structures and improving the stability of the training process.

[0036] Fifth, this invention addresses the problems of existing ultrasound-assisted diagnostic systems, which typically rely on medical device manufacturers' open underlying DICOM interfaces or data protocols for data acquisition, resulting in high deployment barriers and poor versatility. It also addresses the issues of large model solutions, such as CN120635459A, having a massive number of parameters requiring high-end GPU hardware support and making real-time inference difficult on ordinary clinical ultrasound workstations. To address these problems, this invention constructs a universal, non-invasive interface based on screen capture, adapting to various ultrasound equipment models without needing to interface with any underlying device protocols. Simultaneously, it employs a LoRA lightweight fine-tuning strategy, requiring only a minimal number of low-rank parameters to adapt to specific data distributions. Furthermore, it combines ONNX Runtime inference optimization and a producer-consumer parallel pipeline architecture to achieve edge-side acceleration, effectively reducing the hardware computing power threshold of the system. This enables plug-and-play functionality and real-time high-frame-rate inference on ordinary existing ultrasound workstations, greatly promoting the feasibility of clinical deployment. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the implementation of the real-time echocardiography segmentation method according to an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the echocardiographic segmentation model constructed in the method of this invention;

[0039] Figure 3 This is a schematic diagram illustrating the definition of the anatomical relationship matrix and the principle of anatomical consistency constraints in the method of this invention;

[0040] Figure 4 This is a block diagram of the real-time echocardiography segmentation system according to an embodiment of the present invention;

[0041] Figure 5 This is a diagram showing the segmentation results of this invention on the CAMUS dataset. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions and effects of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the present invention and not all of the embodiments.

[0043] The step numbers in this specification and claims are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

[0044] Example 1: A real-time echocardiogram segmentation method based on hybrid visual and anatomical perception.

[0045] Reference Figure 1 The implementation steps of this embodiment include the following:

[0046] Step 1: Generate the preprocessed image to be segmented.

[0047] 1.1) Acquired fan-shaped images of ultrasound images using a non-invasive screen visual capture method:

[0048] To address the pain points of closed data interfaces of medical devices, complex DICOM protocols, or other complex interfaces, this invention adopts a non-invasive screen visual capture strategy. Specifically, it utilizes the underlying graphics API of the computer operating system to capture the display window in real time. In order to meet the real-time requirements of ultrasound dynamic scanning, which is usually 30-60 FPS, a semi-transparent, borderless overlay window is provided through the client. Users can select the fan-shaped region (ROI) of the ultrasound image by dragging the mouse.

[0049] 1.2) Preprocess the acquired images:

[0050] 1.2.1) Region Clipping: By covering the window, a fan-shaped region ROI is selected, and UI interference elements outside the ROI, such as text labels, menu bars, and measurement rulers, are automatically clipped and removed;

[0051] 1.2.2) Channel Copying and Standardization: Perform channel copying on the original grayscale single-channel data to expand it into... A three-channel tensor is used to adapt to the input of the pre-trained model; then, Z-score normalization is performed based on statistics from the large-scale ImageNet dataset. The normalized output tensor is:

[0052] ;

[0053] in, The input tensor after channel copying. It is a three-channel mean vector. This is a three-channel standard deviation vector;

[0054] 1.2.3) Scale uniformity: The standardized image is scaled to a standard resolution of 224×224 using bilinear interpolation to obtain the preprocessed image to be segmented.

[0055] Step 2: Construct an echocardiographic segmentation model.

[0056] Reference Figure 2 The implementation of this step includes:

[0057] 2.1) Establish a dual-stream hybrid feature extraction network.

[0058] To balance global semantic understanding and local detail extraction, this step designs an asymmetric dual-stream architecture, including the global stream branch Hiera backbone unit and the local stream branch MobileViT unit. The structure and function of each unit are as follows:

[0059] 2.1.1) Global flow branch Hiera backbone unit:

[0060] This branch is responsible for extracting the overall macroscopic topological structure and long-range semantic dependencies of the heart. It adopts the Hiera architecture, specifically the Hiera-Tiny variant. Unlike traditional ViT, which relies on additional [CLS] tokens for classification and aggregation, the Hiera architecture significantly reduces computational redundancy by removing [CLS] tokens, absolute position encoding, and complex attention heads, making it more suitable for pixel-level dense prediction tasks.

[0061] The network is divided into four levels. As the level increases, the spatial resolution of the feature map decreases while the number of channels increases. Specifically, the first level receives a 224×224×3 input, performs patch embedding through convolutional kernels, and outputs a 56×56×96 feature map. The second stage outputs a 28×28×192 feature map. The third stage outputs a 14×14×384 feature map. The fourth stage outputs a 7×7×768 feature map. .

[0062] Within each stage, a masked unit attention mechanism replaces global self-attention. This mechanism divides the feature map into non-overlapping local windows, calculating attention only within each window. This hierarchical design, while preserving local feature interactions, achieves exponential expansion of the receptive field through layer stacking, enabling the fourth-stage features to cover the entire heart region. Furthermore, this branch loads pre-trained weights based on MAE, providing more stable initialization, improving feature generalization and convergence stability, and reducing annotation dependencies.

[0063] 2.1.2) Local Flow Branch MobileViT Unit:

[0064] To address the issue of high-frequency detail loss during downsampling in hierarchical Transformers, this branch introduces an improved MobileViT component in parallel at each hierarchical stage, aiming to construct a hybrid feature representation combining the CNN's local inductive bias and the Transformer's global receptive field. Unlike traditional sequential stacking, this architecture deploys the MobileViT component at all four feature scales and proposes an adaptive patch embedding strategy: targeting the first two high-resolution stages (i.e., the number of channels...) A 2×2 patch size is used to reduce sequence length and aggregate local textures; while in the deep semantic stage, a 1×1 patch size is used to preserve dense semantic information.

[0065] Each MobileViT component internally performs a two-stream branch feature transformation that combines local convolution and global transformation in parallel.

[0066] The local convolution branch uses 3×3 depth convolution combined with group normalization and SiLU activation function to efficiently extract local spatial texture features, and performs channel mixing through 1×1 pointwise convolution.

[0067] The global transformation branch first maps spatial features into a token sequence through Patch embedding via convolution, then uses a Transformer layer to perform self-attention computation to capture long-range dependencies, and finally restores spatial resolution through bilinear interpolation.

[0068] The two features are finally fused across domains through channel concatenation and 1×1 convolution. This design effectively compensates for the smoothing effect of Hiera in shallow feature extraction and significantly enhances the model's ability to distinguish myocardial texture from similar echo non-cardiac tissues.

[0069] 2.2) Establish a feature fusion enhancement network:

[0070] 2.2.1) Cross-scale attention fusion mechanism:

[0071] In each corresponding stage of the encoder, i.e., stages 1 to 4, this example does not simply perform feature concatenation, but instead introduces an attention-guided fusion mechanism. The core idea of ​​this mechanism is to use the deep semantic information of the global stream as guidance to filter background noise in the local stream. Specifically, let... For the first Stage-based global flow characteristics For the first Stage-specific local flow characteristics First, align the two to the same number of channels using a convolutional layer; then use the global features as the query vector. Using local features as key vectors Sum value vector Calculate attention map :

[0072] ;

[0073] in, The channel scaling factor is T, which represents the transpose. This attention map characterizes the degree of attention that global semantics pays to local regions.

[0074] Attention map and value vector After multiplication, the output is projected and then superimposed back onto the global features to obtain the fused features. In this way, only local texture features related to the anatomy of the heart are activated, while rib artifacts, lung air interference, or other impurities unrelated to the heart are effectively suppressed.

[0075] 2.2.2) Constructing an adaptive feature pyramid:

[0076] Considering the significant differences in heart size among different patients—for example, the physical size difference between the enlarged heart of someone with dilated cardiomyopathy and the heart of a child can be several times—fixed-ratio feature fusion is difficult to adapt to such scale variations. Therefore, this example designs a dynamic weight generation network to achieve scale invariance.

[0077] First, the features fused from each stage are adjusted to a uniform size and then concatenated along the channel dimension to obtain the aggregated features. The aggregated features are then compressed into a global context vector using global average pooling. This is followed by a non-linear mapping using a two-layer convolutional layer containing both a dimensionality reduction and an up-dimensionality layer. The output consists of four weight scalars, each corresponding to the importance of the features at one of the four stages. Finally, the vector is normalized using the Softmax function to obtain a dynamic weight vector. :

[0078] ;

[0079] The final multi-scale decoding feature sequence is obtained by weighting the features from each stage according to this weight vector. For large heart samples, the network automatically increases the weight of the fourth stage, i.e., the deep features. To obtain a larger receptive field; for small heart samples, the weight of the first stage, i.e., shallow features, is increased. To retain more details, The weights are for the second and third stages, respectively.

[0080] 2.2.3) Anatomical sensory attention mechanism:

[0081] To enable the decoding process to have explicit structural cognitive ability, this example constructs an anatomical perception attention mechanism before the decoding head.

[0082] Reference Figure 3 The implementation of this step includes:

[0083] First, probability distribution maps of four anatomical regions—background 0, left ventricle 1, left atrium 2, and myocardium 3—are generated from the features at various scales output by the feature fusion enhancement network.

[0084] Subsequently, region-specific attention and global contextual attention are calculated based on the probability distribution, and the two are fused to obtain enhanced features, enabling the network to maintain a stable response to anatomical structures even in blurred boundaries and low signal-to-noise ratio regions.

[0085] 2.2.4) Anatomical Relationship Matrix and Anatomical Consistency Loss:

[0086] This example constructs a 4×4 anatomical relation matrix. This quantifies the structural relationship between the heart chambers and myocardium as understood by medical experts, using it as a static prior standard. Elements in the matrix... Defined categories With category The strength of the prior association between them, for example, setting the strength of the prior association between left ventricle 1 and myocardium 3. To enforce the physiological fact that the left ventricle (LV) is surrounded by myocardium, a high value close to 1 is used to achieve strong constraint; the prior correlation strength between LV 1 and background 0 is set. To strongly penalize erroneous predictions where the left ventricle is in direct contact with the background; to define the prior association strength between left atrium 2 and myocardium 3. To characterize the adjacency relationship between the left atrial LA and myocardial MYO at the base; to set To constrain its own connectivity and intra-class consistency.

[0087] To embed the aforementioned prior knowledge into network training, a differentiable anatomical consistency loss function is defined. :

[0088] For the predicted probability graph of the network output According to pixel coordinates This belongs to the category Predicted probability Calculate any two categories and Spatial overlap intensity between :

[0089] ;

[0090] in, This represents the average of all pixels within the sector-shaped ROI, used to measure the spatial correlation between the two classes.

[0091] All non-diagonal category pairs ( Predicted overlap The sum of the absolute errors of the differences between the elements of the matrix and the preset matrix is ​​defined as the loss function. :

[0092] ;

[0093] in, This represents the total number of off-diagonal category pairs. When the network predicts results that violate anatomical common sense, such as the left ventricle being isolated in the background, The calculated value will be much larger than the preset value. This results in a huge gradient penalty, driving the network parameters to be updated in a direction that conforms to anatomical constraints.

[0094] 2.3) Establishing the MLP decoding head:

[0095] To further improve the granularity of the final segmentation probability map and reduce the computational burden, this example introduces a lightweight MLP decoding head after feature fusion. This decoding head abandons the traditional complex deconvolution operation and adopts a lightweight MLP architecture, the implementation of which includes:

[0096] First, features from different scales of the adaptive feature pyramid are uniformly projected to a fixed channel dimension using linear projection;

[0097] Subsequently, all features were upsampled to a uniform 1 / 4 resolution using bilinear interpolation and then stitched together.

[0098] Finally, after spatial refinement of the fused features by a lightweight convolutional layer, the final segmentation probability map is predicted by an MLP layer.

[0099] This design not only effectively avoids the checkerboard effect often caused by deconvolution operations and ensures the smoothness of the segmentation boundaries, but also significantly reduces the number of parameters and improves inference speed.

[0100] 2.4) The above dual-stream hybrid feature extraction network, feature fusion enhancement network, and MLP decoder are cascaded in sequence to form an echocardiogram segmentation model.

[0101] Step 3: Define the loss function and train the echocardiogram segmentation model.

[0102] 3.1) Set multiple losses:

[0103] 3.1.1) This example employs a strategy of joint optimization using multiple loss functions, setting multiple losses including: Dice loss, Focal loss, dissection consistency loss, and deep supervision loss, where:

[0104] The Dice loss As the main loss function, it is used to solve the problem of severe imbalance between foreground and background pixels in echocardiography segmentation. This Dice loss directly optimizes the overlap between the prediction and the ground truth, making the model pay more attention to the smaller foreground region.

[0105] The Focal loss This Focal loss is used to discover difficult-to-classify samples such as those with ambiguous boundaries. It improves the classification accuracy of focusing on ambiguous regions such as the endocardial boundary by down-weighting easily classified samples and up-weighting difficult samples.

[0106] The anatomical consistency loss is calculated using the anatomical consistency loss function defined in step 2.2.4). Constrain topological correctness to ensure that the segmentation results conform to anatomical priors;

[0107] The deep supervised loss is a deep supervised training mechanism introduced to address the gradient vanishing problem that may occur during the training of deep networks and to force the intermediate layers of the network to learn discriminative anatomical features. Specifically, in the feature fusion enhancement network, lightweight auxiliary classification heads are introduced in parallel as auxiliary prediction heads on multiple intermediate feature layers of different resolution scales of the adaptive feature pyramid output. Each auxiliary head directly predicts the final segmentation result and calculates the loss with the ground truth label. Its mathematical expression is: ;

[0108] in, To assist in determining the number of prediction heads, the value is taken as the number of layers in the feature pyramid; To assist in predicting the head index, For the first Auxiliary classification head, For the first Intermediate feature map of the layer Use truth-value separators as labels. For the standard cross-entropy function, Deep monitoring of losses.

[0109] 3.1.2) Based on the above losses, the joint loss function is calculated. :

[0110] ,

[0111] in, The weighting coefficients for Dice loss, Focal loss, anatomical consistency loss, and deep supervision loss are 0.8, 0.2, 0.1, and 0.4, respectively. As the main loss function, it is used to solve the problem of foreground and background pixel imbalance; It is used to mine difficult-to-distinguish samples such as fuzzy boundaries, and its focusing parameter is set to 2.0. This multi-point supervision mechanism ensures that the network can obtain direct gradient feedback in shallow layers, which accelerates model convergence and significantly improves the ability to capture weak signal structures.

[0112] 3.2) Training the echocardiogram segmentation model:

[0113] During training, the loss gradient output by the network's main prediction head needs to be backpropagated through the entire decoder to the encoder. This can lead to vanishing gradients or slow training in deep networks. Deep supervised training mechanisms, by directly applying supervisory signals to intermediate layers, allow each layer to receive gradient feedback directly from the ground truth labels. This accelerates convergence, alleviates vanishing gradients, enhances the quality of feature extraction from intermediate layers, and improves training stability. Its implementation includes the following:

[0114] 3.2.1) Dataset partitioning: Images from different subjects in the CAMUS dataset are used as the training and testing sets to ensure that data is not leaked.

[0115] 3.2.2) Set up a fine-tuning strategy for differentiated parameters.

[0116] To enable complex Transformer models to be trained on consumer-grade GPUs, such as the RTX 3090 or lower, this example employs LoRA technology and differentiates the training strategy for the dual-stream architecture, specifically:

[0117] For the Hiera backbone unit of the global flow branch, a LoRA structure is introduced into its attention layer to freeze the original pre-trained weight matrix. Only two low-rank matrices are trained. and Perform forward propagation to obtain the current output of the linear layer. .

[0118] ;

[0119] in, This is the original pre-trained weight matrix. For the input feature vector, and It is a trainable low-rank matrix. This is the scaling factor. The rank is the rank of the low-rank decomposition. Using the LoRA strategy can not only reduce the number of parameters trained on the Hiera backbone unit, but also achieve good feature extraction results;

[0120] For the local flow branch MobileViT unit, since it has a small number of parameters and is mainly responsible for local texture correction, full fine-tuning is performed to obtain the best feature interaction capability.

[0121] For the feature fusion enhancement network and the MLP decoder, the same approach of training all network parameters is adopted to enhance multi-scale interaction and decoding capabilities.

[0122] This hybrid strategy freezes most of the backbone network parameters, significantly reduces memory usage, and effectively lowers the hardware cost for clinical deployment.

[0123] 3.3) Iterative training process:

[0124] 3.3.1) In each training iteration, the images from the CAMUS dataset are preprocessed and input into the echocardiography segmentation model for forward propagation. The main segmentation probability map is output through the MLP decoding head; simultaneously, the auxiliary segmentation result is output through the auxiliary prediction head, and the deep supervision loss is calculated by combining the ground truth segmentation labels. ;

[0125] 3.3.2) Calculate the joint loss by combining the outputs of the MLP decoder and the auxiliary prediction head. Perform backpropagation, calculate its gradient, and use the optimizer to update the trainable parameters in the network;

[0126] 3.3.3) Repeat steps 3.3.1) and 3.3.2) above until the combined loss is reached. Once convergence is achieved, a well-trained echocardiogram segmentation model is obtained.

[0127] Step 4: Input the acquired images for segmentation and inference.

[0128] The preprocessed acquired images are then input into the trained echocardiography segmentation model to obtain the segmentation results of the echocardiography. The segmentation results can be overlaid on the original ultrasound image in a semi-transparent manner on the client side to achieve real-time visualization assistance.

[0129] It should be noted that during the inference phase, all auxiliary prediction heads are discarded, and only the output of the main prediction head is used. Therefore, the deep supervision mechanism does not increase the computational overhead during inference.

[0130] Example 2: Real-time echocardiography segmentation system based on hybrid vision and anatomical perception.

[0131] Reference Figure 4 The echocardiography segmentation system of this embodiment includes: an image acquisition module 1, a preprocessing module 2, a segmentation model construction module 3, a training module 4, an inference engine module 5, and a display rendering module 6. The segmentation model construction module 3 includes: a dual-stream hybrid feature extraction submodule 31, a feature fusion enhancement submodule 32, and an MLP prediction decoding submodule 33; the inference engine module 5 includes: a parallel pipeline submodule 51 and an ONNX Runtime inference optimization submodule 52.

[0132] The working principle of the entire system is as follows:

[0133] The image acquisition module 1 is used to acquire the pixel stream of ultrasound images in real time by screen capture, and provides a semi-transparent, borderless overlay window to select the fan-shaped ROI and transmit it to the preprocessing module 2.

[0134] The preprocessing module 2 is used to perform cropping, channel duplication, Z-Score normalization and scale unification on the fan-shaped ROI image to obtain a preprocessed image, and then transmit it to the inference engine module 5.

[0135] The segmentation model construction module 3 is used to construct an echocardiogram segmentation model consisting of a dual-stream hybrid feature extraction network, a feature fusion enhancement network, and an MLP decoder connected in series, wherein:

[0136] The dual-stream hybrid feature extraction submodule 31 includes a global stream branch Hiera backbone unit and a local stream branch MobileViT unit, which are used to extract global semantic features and local texture features of the preprocessed image respectively, and output multi-scale dual-stream features.

[0137] The feature fusion enhancement submodule 32 is used to perform cross-scale attention fusion and adaptive feature pyramid processing on the extracted dual-stream multi-scale features to obtain fused features. By introducing an anatomical perception attention mechanism and an anatomical relationship matrix, the topology and adjacency relationship of the fused features are constrained to obtain enhanced fused features.

[0138] MLP prediction decoding submodule 33 is used to perform channel mapping and uniform resolution upsampling on the enhanced fusion features, refine them with lightweight convolution, and then predict the final segmentation mask through a multilayer perceptron to generate a segmentation probability map of the preprocessed image.

[0139] The training module 4 is used to define the joint loss function of the echocardiogram segmentation model. The existing echocardiogram dataset is input into the segmentation model for forward propagation to output the prediction result, and backpropagation is performed based on the joint loss function to update the network parameters. The process is iterated until the joint loss function converges to obtain the trained echocardiogram segmentation model.

[0140] The inference engine module 5 is used to load the trained echocardiogram segmentation model and perform forward inference on the preprocessed images, outputting a segmentation probability map and transmitting it to the display and rendering module 6. This achieves a processing speed of 30-60 FPS or higher on medical workstations where hardware resources may be limited. Through a precisely parallel pipeline architecture designed on the client side, a producer-consumer model is used to decouple the image acquisition, inference, and rendering processes, wherein:

[0141] The parallel pipeline submodule 51 is used to complete three core threads: the acquisition thread is responsible for frequently calling the underlying graphics API to capture the screen pixel stream, encapsulating it into frame objects and pushing it into the raw data queue; the inference thread retrieves frame data from the queue, performs preprocessing, and calls the ONNX Runtime inference engine to push the generated segmentation mask into the result queue; the rendering thread obtains the mask from the result queue and performs post-processing and drawing. The threads are synchronized using mutexes and condition variables to avoid wasting CPU resources due to busy waiting.

[0142] The ONNX Runtime Inference Optimization Submodule 52 is used to enable the Memory Arena function, pre-allocate a large contiguous memory pool, and calculate intermediate tensors to avoid fragmentation and performance jitter caused by repeatedly requesting and releasing memory during each frame of inference; at the same time, it enables operator fusion optimization, which merges consecutive convolution, normalization and activation layers into a super operator to reduce the number of GPU kernel startups.

[0143] The display rendering module 6 is used to restore the segmentation probability map to the original resolution through nearest neighbor interpolation. At the rendering level, in order to accelerate post-processing, a lookup table method is used to replace floating-point color mixing calculations. A mapping table of category index and transparency to RGB values ​​is pre-calculated, simplifying the color mixing rendering operation into extremely low-latency integer addition and bitwise operations, reducing the rendering time of a single frame to less than 15ms. Finally, it is displayed on the ultrasound image as a semi-transparent color layer, realizing real-time visualization assistance.

[0144] It should be noted that the above functional modules can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as program instruction products. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium, or transferred from one computer's readable and writable storage medium to another.

[0145] The direct coupling or communication connections between the modules shown or discussed in this embodiment can be achieved through indirect coupling or communication connections via interfaces, devices, or modules. The various functional modules and sub-modules in this embodiment can dynamically reside within a single processing unit, or each module can exist physically independently, or two or more modules can dynamically reside within a single processing unit. When these dynamic components are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. This storage medium can be a memory, disk, or optical disc, etc.

[0146] The effectiveness of this invention can be further illustrated by the following experimental results:

[0147] I. Experimental Conditions

[0148] Training and testing used images from different subjects to ensure data confidentiality.

[0149] The Dice coefficient is used as a quantitative evaluation metric for segmentation accuracy, and inference latency is measured on an NVIDIA RTX 3090 configuration.

[0150] Measurements were taken on a workstation containing a graphics card.

[0151] II. Experiment Content

[0152] Experiment 1: The method of this invention was used to perform complete acquisition, preprocessing, inference, and post-processing of echocardiogram videos, and single-frame images were extracted for result display. The results are as follows: Figure 5 The image consists of four parts: the top left is the acquired and preprocessed Original Image; the top right is the expert-annotated Ground Truth; the bottom left is the Prediction output by the model of this invention; and the bottom right is the Overlay, which overlays the predicted mask onto the original ultrasound image through the display rendering module. In the segmentation image, black represents the Background, red represents the left ventricle, blue represents the left atrium, and green represents the myocardium.

[0153] from Figure 5 As can be seen, the prediction results obtained by the method of this invention are highly consistent with the actual labels. Combined with the overlay display, it can be seen that even with speckle noise interference and blurred boundaries in the original ultrasound image, this method can still accurately extract the boundaries of the endocardium and epicardium. Furthermore, the prediction results strictly adhere to the physiological and anatomical topology of the left ventricle being surrounded by myocardium and the left atrium being connected to the base of the myocardium, without any topological errors such as myocardial rupture, chamber adhesion, or isolated outlier artifacts. This fully demonstrates the high accuracy and anatomical rationality of the hybrid visual architecture and anatomical perception mechanism of this invention in echocardiographic segmentation.

[0154] Experiment 2: Under the above conditions, a comparative experiment on echocardiogram segmentation was conducted using the method of this invention and existing SAM-based medical ultrasound segmentation models SAMUS and adaptive U-shaped networks nnU-Net on the publicly available CAMUS dataset. The results are shown in Table 1:

[0155] Table 1. Comparison of segmentation results of the method of this invention with other methods on the CAMUS dataset.

[0156]

[0157] In Table 1, LV represents the left ventricle, LA represents the left atrium, and MYO represents the myocardium.

[0158] As shown in Table 1, the present invention achieves an average Dice coefficient of 0.911, which is competitive with or superior to the comparison methods in terms of overall segmentation accuracy compared to SAMUS's 0.909 and nnU-Net's 0.893. Notably, the present invention achieves a Dice coefficient of 0.898 for myocardial segmentation, significantly better than SAMUS's 0.875 and nnU-Net's 0.860, indicating that the anatomical perception attention mechanism and anatomical consistency loss have a significant effect on improving the segmentation accuracy of weakly boundary structures such as myocardium.

[0159] In terms of topology, the contrastive method occasionally predicts isolated pixel islands that are free outside the left ventricle, while the method of this invention, by introducing anatomical consistency loss, forces the model to learn a topological distribution that conforms to physiological structure by penalizing spatial distributions that violate common anatomical knowledge, thus significantly eliminating outlier artifacts. Its output strictly follows the topological constraint that the left ventricle is surrounded by myocardium.

[0160] In terms of inference speed, the single-frame inference latency of this invention is only 13-15ms, far lower than SAMUS's 35-50ms and nnU-Net's 40-60ms, fully meeting the stringent real-time requirements of 30-60FPS for dynamic ultrasound scanning in clinical settings. This is due to the combined effects of the Hiera lightweight architecture, the reduced computational load from the LoRA freezing strategy, and the ONNX Runtime inference optimization.

[0161] It should be noted that the above descriptions are merely two preferred embodiments of the present invention and do not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and detail without departing from the principles and structure of the present invention, for example:

[0162] In terms of backbone network architecture, Hiera-Tiny in the global flow branch can be replaced by a hierarchical backbone network such as SwinTransformer, Pyramid Vision Transformer (PvT) or ConvNeXt, which also have the ability to extract multi-scale global features. MobileViT in the local flow branch can be replaced by a lightweight CNN architecture such as EfficientNet, ShuffleNetV2 or GhostNet, or even feature maps extracted by traditional Sobel or Canny edge detection operators can be used as auxiliary input.

[0163] Regarding feature fusion mechanisms, cross-scale attention fusion can be replaced by simple channel concatenation followed by 1×1 convolution. Although this may reduce the ability to capture long-distance dependencies, it can further reduce the amount of computation. The adaptive feature pyramid can be replaced by the bidirectional feature pyramid network BiFPN, which enhances feature flow through bidirectional paths.

[0164] In terms of anatomical constraints, in addition to the matrix-defined anatomical consistency loss, Hausdorff distance loss or boundary loss can be introduced to constrain the segmentation shape. Star-shaped convexity constraints can also be introduced to prevent topological errors from a geometric perspective, or conditional random fields, maximum connected component analysis and other methods can be used in the post-processing stage to correct topological errors.

[0165] In terms of fine-tuning strategies, in addition to LoRA, other efficient fine-tuning methods can be adopted, such as inserting small MLP modules (Adapter Tuning) between layers or concatenating learnable vectors (Prompt Tuning) at the input.

[0166] In terms of image acquisition, for devices with higher openness, a real-time monitoring service based on DICOM network transmission can be developed to replace screen capture.

[0167] However, these modifications and alterations based on the ideas of this invention are still within the scope of protection of the claims of this invention.

Claims

1. A real-time echocardiogram segmentation method based on hybrid visual and anatomical perception, characterized in that, include: (1) Acquire the fan-shaped region image of ultrasound image using a non-invasive screen visual capture method, and preprocess it to generate a preprocessed image to be segmented; (2) Construct an echocardiographic segmentation model including a dual-stream hybrid feature extraction network, a feature fusion enhancement network, and an MLP decoder: The dual-stream hybrid feature extraction network includes a global stream branch Hiera backbone unit and a local stream branch MobileViT unit, which are used to extract global semantic features and local texture features of the preprocessed image and output multi-scale dual-stream features. The feature fusion enhancement network is used to perform cross-scale attention fusion and adaptive feature pyramid processing on the extracted dual-stream multi-scale features to obtain fused features. By introducing an anatomically perceptual attention mechanism, the topology and adjacency relationship of the fused features are constrained to obtain enhanced fused features. The MLP decoding head is used to perform channel mapping, feature refinement and upsampling on the enhanced fusion features to generate a segmentation probability map of the preprocessed image; (3) Define the loss function of the echocardiogram segmentation model, input the existing CAMUS dataset into the model for iterative training until the loss function converges, and obtain the trained echocardiogram segmentation model; (4) Input the acquired image into the trained echocardiography segmentation model to obtain the echocardiography segmentation result.

2. The method according to claim 1, characterized in that, The preprocessing package for the acquired images in (1) Includes: 1a) Selecting a sector-shaped region of interest (ROI) by covering the window and automatically cropping out interface interference elements outside the ROI; 1b) Copying and expanding the original grayscale single-channel data into a three-channel tensor, and performing Z-strain analysis according to ImageNet statistics. Score normalization, the normalized output tensor: ,in, The input tensor after channel copying. This is the three-channel mean vector; This is a three-channel standard deviation vector; 1c) Scale the normalized image to [size using bilinear interpolation]. .

3. The method according to claim 1, characterized in that, The global flow branch Hiera backbone unit and the local flow branch MobileViT unit in the dual-flow hybrid feature extraction network in (2) adopt an asymmetric architecture, wherein: 2a) the global flow branch Hiera backbone unit includes 4 stages, and its input is The output multi-scale global features are as follows: , , , This unit employs a masked unit attention mechanism within each stage, dividing the feature map into non-overlapping local windows and calculating attention only within those windows to balance computational efficiency and receptive field expansion. Based on pre-trained weights from MAE, it provides more stable initialization to improve feature generalization and convergence stability. 2b) The local flow branch MobileViT unit introduces a dual-branch transformation of local convolution and global transformation in parallel at each scale to enhance weak boundaries and high-frequency texture details, wherein: This local convolution branch uses Depthwise convolution combined with grouping normalization and SiLU activation function, followed by channel blending via 1×1 pointwise convolution, is used to extract local spatial texture features. It employs an adaptive patch embedding strategy, where the number of channels... When adopted Patch size is adjusted to reduce sequence length and aggregate local textures, employing [a specific approach] in the deep semantic stage. Patch size to preserve dense semantic information; This global transformation branch uses convolution to embed Patch to obtain a Token sequence, and then performs self-attention computation through the Transformer layer to capture long-range dependencies. Finally, it restores the spatial resolution through bilinear interpolation. The two outputs of the local convolution branch and the global transform branch are concatenated via a channel and then... Convolution is used for cross-domain feature fusion.

4. The method according to claim 1, characterized in that, The feature fusion enhancement network in (2) performs cross-scale attention fusion and adaptive feature pyramid processing on the extracted dual-stream multi-scale features, including: 2c) in each corresponding stage Global flow features and local flow features are aligned to the same number of channels using convolutional layers. Indicates the stage number; 2d) Uses global flow features as query vectors Using local flow features as key vectors AND value vector And calculate attention map ,in, For attention maps, For normalization function, For vector / matrix multiplication, for transpose, 2e) Multiply the attention map by the value vector and project it onto the output to obtain the fused features. To explicitly suppress background noise and highlight endocardial boundary-related responses, among which Indicates the first Features after stage fusion; 2f) Construct an adaptive feature pyramid based on dynamic weights, first processing the fused features. Global average pooling is performed, followed by two convolutional layers to generate importance weights for the four stages of features, which are then normalized using Softmax to obtain a dynamic weight vector. : , in, For dynamic weight vectors, These are the weight scalars corresponding to the four stages; 2g) The fusion features of each stage are weighted according to the dynamic weight vector to obtain a multi-scale decoding feature sequence, so as to improve the scale invariance adaptation to the differences in heart size among different patients.

5. The method according to claim 1, characterized in that, The feature fusion enhancement network introduced in (2) is dissected The perceptual attention mechanism and anatomical consistency loss function are used to constrain the topology and adjacency relationships of the fused features. This includes: 2h) Before the decoder, probability distributions for four anatomical regions (background, left ventricle, left atrium, and myocardium) are generated for each scale of the feature fusion enhancement network output. Region-specific attention and global contextual attention are calculated based on these probability distributions and then fused to enhance the response stability of blurred boundaries and low signal-to-noise ratio regions; 2i) Construction... Anatomical Relationship Matrix The category set is defined as background 0, left ventricle 1, left atrium 2, and myocardium 3, where Indicates category With category The strength of the prior association; 2j) Predicted probability graph of network output Calculate any two classes and Spatial overlap strength ; ， in, For category With category Spatial overlap intensity; pixel coordinates This belongs to the category The predicted probability; This indicates that the average value is calculated for all pixels within the sector-shaped region of interest (ROI). 2k) Define anatomical consistency loss : ， in, Loss of anatomical consistency; It is the absolute value; For the total number of non-diagonal category pairs, The number of categories is used to suppress chamber misalignment, boundary breakage, and isolated artifacts by gradient penalty when the prediction results violate topological common sense.

6. The method according to claim 1, characterized in that, The MLP decoding head includes: 2l) Unify the multi-scale features output by the adaptive feature pyramid to a fixed channel dimension through linear projection; 2m) Upsample the above features to a unified dimension through bilinear interpolation. The resolution is then stitched and fused; 2n) After spatial refinement of the fused features by a lightweight convolutional layer, the final segmentation probability map is predicted through an MLP layer to avoid the checkerboard effect caused by deconvolution and reduce the computational burden.

7. The method according to claim 1, characterized in that, The loss function for the echocardiogram segmentation model defined in (3) includes the following implementation: 3a) using Dice loss. As the main loss function, it is used to address the pixel imbalance between foreground and background; 3b) Focal loss is employed. Mining difficult-to-separate samples such as those with fuzzy boundaries; 3c) Using anatomical consistency loss Constraint topological correctness; 3d) Define deep supervised loss: ;in, To assist in determining the number of prediction heads, the value is taken as the number of layers in the feature pyramid. To assist in predicting the head index, For the first Auxiliary classification head, For the first Intermediate feature map of the layer; Use true values ​​as delimiters; The standard cross-entropy function; 3e) The joint loss function is calculated based on the above results. : ; in, These are the weighting coefficients for Dice loss, Focal loss, anatomical consistency loss, and deep supervision loss, respectively.

8. The method according to claim 1, characterized in that, The step (3) of inputting the CAMUS dataset into the echocardiography segmentation model for iterative training includes: 3f) using images from different subjects for training and testing to ensure data integrity; 3g) introducing a LoRA structure into the attention layer of the global flow branch, while maintaining the original pre-trained weight matrix, and training the low-rank matrix, i.e., updating the weights through forward propagation: This allows for fine-tuning of model parameters; among which, For linear layer output, For the frozen pre-trained weight matrix, For the input feature vector, and It is a trainable low-rank matrix. This is the scaling factor. 3h) The rank of the low-rank decomposition is used; 3i) The local flow branch, feature fusion enhancement network and MLP decoder head are trained with all network parameters to enhance local texture correction, multi-scale interaction and decoding capabilities; 3i) In each iteration of training, the images of the CAMUS dataset are input into the model for forward propagation, and the main segmentation probability map is output through the MLP decoder head. At the same time, on the intermediate feature layers of different resolution scales output by the adaptive feature pyramid in the feature fusion enhancement network, an auxiliary classification head is introduced in parallel as an auxiliary prediction head to output the auxiliary segmentation result. The deep supervision loss is calculated by combining the ground truth segmentation label. ; 3j) Calculate the joint loss by combining the outputs of the MLP decoder and the auxiliary prediction head. Perform backpropagation, calculate its gradient, and use the optimizer to update the trainable parameters in the network; 3k) Repeat steps 3i) and 3j) until the joint loss is reached. Once convergence is achieved, the trained echocardiogram segmentation model is obtained, and the training ends.

9. A real-time echocardiographic segmentation system based on hybrid visual and anatomical perception, characterized in that, include: The image acquisition module is used to acquire ultrasound image pixel streams in real time via screen capture, and provides semi-transparent, borderless... The bounding window selects the sector-shaped ROI; The preprocessing module performs cropping, channel duplication, Z-score normalization, and scale unification on the fan-shaped ROI image to obtain a preprocessed image; the segmentation model construction module is used to construct an echocardiogram segmentation model consisting of a dual-stream hybrid feature extraction network, a feature fusion enhancement network, and an MLP decoder connected in series. The training module is used to define the joint loss function of the echocardiogram segmentation model. The existing echocardiogram dataset is input into the segmentation model for forward propagation to output the prediction result, and backpropagation is performed based on the joint loss function to update the network parameters. The process is iterated until the joint loss function converges to obtain the trained echocardiogram segmentation model. The inference engine module is used to load the trained echocardiogram segmentation model and perform forward inference on the preprocessed image, outputting a segmentation mask or segmentation probability map. The display rendering module is used to restore the segmentation mask to the original resolution through nearest neighbor interpolation. It pre-calculates the mapping table from category index to transparency to RGB value using a lookup table method, simplifies the color mixing rendering operation to integer operations to reduce the rendering time of a single frame, and displays it on the ultrasound image as a semi-transparent color layer to achieve real-time visualization assistance.

10. The system according to claim 9, characterized in that: The segmentation model construction module includes: The dual-stream hybrid feature extraction submodule includes a global stream branch Hiera backbone network and a local stream branch MobileViT network, which are used to extract global semantic features and local texture features of the preprocessed image respectively, and output multi-scale dual-stream features; The feature fusion enhancement submodule is used to perform cross-scale attention fusion and adaptive feature pyramid processing on the extracted dual-stream multi-scale features to obtain fused features. By introducing an anatomical perception attention mechanism and an anatomical relation matrix, the topology and adjacency relationships of the fused features are constrained to obtain enhanced fused features. The MLP prediction decoding submodule uses an MLP decoding head to perform channel mapping and uniform resolution upsampling on the enhanced fusion features. After lightweight convolution refinement, it predicts the final segmentation mask through a multilayer perceptron and generates a segmentation probability map of the preprocessed image. The inference engine module includes: The parallel pipeline submodule is used for data transfer. It adopts a producer-consumer model to decouple the image acquisition, inference and rendering processes, and sets up acquisition threads, inference threads and rendering threads to transfer data through queues. The ONNX Runtime Inference Optimization Submodule enables a memory pool mechanism to pre-allocate a contiguous memory pool for intermediate tensor computation and enables operator fusion optimization to reduce the number of computation kernel startups.

Images