A ventricular septal defect positioning system and method based on TSDNet

Abstract

The present invention discloses a ventricular septal defect localization system and method based on TSDNet, the system includes: an acquisition module, a construction module and a positioning module; the acquisition module is used to acquire ultrasound electrocardiogram videos under multiple standard views, and use the ultrasound cardiogram videos with labels as sample data sets; the construction module is used to construct a ventricular septal defect localization model based on the data set, and the ventricular septal defect localization model adopts the TSDNet network architecture, including a UniFormer model for judging the positive and negative of the ultrasound cardiogram video and an optimized YOLOv5 model for detecting the lesion area of ​​the ventricular septal defect; the positioning module is used to use the ventricular septal defect localization model to complete the ventricular septal defect localization. The present invention proposes a spatiotemporal decoupling network, which realizes the video-level classification of ventricular septal defects and the precise positioning of shunt areas by separating the temporal and spatial features in the ultrasound cardiogram video for efficient analysis.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a ventricular septal defect positioning system and method based on TSDNet. Background Art

[0002] Transthoracic echocardiography (TTE) can provide sufficient information about cardiac structure and evaluate hemodynamics and cardiac function, and is an effective method for VSD examination. In color Doppler echocardiography videos, the shunt signal at the level of the left and right ventricles displayed by color Doppler blood flow imaging can clearly and accurately diagnose VSD. The advantages of echocardiography include simplicity, non-invasiveness, inexpensiveness, repeatability, and accuracy.

[0003] Over the past few years, artificial intelligence (AI)-based image recognition techniques, especially deep learning, have been increasingly applied to echocardiography. Conventional machine learning usually requires the inference of predefined features in the input image. In contrast, deep learning automatically estimates results from the image without the need to predefine specific imaging features. In addition, the deep structure of convolutional neural networks is able to extract detailed low-level information from the original image and combine this information into high-order structural information, thereby enabling the recognition of complex entities from the image. Recently, deep learning has achieved outstanding performance in the field of video understanding, and its value has been demonstrated in both two-dimensional and three-dimensional networks. Summary of the invention

[0004] The VSD detection task in cardiac ultrasound videos is significantly different from standard video understanding tasks. While general video analysis usually focuses on extracting semantic information from extended sequences, VSD detection requires identifying specific instantaneous events within the cardiac cycle. In VSD ultrasound videos, this requires a temporal analysis method that can effectively isolate and examine these short critical periods.

[0005] To achieve the above object, the present invention provides a ventricular septal defect positioning system based on TSDNet, comprising: an acquisition module, a construction module and a positioning module;

[0006] The acquisition module is used to acquire ultrasound electrocardiogram videos under multiple standard views, and use the ultrasound electrocardiogram videos with labels as sample data sets;

[0007] The construction module is used to construct a ventricular septal defect localization model based on the data set, and the ventricular septal defect localization model adopts a TSDNet network architecture, including a UniFormer model for judging the positive and negative nature of an echocardiogram video and an optimized YOLOv5 model for detecting a ventricular septal defect lesion area;

[0008] The positioning module is used to complete the positioning of the ventricular septal defect using the ventricular septal defect positioning model.

[0009] Preferably, the first step of the constructed ventricular septal defect localization model video classification adopts the UniFormer architecture to process an 8-frame input window for VSD binary classification; the second step uses the optimized YOLOv5 model to perform VSD localization in a single frame; for each frame in the input video, the time branch processes an 8-frame window centered on the current frame, while the spatial branch is used to process the current frame itself.

[0010] Preferably, the ventricular septal defect localization model training is performed using a decoupling strategy, the temporal branch is optimized using binary labeled video sequences, and the spatial branch is trained through frame-level bounding box annotations.

[0011] Preferably, the positioning module uses the ventricular septal defect positioning model to adopt a three-threshold judgment mechanism to perform ventricular septal defect positioning:

[0012] When J smooth >θ high When the VSD is detected,

[0013] When J smooth <θ low When the VSD is not present,

[0014] When θ low ≤J smooth ≤θ high When , keep the judgment result of the previous frame;

[0015] Among them, J smooth represents the smoothed decision output of the VSD localization model; θ high =0.75; θ low =0.45.

[0016] The present invention also provides a ventricular septal defect positioning method based on TSDNet, which is applied to the above system and comprises the following steps:

[0017] Collecting echocardiogram videos under multiple standard views and using the labeled echocardiogram videos as sample datasets;

[0018] Used to construct a ventricular septal defect localization model based on the data set, wherein the ventricular septal defect localization model adopts a TSDNet network architecture, including a UniFormer model for judging the positivity of an echocardiogram video and an optimized YOLOv5 model for detecting a ventricular septal defect lesion area;

[0019] The ventricular septal defect positioning model is used to complete the ventricular septal defect positioning.

[0020] Preferably, the first step of the constructed ventricular septal defect localization model video classification adopts the UniFormer architecture to process an 8-frame input window for VSD binary classification; the second step uses the optimized YOLOv5 model to perform VSD localization in a single frame; for each frame in the input video, the time branch processes an 8-frame window centered on the current frame, while the spatial branch is used to process the current frame itself.

[0021] Preferably, the ventricular septal defect localization model training is performed using a decoupling strategy, the temporal branch is optimized using binary labeled video sequences, and the spatial branch is trained through frame-level bounding box annotations.

[0022] Preferably, the method for locating a ventricular septal defect comprises: using the ventricular septal defect locating model to locate the ventricular septal defect using a three-threshold decision mechanism:

[0023] When J smooth >θ high When the VSD is detected,

[0024] When J smooth <θ low When the VSD is not present,

[0025] When θ low ≤J smooth ≤θ high When , keep the judgment result of the previous frame;

[0026] Among them, J smooth represents the smoothed decision output of the VSD localization model; θ high =0.75; θ low =0.45.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] This paper proposes a spatiotemporal decoupling network (TSDNet), which achieves video-level classification of ventricular septal defects and precise localization of shunt areas by separating the temporal and spatial features in echocardiographic videos for efficient analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 Schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Ventricular septal defect is a common congenital heart disease. The abnormal blood flow signal will change significantly with the cardiac cycle. Due to the continuous beating of the heart, the morphology and position of the VSD area are also changing dynamically. In certain phases of the cardiac cycle, a single frame image may not be able to capture the VSD characteristics at all. Doctors need to carefully observe the complete cardiac cycle to make an accurate judgment.

[0034] Embodiment 1

[0035] like Figure 1 As shown, it is a schematic diagram of the system structure of the present invention, including: an acquisition module, a construction module and a positioning module; the acquisition module is used to acquire ultrasound electrocardiogram videos under multiple standard views, and use the ultrasound cardiogram videos with labels as sample data sets; the construction module is used to build a ventricular septal defect positioning model based on the data set, and the ventricular septal defect positioning model adopts the TSDNet network architecture, including a UniFormer model for judging the positive and negative nature of the ultrasound cardiogram video and an optimized YOLOv5 model for detecting the ventricular septal defect lesion area; the positioning module is used to use the ventricular septal defect positioning model to complete the ventricular septal defect positioning.

[0036] The following will explain in detail how the present invention solves technical problems in real life in conjunction with this embodiment.

[0037] First, the acquisition module is used to collect ultrasound electrocardiogram videos under multiple standard views, and the ultrasound electrocardiogram videos with labels (positive video labels are 1, and negative video labels are 0) are used as sample data sets.

[0038] Then, a ventricular septal defect positioning model is constructed using the construction module. The process includes: initializing a deep neural network model for ventricular septal defect shunt area detection, and pre-training the deep neural network model using a sample data set to obtain a pre-trained deep neural network model.

[0039] Specifically, the constructed model is used to determine the location and orientation of the defect in confirmed patients. The model adopts the TSDNet network architecture, which includes a UniFormer model for determining the positivity of echocardiographic videos and an optimized YOLOv5 model for detecting the lesion area of ​​ventricular septal defect. The first step of video classification uses the UniFormer architecture, which is adapted to process 8-frame input windows for VSD binary classification. The second step of lesion detection uses the optimized YOLOv5 model to locate the VSD in a single frame. For each frame in the input video, the temporal branch processes an 8-frame window centered on the current frame, while the spatial branch focuses on the frame itself.

[0040] Training is performed using a decoupled strategy, with the temporal branch optimized using binary annotated video sequences, while the spatial branch is trained with frame-level bounding box annotations. This approach enables each branch to focus on its respective task. By simultaneously leveraging dynamic temporal features between video frames and spatial localization information within a single frame, TSDNet aims to achieve VSD detection in pediatric echocardiography. During the inference phase, the temporal branch analyzes a 32-frame window around the current frame to determine the presence of a VSD, while the current frame is fed into the spatial branch for localization; the results of the two branches are finally integrated.

[0041] Finally, the positioning module uses the ventricular septal defect positioning model to complete the ventricular septal defect positioning.

[0042] Given an input ultrasound video sequence , where T is the temporal dimension, H and W are the spatial resolutions, and 3 is the number of RGB channels. Represents the real number domain. The video sequence is sampled using a fixed-step sliding window strategy, where each window contains a continuous sequence of 9 frames: t ={I t ,I t+1 ,...,I t+8}.

[0043] In order to effectively extract hemodynamic features, the present invention designs a feature separation module based on channel decoupling. , the decoupling function D(I) is used to decouple the structural information and blood flow information:

[0044] D(I)={S(I),F(I)},

[0045] Among them, S(I) represents structural information; F(I) represents blood flow information.

[0046] Structural information is obtained through RGB weighted transformation:

[0047] ,

[0048] Among them, R, G, and B represent the three colors of red, green, and blue.

[0049] Blood flow information is obtained through residual mapping:

[0050] .

[0051] In the temporal classification branch, an improved UniFormer backbone network is used for VSD classification. A temporal attention enhancement module based on decoupled features is designed. At the input stage of UniFormer, the original sequence is received. And the decoupled grayscale sequence . Through the convolutional neural network, X g Encode to get F g :

[0052] ,

[0053] Among them, CNN represents convolutional neural network; T represents the time series length; H and W represent the height and width of the feature map respectively; C represents the number of feature channels.

[0054] F g The temporal feature F is obtained by global average pooling gap :

[0055] ,

[0056] Among them, GAP stands for global average pooling. The GAP operation averages the feature map of the spatial dimension into a scalar value, effectively reducing the feature dimension and retaining the channel information.

[0057] Design a temporal attention mechanism to model temporal dependencies:

[0058] Q=W q ·F gap ,

[0059] K=W k ·F gap ,

[0060] V=W v ·F gap ,

[0061] Among them, Q represents query; K represents key; V represents value; W represents q , W k , W v represents the learnable weight matrix.

[0062] The attention weight matrix A is calculated by the softmax function:

[0063] ,

[0064] Perform temporal pooling on the attention feature A to obtain the feature enhancement weight V t :

[0065] ,

[0066] Used to adaptively enhance the original feature X to obtain X enhanced :

[0067] ,

[0068] in, Represents element-wise multiplication.

[0069] Enhanced feature X enhanced Input the UniFormer backbone network and finally output the VSD classification result.

[0070] The classification branch uses a mixed loss function L temporal :

[0071] ,

[0072] Among them, L ce represents the cross entropy loss; L focal represents the focal loss (λ1=0.5, focal loss parameter γ=2.0, α=0.25).

[0073] In the spatial positioning branch, an improved YOLOv5 architecture is used to locate the VSD area. A feature reconstruction enhancement module is designed to receive a single frame image. and decoupling features The decoupled features are transformed nonlinearly through cascaded convolutional layers:

[0074] ,

[0075] Conv1 is the feature extraction layer (convolution kernel size: 3×3, number of output channels: 16), Conv2 is the feature reconstruction layer (convolution kernel size: 1×1, number of output channels: 3); F d Represents decoupled features.

[0076] Perform feature fusion:

[0077] ,

[0078] Among them, F fused Represents the fused features; represents the Hadamard product.

[0079] The fused features are input into the YOLOv5 network for target detection:

[0080] ,

[0081] Among them, B t Represents the positioning result; Det represents the detection function of YOLOv5.

[0082] In this embodiment, the output vector includes the bounding box coordinates and the confidence score.

[0083] The spatial branch uses a composite loss function:

[0084] ,

[0085] Among them, L ciou represents the completeness intersection-combination ratio loss; L single =log(1+∑exp(si)) is the single objective constraint loss (λ2=1.0, λ3=0.1).

[0086] A dual-branch joint reasoning strategy is designed, and a sliding window mechanism is used to process the video sequence. For each time step t, the classification confidence P is obtained through UniFormer. t , the center frame is detected by YOLOv5 to obtain the positioning result B t The joint decision score J at time step t is t The calculation is as follows:

[0087] ,

[0088] Where α=0.6; conf represents the confidence score extracted from the detection results.

[0089] Design a timing smoothing mechanism and maintain a decision queue:

[0090] ,

[0091] The smooth decision J is obtained by median filtering smooth :

[0092] .

[0093] The final design of the three-threshold decision mechanism is:

[0094] When J smooth >θ high When the VSD is detected,

[0095] When J smooth <θ low When the VSD is not present,

[0096] When θ low ≤Jsmooth ≤θ high When , keep the judgment result of the previous frame (where θ high =0.75,θ low =0.45).

[0097] This embodiment achieves accurate detection and positioning of VSD through feature decoupling, multi-branch deep network and spatiotemporal joint reasoning.

[0098] Embodiment 2

[0099] This embodiment also provides a ventricular septal defect positioning method based on TSDNet, the steps comprising:

[0100] S1. Collect echocardiogram videos under multiple standard views and use the labeled echocardiogram videos as sample datasets.

[0101] S2. Used to construct a ventricular septal defect positioning model based on the data set.

[0102] The ventricular septal defect localization model adopts the TSDNet network architecture, and the method includes: initializing a deep neural network model for ventricular septal defect shunt area detection, and using a sample data set to pre-train the deep neural network model to obtain a pre-trained deep neural network model.

[0103] Specifically, the constructed model is used to determine the location and orientation of the defect in confirmed patients. The model adopts the TSDNet network architecture, which includes a UniFormer model for determining the positivity of echocardiographic videos and an optimized YOLOv5 model for detecting the lesion area of ​​ventricular septal defect. The first step of video classification uses the UniFormer architecture, which is adapted to process 8-frame input windows for VSD binary classification. The second step of lesion detection uses the optimized YOLOv5 model to locate the VSD in a single frame. For each frame in the input video, the temporal branch processes an 8-frame window centered on the current frame, while the spatial branch focuses on the frame itself.

[0104] Training is performed using a decoupled strategy, with the temporal branch optimized using binary annotated video sequences, while the spatial branch is trained with frame-level bounding box annotations. This approach enables each branch to focus on its respective task. By simultaneously leveraging dynamic temporal features between video frames and spatial localization information within a single frame, TSDNet aims to achieve VSD detection in pediatric echocardiography. During the inference phase, the temporal branch analyzes a 32-frame window around the current frame to determine the presence of a VSD, while the current frame is fed into the spatial branch for localization; the results of the two branches are finally integrated.

[0105] S3. Use the ventricular septal defect localization model to complete the ventricular septal defect localization.

[0106] Given an input ultrasound video sequence , where T is the temporal dimension, H and W are the spatial resolutions, and 3 is the number of RGB channels. Represents the real number domain. The video sequence is sampled using a fixed-step sliding window strategy, where each window contains a continuous sequence of 9 frames: t ={I t ,I t+1 ,...,I t+8}.

[0107] In order to effectively extract hemodynamic features, the present invention designs a feature separation module based on channel decoupling. , the decoupling function D(I) is used to decouple the structural information and blood flow information:

[0108] D(I)={S(I),F(I)},

[0109] Among them, S(I) represents structural information; F(I) represents blood flow information.

[0110] Structural information is obtained through RGB weighted transformation:

[0111] ,

[0112] Among them, R, G, and B represent the three colors of red, green, and blue.

[0113] Blood flow information is obtained through residual mapping:

[0114] .

[0115] In the temporal classification branch, an improved UniFormer backbone network is used for VSD classification. A temporal attention enhancement module based on decoupled features is designed. At the input stage of UniFormer, the original sequence is received. And the decoupled grayscale sequence . Through the convolutional neural network, X g Encode to get F g :

[0116] ,

[0117] Among them, CNN represents convolutional neural network; T represents the time series length; H and W represent the height and width of the feature map respectively; C represents the number of feature channels.

[0118] F g The temporal feature F is obtained by global average pooling gap :

[0119] ,

[0120] Among them, GAP stands for global average pooling. The GAP operation averages the feature map of the spatial dimension into a scalar value, effectively reducing the feature dimension and retaining the channel information.

[0121] Design a temporal attention mechanism to model temporal dependencies:

[0122] Q=W q ·F gap ,

[0123] K=W k ·F gap ,

[0124] V=W v ·F gap ,

[0125] Among them, Q represents query; K represents key; V represents value; W represents q , W k , W v represents the learnable weight matrix.

[0126] The attention weight matrix A is calculated by the softmax function:

[0127] ,

[0128] Perform temporal pooling on the attention feature A to obtain the feature enhancement weight V t :

[0129] ,

[0130] Used to adaptively enhance the original feature X to obtain X enhanced :

[0131] .

[0132] in, Represents element-wise multiplication.

[0133] Enhanced feature X enhanced Input the UniFormer backbone network and finally output the VSD classification result.

[0134] The classification branch uses a mixed loss function L temporal :

[0135] ,

[0136] Among them, L ce represents the cross entropy loss; L focalrepresents the focal loss (λ1=0.5, focal loss parameter γ=2.0, α=0.25).

[0137] In the spatial positioning branch, an improved YOLOv5 architecture is used to locate the VSD area. A feature reconstruction enhancement module is designed to receive a single frame image. and decoupling features The decoupled features are transformed nonlinearly through cascaded convolutional layers:

[0138] ,

[0139] Conv1 is the feature extraction layer (convolution kernel size: 3×3, number of output channels: 16), Conv2 is the feature reconstruction layer (convolution kernel size: 1×1, number of output channels: 3); F d Represents decoupled features.

[0140] Perform feature fusion:

[0141] ,

[0142] Among them, F fused Represents the fused features; represents the Hadamard product.

[0143] The fused features are input into the YOLOv5 network for target detection:

[0144] ,

[0145] Among them, B t Represents the positioning result; Det represents the detection function of YOLOv5.

[0146] In this embodiment, the output vector includes the bounding box coordinates and the confidence score.

[0147] The spatial branch uses a composite loss function:

[0148] ,

[0149] Among them, L ciou represents the completeness intersection-combination ratio loss; L single =log(1+∑exp(si)) is the single objective constraint loss (λ2=1.0, λ3=0.1).

[0150] A dual-branch joint reasoning strategy is designed, and a sliding window mechanism is used to process the video sequence. For each time step t, the classification confidence P is obtained through UniFormer. t , the center frame is detected by YOLOv5 to obtain the positioning result B tThe joint decision score J at time step t is t The calculation is as follows:

[0151] ,

[0152] Where α=0.6; conf represents the confidence score extracted from the detection results.

[0153] The smooth decision J is obtained by median filtering smooth :

[0154] .

[0155] The final design of the three-threshold decision mechanism is:

[0156] When J smooth >θ high When the VSD is detected,

[0157] When J smooth <θ low When the VSD is not present,

[0158] When θ low ≤J smooth ≤θ high When , keep the judgment result of the previous frame (where θ high =0.75,θ low =0.45).

[0159] This embodiment achieves accurate detection and positioning of VSD through feature decoupling, multi-branch deep network and spatiotemporal joint reasoning.

[0160] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A ventricular septal defect localization system based on TSDNet, characterized in that: include: Acquisition module, construction module and positioning module; The acquisition module is used to acquire ultrasound electrocardiogram videos under multiple standard views, and use the ultrasound electrocardiogram videos with labels as sample data sets; The construction module is used to construct a ventricular septal defect localization model based on the data set. The ventricular septal defect localization model adopts the TSDNet network architecture, including a UniFormer model for judging the yin and yang of the echocardiogram video and an optimized YOLOv5 model for detecting the ventricular septal defect lesion area; the first step of video classification adopts the UniFormer architecture to process an 8-frame input window for VSD binary classification; the second step uses the optimized YOLOv5 model to locate the VSD in a single frame; wherein, the time branch adopts an improved UniFormer backbone network for VSD classification, and designs a temporal attention enhancement module based on decoupled features, which is used to receive the original sequence at the input stage of UniFormer; the spatial branch adopts an improved YOLOv5 architecture for VSD area localization, and designs a feature reconstruction enhancement module to receive a single frame image and decoupled features; for each frame in the input video, the time branch processes an 8-frame window centered on the current frame, while the spatial branch is used to process the current frame itself; The positioning module is used to complete the positioning of the ventricular septal defect using the ventricular septal defect positioning model.

2. The ventricular septal defect positioning system based on TSDNet according to claim 1, characterized in that: The VSD localization model training is performed using a decoupling strategy, where the temporal branch is optimized using binary labeled video sequences, while the spatial branch is trained using frame-level bounding box annotations.

3. The ventricular septal defect positioning system based on TSDNet according to claim 1, characterized in that: The positioning module uses the ventricular septal defect positioning model to adopt a three-threshold judgment mechanism to perform ventricular septal defect positioning: when J smooth >θ high When the VSD is detected, when J smooth <θ low When , it is determined that there is no VSD; when θ low ≤J smooth ≤θ high When , keep the judgment result of the previous frame; in, J smooth A smoothing decision representing the output of the ventricular septal defect localization model; θ high =0 . 75; θ low =0 .

45.

4. A ventricular septal defect positioning method based on TSDNet, the method is applied to the system according to any one of claims 1 to 3, characterized in that the steps include: Collecting echocardiogram videos under multiple standard views and using the labeled echocardiogram videos as sample datasets; Used to construct a ventricular septal defect localization model based on the data set, the ventricular septal defect localization model adopts the TSDNet network architecture, including a UniFormer model for judging the yin and yang of the echocardiogram video and an optimized YOLOv5 model for detecting the ventricular septal defect lesion area; the first step of video classification adopts the UniFormer architecture to process an 8-frame input window for VSD binary classification; the second step uses the optimized YOLOv5 model to locate the VSD in a single frame; wherein, the time branch adopts an improved UniFormer backbone network for VSD classification, and a temporal attention enhancement module based on decoupled features is designed to receive the original sequence at the input stage of UniFormer; the spatial branch adopts an improved YOLOv5 architecture for VSD area localization, and a feature reconstruction enhancement module is designed to receive a single frame image and decoupled features; for each frame in the input video, the time branch processes an 8-frame window centered on the current frame, while the spatial branch is used to process the current frame itself; The ventricular septal defect positioning model is used to complete the ventricular septal defect positioning.

5. The ventricular septal defect positioning method based on TSDNet according to claim 4, characterized in that: The VSD localization model training is performed using a decoupling strategy, where the temporal branch is optimized using binary labeled video sequences, while the spatial branch is trained using frame-level bounding box annotations.

6. The ventricular septal defect positioning method based on TSDNet according to claim 4, characterized in that: The method for locating a ventricular septal defect includes: using the ventricular septal defect locating model to locate the ventricular septal defect using a three-threshold judgment mechanism: when J smooth >θ high When the VSD is detected, when J smooth <θ low When , it is determined that there is no VSD; when θ low ≤J smooth ≤θ high When , keep the judgment result of the previous frame; in, J smooth A smoothing decision representing the output of the ventricular septal defect localization model; θ high =0 . 75; θ low =0 . 45.

Images