Ultrasound video breast lesion segmentation method integrating spatiotemporal information

Abstract

The present invention provides a method for segmenting breast lesions from ultrasound videos that integrates spatiotemporal information, relating to the field of image processing technology. The method extracts temporal information from ultrasound video sequences in parallel through interval sampling, and adds the extracted temporal information sequence to the original sequence, using this as input information for spatial information extraction. During the spatial information extraction process, the temporal dimension and the channel dimension are merged to streamline the data structure, further reducing computational complexity through window grouping. Furthermore, through attention learning with a moving window, more comprehensive information learning and feature extraction can be performed. This method achieves effective extraction of spatiotemporal information, has better performance, and significantly reduces computational complexity, cost, and time, enabling end-to-end segmentation of ultrasound videos.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for segmenting breast lesions using ultrasound video that integrates temporal and spatial information. Background Art

[0002] Breast cancer is one of the most dreaded diseases affecting women worldwide. Automatic segmentation of breast lesions in ultrasound videos is crucial for computer-assisted clinical examination and treatment.

[0003] Some ultrasound image segmentation methods do not handle the long-term dependencies and local information between images well, and there will still be problems such as blurred segmentation boundaries or even segmentation errors, large errors and poor generalization.

[0004] Compared to ultrasound image segmentation, ultrasound videos contain additional temporal information, and segmentation based on videos is more practical in ultrasound imaging. However, automatic segmentation of breast lesions based on ultrasound videos remains a challenging task.

[0005] Currently, methods for segmenting breast lesions based on ultrasound videos all have problems of high computational complexity, high computational cost, and long computational time.

[0006] For example, the STM network has high computational complexity and long computation time. As the length of the video increases, the computational cost also increases. In addition, the STM network does not consider local features too much, and is prone to mismatching problems between memory frames.

[0007] Other Transformer-based methods for breast lesion segmentation in ultrasound videos also incur significant computational overhead due to tokenization. For example, the DPSTT network is relatively complex and not lightweight enough.

[0008] Although the Video Swin Transformer network effectively integrates spatiotemporal information, the information redundancy of the time series is high, the computational complexity is still very high, and it has not been applied to medical image segmentation tasks. Summary of the Invention

[0009] (1) Technical problems solved

[0010] In view of the shortcomings of the existing technology, the present invention provides an ultrasound video breast lesion segmentation method that integrates spatiotemporal information, which solves the problems of high computational complexity, high computational cost and long computational time of the ultrasound video breast lesion segmentation method.

[0011] (2) Technical solution

[0012] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0013] A method for segmenting breast lesions using ultrasound video that integrates spatiotemporal information, comprising the following steps:

[0014] S1, ultrasound video sequence X T Enter the norm layer for normalization;

[0015] S2, split the normalized sequence into s subsequences by non-overlapping interval sampling in the time dimension T;

[0016] S3, each subsequence is learned by self-attention calculation of time information

[0017] S4, all the sequences Splice them together in the order before splitting to get a sequence with extracted time information

[0018] S5, the ultrasound video sequence X T and the sequence after extracting time information Add and normalize to get X;

[0019] S6. Input the sequence X into the spatial information extraction network to further extract the spatial features of the data;

[0020] S7. Restoring the breast lesion ultrasound data after the spatiotemporal information is extracted to the resolution of the original ultrasound video through a restoration module to obtain a breast lesion segmentation result.

[0021] Preferably, the ultrasound video sequence X T ∈R V×T , consists of T frames, and the feature vector length of each frame is V, where V = H × W × 3, H is the height, W is the width, and 3 is the number of channels.

[0022] Preferably, the extraction formula of the sequence X is as follows:

[0023] X T,1 ,...,X T,s =sampled(norm(X T ))

[0024]

[0025]

[0026]

[0027] Where i represents the i-th subsequence;

[0028] s represents s time subsequences;

[0029] sampled is interval sampling;

[0030] Temporal is for self-attention calculation;

[0031] merge is the original order splicing.

[0032] Preferably, the temporal specific operation is as follows:

[0033] X T,i =X T,i +P

[0034] Q i ,K i ,V i =X T,i W Q ,X T,i W K ,X T,i W V

[0035]

[0036] Where: P is the sinusoidal position code;

[0037] W Q ,W K ,W V There are three trainable parameters, input X T,i Multiply them respectively to get Q, K, and V;

[0038] Q stands for Query, K for Key, and V for Value;

[0039] T in the equation is the transpose operation, d k is the scaling factor.

[0040] Preferably, in S6, the method for identifying breast lesion segmentation data comprises the following steps:

[0041] S6.1. Fuse the time dimension T and channel dimension of the input sequence X into one dimension.

[0042] S6.2. Divide the fused sequence X into N non-overlapping pachs of equal size in the dimensions H and W, where H is the height and W is the width;

[0043] S6.3, pach is encoded through a linear embedding layer and mapped to an arbitrary dimension;

[0044] S6.4. The encoded information is passed through the STFT module → PM-STFT module*3 to extract spatial information.

[0045] Preferably, the operating steps of the STFT module are as follows:

[0046]

[0047]

[0048]

[0049]

[0050] Where z represents the output of each part, l represents the lth layer, W-MSA represents window-based multi-head self-attention, LN represents layer normalization, MLP is a two-layer perceptron with GeLU nonlinearity in the middle, and SW-MSA represents moving window-based self-attention;

[0051] The operation steps of the MSA module include: dividing the H×W feature map into H / M×W / M windows in a non-overlapping manner, each window contains M×M paches; then performing multi-head self-attention calculation in each window;

[0052] The operation steps of the SW-MSA module include: Based on the W-MSA divided window, move to the lower right corner The distance between the upper left corner and the pach that does not meet the size is spliced together, and then the multi-head self-attention calculation is performed inside the window;

[0053] Multi-head self-attention: The input is divided into M parts, each part is self-attention calculated separately, and then spliced together. The self-attention calculation of each head can be expressed as:

[0054] Attention(Q,K,V)=SoftMax(QKT / √d+B)V

[0055] in, B is the relative position deviation d is the dimension, M2 is the number of pach in the window, and the value of B is taken from the deviation matrix Λ∈R (2M-1)×(2M-1) .

[0056] Preferably, the PM-STFT module is an STFT module with Pach Merging, and its operation steps are as follows:

[0057] The PM of the first layer connects the features of each group of 2×2 adjacent pachs, and the feature dimension after connection is 4C; then a linear layer is applied, the feature dimension is changed from 4C to 2C, the resolution is downsampled to 1 / 2 of the original, that is, H / 8×W / 8, and N becomes 1 / 4 of the original; finally, the STFT operation mentioned above is applied; the operations of the second and third layers are similar, and the resolution and dimension after Pach Merging are H / 16×W / 16, 4C; H / 32×W / 32, 8C respectively.

[0058] Preferably, in S7, the operation steps of the recovery module are as follows:

[0059] S7.1. Apply a linear layer to the input features to increase the feature dimension to twice the original dimension (H / 32×W / 32×8C → H / 32×W / 32×16C).

[0060] S7.2. Use rearrange to double the feature resolution and reduce the feature dimension to 1 / 4 of the input dimension, i.e., H / 32×W / 32×16C → H / 16×W / 16×4C.

[0061] S7.3. Add the output of the second PM-STFT module using a jump connection, and then perform the operations in S7.1 and S7.2 (H / 16×W / 16×4C → H / 8×W / 8×2C).

[0062] S7.4. Add the output of the first PM-STFT module using a jump connection, and then perform the operations in S7.1 and S7.2 (H / 8×W / 8×2C → H / 4×W / 4×C).

[0063] S7.5. Use skip connections to add the output of the first STFT module, then use rearrange to change the dimension (H / 4×W / 4×C→T×H×W×(C / 16T)), and finally use linear projection of the fully connected layer to obtain the T-frame breast lesion segmentation result image output (T×H×W×2).

[0064] (3) Beneficial effects

[0065] The present invention provides a method for segmenting breast lesions from ultrasound video by integrating spatiotemporal information. Compared with the existing technology, it has the following advantages:

[0066] In this paper, the ultrasound video breast lesion segmentation method extracts temporal information from ultrasound video sequences in parallel through interval sampling. The extracted temporal information is then added to the original sequence, using this as input for spatial information extraction. During spatial information extraction, the temporal and channel dimensions are merged to streamline the data structure. Window grouping further reduces computational complexity, and attention learning with a moving window enables more comprehensive information learning and feature extraction. This method effectively extracts spatiotemporal information, achieves better performance, and significantly reduces computational complexity, cost, and time, enabling end-to-end segmentation of ultrasound videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 This is a diagram of the overall network architecture of the ultrasound video breast lesion segmentation method according to an embodiment of the present invention;

[0069] Figure 2 This is a network architecture diagram of a time dimension information extraction module according to an embodiment of the present invention;

[0070] Figure 3 2 is a network architecture diagram of the STFT module in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0072] The embodiments of the present application provide an ultrasound video breast lesion segmentation method that integrates spatiotemporal information, thereby solving the problems of high computational complexity, high computational cost, and long computational time in ultrasound video breast lesion segmentation methods.

[0073] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0074] In an embodiment of the present invention, the ultrasound video breast lesion segmentation method extracts temporal information from ultrasound video sequences in parallel through interval sampling. The extracted temporal information sequence is then added to the original sequence, using this as input for spatial information extraction. During spatial information extraction, the temporal and channel dimensions are merged to streamline the data structure. Window grouping further reduces computational complexity, and attention learning with a moving window enables more comprehensive information learning and feature extraction. This method effectively extracts spatiotemporal information, achieves better performance, and significantly reduces computational complexity, cost, and time, enabling end-to-end segmentation of ultrasound videos.

[0075] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0076] Example:

[0077] like Figures 1 to 3 As shown, the present invention provides a method for segmenting breast lesions using ultrasound video that integrates spatiotemporal information. The method comprises the following steps:

[0078] 1. Input:

[0079] The ultrasound video sequence X T Input network, X T ∈R V×T , consists of T frames, and the feature vector length of each frame is V. Where V = H × W × 3, H is the height, W is the width, and 3 is the number of channels.

[0080] 2. Time dimension information extraction (Parallel Temporal Block):

[0081] 2.1. First, the original sequence X T Enter the norm layer for normalization.

[0082] 2.2. Then split the sequence into s non-overlapping subsequences by interval sampling in the time dimension T, X T,1 ,…,X T,s , the length of each sequence is T / s, and by splitting the sequence, more time feature information can be learned.

[0083] 2.3. Each subsequence is learned by performing self-attention calculation on the temporal information Before temporal self-attention is performed on each subsequence, a positional encoding is added, and parameters are not shared during the learning process. Extracting temporal information through self-attention has lower computational complexity than other attention methods and can be run in parallel, effectively improving computational efficiency.

[0084] 2.4. Then, the s sequences after learning the time information are spliced together in the original order to obtain the sequence with the time dimension information learned.

[0085] 2.5. Add the original sequence and the sequence after learning the time information, and perform normalization again to obtain X, so as to compensate for the information loss in the process of learning the time information.

[0086] X T,1 ,…,X T,s =sampled(norm(X T ))

[0087]

[0088]

[0089]

[0090] Among them, sampled means interval sampling, temporal means self-attention calculation, merge means splicing in the original order; i represents the i-th subsequence, and s represents s time subsequences.

[0091] The specific operations of temporal are as follows:

[0092] X T,i =X T,i +P

[0093] Q i ,K i ,V i =X T,i W Q ,X T,i W K ,X T,i W V

[0094]

[0095] Where: P is the sinusoidal position code;

[0096] W Q ,W K ,W V There are three trainable parameters, input X T,i Multiply them respectively to get Q, K, and V;

[0097] Q stands for Query, K for Key, and V for Value;

[0098] T in the equation is the transpose operation, dk is the scaling factor.

[0099] 3. Spatial Information Extraction Block:

[0100] 3.1、Time series block segmentation: change the dimension of X array to X∈R H×W×3×T , then divide it into N non-overlapping pachs (P) of equal size in the dimensions of H and W. The shape of each pach in the height and width dimensions is 4×4, so the feature dimension of each pach is 4×4×3×T=48T, N=H / 4×W / 4, X=[P1,…,P N ].

[0101] 3.2. Encoding: Apply a linear embedding layer to encode the original value feature dimension 48T into an arbitrary dimension, denoted as C.

[0102] 3.3. Spatial dimension information extraction: STFT module → PM-STFT module * 3;

[0103] STFT module: Swin Transformer of Fusion Time Block operation steps are as follows:

[0104]

[0105]

[0106]

[0107]

[0108] Among them, z represents the output of each part, l represents the lth layer, W-MSA represents window-based multi-head self-attention, LN represents layer normalization, MLP is a two-layer perceptron with GeLU nonlinearity in the middle, and SW-MSA represents moving window-based self-attention.

[0109] The operation example of the MSA module is as follows: the H×W feature map is divided into H / M×W / M windows in a non-overlapping manner, each window contains M×M paches; then multi-head self-attention calculation is performed in each window.

[0110] The operation example of SW-MSA module is as follows: Based on the divided window of W-MSA, move to the lower right corner The distance between the upper left corner and the pach that does not meet the size is spliced together, and then the multi-head self-attention calculation is performed inside the window.

[0111] Multi-head self-attention: The input is divided into M parts, each part is self-attention calculated separately, and then spliced together. The self-attention calculation of each head can be expressed as:

[0112]

[0113] in, B is the relative position deviation d is the dimension, M 2 is the number of pach in the window, and the value of B is taken from the deviation matrix Λ∈R (2M-1)×(2M-1) .

[0114] PM-STFT module: STFT module with Pach Merging; the operation is as follows:

[0115] The PM layer in the first layer concatenates the features of each set of 2×2 adjacent pachs, resulting in a concatenated feature dimension of 4C. A linear layer is then applied to reduce the feature dimension from 4C to 2C, downsampling the resolution to 50%, that is, H / 8 × W / 8, and N to 40%. Finally, the STFT operation described above is applied. The second and third layers operate similarly, with the resolution and dimension after pach merging being H / 16 × W / 16, 4C, and H / 32 × W / 32, 8C, respectively.

[0116] 4. Recovery Block: Restores low-resolution and high-dimensional feature information to its original resolution and dimension.

[0117] 4.1. Apply a linear layer to the input features to increase the feature dimension to twice the original dimension (H / 32×W / 32×8C→H / 32×W / 32×16C).

[0118] 4.2. Use rearrange to expand the feature resolution to twice the original one, and reduce the feature dimension to 1 / 4 of the input dimension, that is, H / 32×W / 32×16C→H / 16×W / 16×4C.

[0119] 4.3. Use a jump connection to add the output of the second PM-STFT module, and then perform the operations in 4.1 and 4.2 (H / 16×W / 16×4C→H / 8×W / 8×2C).

[0120] 4.4. Use a jump connection to add the output of the first PM-STFT module, and then perform the operations in 4.1 and 4.2 (H / 8×W / 8×2C→H / 4×W / 4×C).

[0121] 5. Output: Use skip connections to add the output of the first STFT module, then use rearrange to change the dimension (H / 4×W / 4×C → T×H×W×(C / 16T)), and finally use linear projection of the fully connected layer to obtain the T-frame breast lesion segmentation result image output (T×H×W×2).

[0122] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0123] In an embodiment of the present invention, the ultrasound video breast lesion segmentation method extracts temporal information from ultrasound video sequences in parallel through interval sampling. The extracted temporal information sequence is then added to the original sequence, using this as input for spatial information extraction. During spatial information extraction, the temporal and channel dimensions are merged to streamline the data structure. Window grouping further reduces computational complexity, and attention learning with a moving window enables more comprehensive information learning and feature extraction. This method effectively extracts spatiotemporal information, achieves better performance, and significantly reduces computational complexity, cost, and time, enabling end-to-end segmentation of ultrasound videos.

[0124] 2. In the embodiment of the present invention, during the time information extraction process, the time series is sampled at intervals and self-attention calculations are performed separately, which can achieve parallelization, fully extract the time series features, effectively reduce the computational complexity, and improve the training efficiency.

[0125] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0126] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for segmenting breast lesions from ultrasound video by integrating spatiotemporal information, characterized in that: The ultrasound video breast lesion segmentation method comprises the following steps: S1, ultrasound video sequence X T Enter the norm layer for normalization; S2, split the normalized sequence into s subsequences by non-overlapping interval sampling in the time dimension T; S3, each subsequence is learned by self-attention calculation of time information S4, all the sequences Splice them together in the order before splitting to get a sequence with extracted time information S5, the ultrasound video sequence X T and the sequence after extracting time information Add and normalize to get X; S6. Input the sequence X into the spatial information extraction network to further extract the spatial features of the data; S7. Restoring the breast lesion ultrasound data after the spatiotemporal information is extracted to the resolution of the original ultrasound video through a restoration module to obtain a breast lesion segmentation result.

2. The ultrasound video breast lesion segmentation method integrating spatiotemporal information according to claim 1, characterized in that: The ultrasound video sequence X T ∈R V×T , consists of T frames, and the feature vector length of each frame is V, where V = H × W × 3, H is the height, W is the width, and 3 is the number of channels.

3. The ultrasound video breast lesion segmentation method integrating spatiotemporal information according to claim 1, characterized in that: The extraction formula of the sequence X is as follows: X T, ,…,X T, =sampled(norm(X T )) Where i represents the i-th subsequence; s represents s time subsequences; sampled is interval sampling; Temporal is for self-attention calculation; merge is the original order splicing.

4. The ultrasound video breast lesion segmentation method integrating spatiotemporal information according to claim 3, wherein: The specific operation of the temporal is as follows: X T, =X T,i +P Q i ,K i ,V i =X T,i W Q ,X T, W K ,X T, W V Where: P is the sinusoidal position code; W Q ,W K ,W V There are three trainable parameters, input X T, Multiply them respectively to get Q, K, and V; Q stands for Query, K for Key, and V for Value; T in the equation is the transpose operation, d k is the scaling factor.

5. The ultrasound video breast lesion segmentation method integrating spatiotemporal information according to claim 1, wherein: In S6, the method for identifying breast lesion segmentation data includes the following steps: S6.

1. Fuse the time dimension T and channel dimension of the input sequence X into one dimension. S6.

2. Divide the fused sequence X into N non-overlapping pachs of equal size in the dimensions H and W, where H is the height and W is the width; S6.3, pach is encoded through a linear embedding layer and mapped to an arbitrary dimension; S6.

4. The encoded information is passed through the STFT module → PM-STFT module*3 to extract spatial information.

6. The ultrasound video breast lesion segmentation method integrating spatiotemporal information according to claim 5, characterized in that: The operating steps of the STFT module are as follows: Where z represents the output of each part, l represents the lth layer, W-MSA represents window-based multi-head self-attention, LN represents layer normalization, MLP is a two-layer perceptron with GeLU nonlinearity in the middle, and SW-MSA represents moving window-based self-attention; The operation steps of the MSA module include: dividing the H×W feature map into H / M×W / M windows in a non-overlapping manner, each window contains M×M paches; then performing multi-head self-attention calculation in each window; The operation steps of the SW-MSA module include: Based on the W-MSA divided window, move to the lower right corner The distance between the upper left corner and the pach that does not meet the size is spliced together, and then the multi-head self-attention calculation is performed inside the window; Multi-head self-attention: The input is divided into M parts, each part is self-attention calculated separately, and then spliced together. The self-attention calculation of each head can be expressed as: Attention(Q,K,V)=SoftMax(QKT / √d+B)V Among them, Q, K, B is the relative position deviation d is the dimension, M2 is the number of pach in the window, and the value of B is taken from the deviation matrix Λ∈R (2M-1)×(2M-1) .

7. The ultrasound video breast lesion segmentation method integrating spatiotemporal information according to claim 5, characterized in that: The PM-STFT module is an STFT module with Pach Merging. Its operation steps are as follows: The PM of the first layer connects the features of each group of 2×2 adjacent pachs, and the feature dimension after connection is 4C; then a linear layer is applied, the feature dimension is changed from 4C to 2C, the resolution is downsampled to 1 / 2 of the original, that is, H / 8×W / 8, and N becomes 1 / 4 of the original; finally, the STFT operation mentioned above is applied; the operations of the second and third layers are similar, and the resolution and dimension after Pach Merging are H / 16×W / 16, 4C; H / 32×W / 32, 8C respectively.

8. The ultrasound video breast lesion segmentation method integrating spatiotemporal information according to claim 1, wherein: In S7, the operation steps of the recovery module are as follows: S7.

1. Apply a linear layer to the input features to increase the feature dimension to twice the original dimension (H / 32×W / 32×8C → H / 32×W / 32×16C). S7.

2. Use rearrange to double the feature resolution and reduce the feature dimension to 1 / 4 of the input dimension, i.e., H / 32×W / 32×16C → H / 16×W / 16×4C. S7.

3. Add the output of the second PM-STFT module using a jump connection, and then perform the operations in S7.1 and S7.2 (H / 16×W / 16×4C → H / 8×W / 8×2C). S7.

4. Add the output of the first PM-STFT module using a jump connection, and then perform the operations in S7.1 and S7.2 (H / 8×W / 8×2C → H / 4×W / 4×C). S7.

5. Use skip connections to add the output of the first STFT module, then use rearrange to change the dimension (H / 4×W / 4×C→T×H×W×(C / 16T)), and finally use linear projection of the fully connected layer to obtain the T-frame breast lesion segmentation result image output (T×H×W×2).

Images