Thyroid ultrasound image nodule segmentation method based on spatial-temporal characteristics and frequency enhancement

Abstract

The invention discloses a thyroid ultrasound image nodule segmentation method based on spatio-temporal characteristics and frequency enhancement, and relates to the technical field of medical image processing, and the method comprises the following specific steps: multi-scale characteristic extraction: carrying out the gray normalization and median filtering preprocessing of a thyroid ultrasound image, and carrying out the segmentation of the thyroid ultrasound image nodule; extracting multi-scale features under different resolutions by using a pre-trained backbone network, obtaining the multi-scale features under different resolutions by the backbone network through multilayer convolution and pooling operation, and generating basic and advanced features through convolution operation; by designing an intra-frame feature extraction architecture based on frequency feature fusion and an efficient spatial-temporal feature aggregation mechanism, intra-frame and inter-frame collaborative optimization is realized, intra-frame and inter-frame high and low frequency features are solved by means of octave convolution, the perceptual ability of a model to ultrasonic image details and semantic information is improved, inter-frame and inter-frame feature fusion is realized, and the perceptual ability of the model to ultrasonic image details and semantic information is improved. A local circulation neighborhood propagation mechanism is adopted, and interference of thyroid movement on time sequence feature extraction is effectively restrained.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a thyroid ultrasound image nodule segmentation method based on spatiotemporal features and frequency enhancement. Background Art

[0002] In the field of medical image processing and computer-aided diagnosis technology, accurate segmentation of thyroid nodules is crucial for clinical diagnosis. Thyroid ultrasound images, as a commonly used examination method, can provide doctors with intuitive information about the internal structure of the thyroid gland. However, due to the poor quality of ultrasound images themselves, such as noise interference and low contrast, it is very challenging to accurately segment the nodule area from ultrasound images. In addition, with the development of medical imaging technology, ultrasound images in the form of video sequences are gradually increasing. How to make full use of the rich information in video sequences to achieve more efficient and accurate nodule segmentation has become a hot topic and difficulty in current research.

[0003] Currently, most studies on thyroid ultrasound nodule segmentation still focus on the segmentation of single-frame ultrasound images. This processing method has obvious limitations and ignores the rich temporal information in video sequences. In actual clinical examinations, ultrasound video sequences can continuously record the state changes of the thyroid gland at different time points, which contains a large amount of important information that is helpful for nodule segmentation. Traditional methods do not make sufficient use of temporal information, resulting in segmentation results that may not be accurate and stable. In addition, the quality of the ultrasound image itself is poor, with problems such as speckle noise and low contrast. The full-frame connection method used in traditional temporal feature extraction is not only computationally intensive, increasing the burden on hardware equipment and computing time cost, but also difficult to accurately extract effective temporal features under the interference of thyroid movement (such as tiny thyroid movements caused by patients breathing, swallowing, etc.). It is impossible to achieve high-precision nodule segmentation while ensuring computational efficiency, and it is difficult to meet the dual requirements of clinical diagnosis for segmentation accuracy and efficiency. Summary of the Invention

[0004] The present invention is to compensate for The above tradition In order to solve the shortcomings of the technology, a thyroid ultrasound image nodule segmentation method based on spatiotemporal features and frequency enhancement is provided, which can be constructed by Pre-trained backbone network extracts multiple scales at different resolutions feature , integrating frequency features and spatiotemporal features to achieve high-precision, low-computation ultrasound video sequence segmentation. This method designs an intra-frame feature extraction architecture based on frequency feature fusion and a local cyclic neighborhood propagation mechanism, which effectively improves the model's perception of ultrasound image details and semantic information, suppresses the interference of thyroid movement on temporal feature extraction, and greatly improves the accuracy and efficiency of ultrasound video nodule segmentation.

[0005] The present invention is to solve The above traditional technologiesThe following technical solution is provided: a thyroid ultrasound image nodule segmentation method based on spatiotemporal features and frequency enhancement, the method comprising the following specific steps:

[0006] Multi-scale feature extraction: After grayscale normalization and median filtering preprocessing of thyroid ultrasound images, a pre-trained backbone network is used to extract multi-scale features at different resolutions, and convolution operations are used to generate basic and advanced features.

[0007] Frequency feature enhancement processing: After cross-scale fusion of multi-scale features and CBAM enhancement, the octave convolution is used to decompose them into high-frequency and low-frequency components. After processing them separately, they are re-fused to generate frequency-enhanced features and passed to the UNet decoder for preliminary segmentation;

[0008] Spatiotemporal feature aggregation: The input features are divided by scale, key-value pairs are constructed through temporal roll operations, attention is calculated within the pixel neighborhood, information is captured using relative position bias, and temporal information is aggregated through multi-scale temporal branch fusion and normalization.

[0009] Post-segmentation processing: Perform morphological operations of dilation and erosion on the segmentation results to remove noise areas, fill holes, optimize segmentation boundaries, and obtain the thyroid ultrasound image nodule segmentation results.

[0010] Furthermore, in the multi-scale feature extraction step, a pre-trained backbone network is used to extract features from the pre-processed image. The backbone network obtains multi-scale features at different resolutions through multi-layer convolution and pooling operations. During the convolution process, a convolution kernel of a specific size and an appropriate step size are used to extract image features layer by layer. The calculation formula is: Among them, F l Multi-scale features of different levels l extracted by the backbone network, I denoise is the image after median filtering, and its calculation formula is: I denoise (i,j)=median{I norm (m,n),(m,n)∈W(i,j)}, where I norrm (m,n) is the pixel value of the normalized image at coordinate (m,n), W(i,j) is the filter window centered at coordinate (i,j), median represents the median operation, Conv l is the l-th layer convolution operation, F deepest It is the deepest feature output by the backbone network, Conv h It is the convolution operation for further processing of the deepest features, F h It is a generated high-level feature. For the deepest features output by the backbone network, an additional convolution layer is introduced for processing. High-level features are generated through convolution operations to further enhance the feature representation capability.

[0011] Furthermore, in the frequency feature enhancement processing step, the multi-scale features are fused across scales. The number of channels of each scale feature is first adjusted by 1×1 convolution, and the adjusted scale features are spliced ​​and fused to obtain the fused features. The fused features are input into CBAM, and CBAM performs attention calculations on the features from the channel and spatial dimensions respectively. In the channel attention calculation, the global information of the feature map in the channel dimension is obtained by global average pooling and global maximum pooling, and then the channel attention weight is calculated through the multi-layer perceptron and activation function. In the spatial attention calculation, the feature map is average pooled and maximum pooled in the channel dimension, and the results are spliced ​​and the spatial attention weight is obtained through the convolution layer and activation function. The attention weight and spatial attention weight are multiplied with the original features to enhance the feature expression and generate mixed features. The mixed features are decomposed into high-frequency and low-frequency components using octave convolution. The octave convolution divides the feature map into high-frequency and low-frequency parts. The high-frequency part retains the original resolution and is used to capture the detailed texture information in the image. The low-frequency part reduces the resolution by downsampling to capture semantic and contextual information. The high-frequency and low-frequency components are processed by 3×3 convolution respectively to further extract features. The processed high-frequency and low-frequency components are re-spliced ​​and fused to generate frequency-enhanced features. The frequency-enhanced features are passed to the UNet decoder for preliminary segmentation. The UNet decoder gradually restores the image resolution through upsampling and convolution operations, and performs classification prediction for each pixel.

[0012] Furthermore, in the frequency feature enhancement processing step, CBAM performs attention calculation on the features from the channel and spatial dimensions respectively. In the channel attention calculation, the global information of the feature map in the channel dimension is obtained through global average pooling and global maximum pooling, and then the channel attention weight is calculated through the multi-layer perceptron and activation function. The formula is: Among them, M c is the channel attention map, H and W are the fused features F merge The height and width of the generated feature map, σ is the activation function, F merge It is the feature after cross-scale fusion. In the spatial attention calculation, the feature map is average pooled and max pooled in the channel dimension. The results are concatenated and passed through the convolution layer and activation function to obtain the spatial attention weight. The formula is: Among them, M s is the spatial attention map, C is the feature map F merge The number of channels, It's F merge The cth channel of Conv 7×7It is a 7×7 convolution, [] represents the concatenation operation, which multiplies the channel attention weight and spatial attention weight with the original feature to enhance the feature expression and generate a mixed feature: Among them, F att It is the feature enhanced by CBAM. Indicates element-by-element multiplication, F merge is the feature after cross-scale fusion, M c and M s are the calculated channel and spatial attention maps respectively.

[0013] Furthermore, in the frequency feature enhancement processing step, octave convolution is used to decompose the mixed features into high-frequency and low-frequency components. The high-frequency part retains the original resolution and is used to capture the detailed texture information in the image. The low-frequency part reduces the resolution by downsampling to capture semantic and contextual information. The formula is: Among them, F H is the high frequency component feature, F L is the low-frequency component feature, OctConv H and OctConv L They are octave convolution operations to extract high-frequency and low-frequency components, and the input is the CBAM enhanced feature F att , 3×3 convolution is used to process the high-frequency and low-frequency components respectively to further extract features, and the processed high-frequency and low-frequency components are re-joined and fused to generate frequency enhancement features: F freq =Conv 3×3 ([F H ;Upsample(F L )]), where F freq It is the feature after frequency enhancement, Conv 3×3 It is a 3×3 convolution, [;] represents the concatenation operation, and Upsample is the upsampling operation, which converts the low-frequency component F L Upsample to the appropriate size, F H and F L It is the high and low frequency features after the decomposition of the octave convolution.

[0014] Furthermore, in the spatiotemporal feature aggregation step, the neighborhood attention mechanism, the time Roll mechanism and the LayerNorm residual structure are used to construct the spatiotemporal feature aggregation process, and the frequency enhancement features are divided into multi-way tensors according to the scale to retain the feature information of different spatial resolutions. The time Roll operation is used to move the features of the current frame in the time dimension and project them into the query matrix Q. The features of the previous frame are also processed and projected into the key matrix K and the value matrix V, thereby constructing key-value pairs, and calculating the attention in the pixel neighborhood. By setting the neighborhood window of appropriate size, the attention weight is calculated in the neighborhood range of each pixel point. At the same time, the relative position bias is used to capture the local Texture and displacement information. When calculating the attention weight, the relative position relationship between pixels is considered, so that the model can better capture the local structural changes in the image. A time branch is constructed for each scale. In each time branch, the current query frame is matched one-to-one with the historical frame, and the attention is calculated based on the constructed key-value pairs. The calculated attention results are fused, and the fused features are subjected to the Dropout operation to randomly discard some neurons to prevent overfitting. Then, after residual connection and LayerNorm normalization, the residual connection adds the original input features to the processed features to avoid network degradation, and outputs a mixed feature map to achieve efficient temporal information aggregation.

[0015] Furthermore, in the spatiotemporal feature aggregation step, the time roll operation is used to move the features of the current frame in the time dimension and project them into the query matrix Q. The features of the previous frame are also processed and projected into the key matrix K and the value matrix V, thereby constructing key-value pairs. The formula is: Among them, Q t is the query matrix of the current frame t, K t-1 is the key matrix of the previous frame t-1, V t-1 is the value matrix of the previous frame t-1, Proj q 、Proj k 、Proj v They are projection function, F freq,t and F freq,t-1 are the frequency enhancement features of the current frame and the previous frame respectively. Attention is calculated within the pixel neighborhood. By setting a neighborhood window of appropriate size, the attention weight is calculated within the neighborhood of each pixel. At the same time, the relative position bias is used to capture local texture and displacement information. When calculating the attention weight, the relative position relationship between pixels is considered, so that the model can better capture the local structural changes in the image. The calculation formula is: V t-1 , where A is the result of attention calculation, d k is the key matrix K t-1dimension, B is the relative position bias matrix, Softmax is the normalization function, Q t , K t-1 、V t-1 is the constructed key-value pair matrix.

[0016] Furthermore, in the spatiotemporal feature aggregation step, the calculated attention results are fused, and the fusion formula is: time =LayerNorm(F freq,t +Dropout(A)), where F time is the fused temporal feature, LayerNorm is the layer normalization operation, Dropout is the random inactivation operation, F freq,t is the frequency enhancement feature of the current frame, and A is the neighborhood attention calculation result.

[0017] Furthermore, in the post-segmentation processing step, the segmentation results are subjected to morphological operations of expansion and corrosion to obtain the thyroid ultrasound image nodule segmentation results, the formula of which is: F final =Erode(Dilate(F seg ,S),S), where F final is the final segmentation result, F seg is the preliminary segmentation result output by the UNet decoder, Dilate is the expansion operation, Erode is the corrosion operation, S is the structural element, and F seg Preliminary segmentation output from UNet for frequency-enhanced features.

[0018] Compared with the existing technology, this thyroid ultrasound image nodule segmentation method based on spatiotemporal features and frequency enhancement has the following beneficial effects:

[0019] 1. The present invention By designing an intra-frame feature extraction architecture based on frequency feature fusion and efficient spatiotemporal features The aggregation mechanism realizes the coordinated optimization between intra-frame and inter-frame. In the frame, the high-frequency and low-frequency features are decomposed by the octave convolution, which significantly improves the Improves the model's ability to perceive ultrasound image details and semantic information, enabling the model to more accurately capture the internal texture of nodules. The local cyclic neighborhood propagation mechanism is used between frames to effectively suppress the effect of thyroid motion on temporal feature extraction. interference, achieving efficient temporal information aggregation and accurately capturing the movement and change information of nodules between different frames. The accuracy and efficiency of ultrasound video nodule segmentation have been greatly improved, providing more reliable support for clinical diagnosis.

[0020] 2. The present invention uses a local cyclic neighborhood propagation mechanism to divide the input features into multi-way tensors according to scale, and performs a Roll operation in the time dimension to construct key-value pairs. It calculates attention within the pixel neighborhood and uses relative position bias to capture local texture and displacement information, thereby achieving efficient temporal information aggregation, effectively suppressing the interference of thyroid movement on temporal feature extraction, reducing computational complexity, and greatly improving computational efficiency while ensuring segmentation accuracy. It can quickly and accurately complete nodule segmentation in ultrasound video sequences, meeting the real-time requirements of clinical diagnosis.

[0021] Other advantages, objects and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be learned from the practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0023] Figure 1 This is a flowchart of a thyroid ultrasound image nodule segmentation method based on spatiotemporal features and frequency enhancement;

[0024] Figure 2 This is a flowchart of a thyroid ultrasound image nodule segmentation method based on spatiotemporal features and frequency enhancement. DETAILED DESCRIPTION

[0025] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0026] Example 1

[0027] In the thyroid ultrasound diagnosis department of a certain tertiary hospital, a large number of patients with thyroid diseases are treated every day. Doctors selected 100 representative cases of thyroid ultrasound imaging data from the hospital's huge ultrasound imaging database based on the patients' clinical symptoms, age, gender and other factors. These data cover cases at different stages of the disease, with different nodule sizes and shapes, such as Figure 1 As shown in the figure, these images are first preprocessed. The grayscale values ​​of the images are accurately adjusted to the range of [0, 1] through grayscale normalization, which completely eliminates the grayscale differences of images collected by different ultrasound devices, so that the images reach a unified standard at both the visual and data levels. Then, a 3×3 median filter window is used to perform detailed denoising on the images. The calculation formula is: I denoise (i,j)=median{I norm (m,n),(m,n)∈W(i,j)}, where I denoise (i,j) is the image after median filtering, I norrm(m,n) is the pixel value of the normalized image at coordinate (m,n), W(i,j) is the filter window centered at coordinate (i,j), and median represents the median operation, which effectively removes interference factors such as salt and pepper noise while retaining the edge information of the nodule to the greatest extent, laying a good foundation for subsequent feature extraction. Subsequently, the pre-trained ResNet34 backbone network is used to perform deep feature extraction on the pre-processed image. After multi-layer convolution and pooling operations, rich multi-scale features are obtained from different resolutions. The deepest features output by the backbone network are then processed through additional designed convolution layers to further explore the feature depth and generate high-level features. The calculation formula is: Among them, F l Multi-scale features of different levels l extracted by the backbone network, I denoise is the image after median filtering, Conv l is the l-th layer convolution operation, F deepest It is the deepest feature output by the backbone network, Conv h It is the convolution operation for further processing of the deepest features, F h It is the generated high-level features that provide more valuable and richer information for subsequent analysis.

[0028] The extracted multi-scale features are first reasonably adjusted to the number of channels through 1×1 convolution to meet the fusion requirements and ensure that the features of each scale can be effectively integrated. Then, the adjusted features of each scale are accurately spliced ​​and fused to obtain the fused features. Then, the fused features are input into the CBAM module. In the channel dimension, the global information of the feature map in the channel dimension is fully obtained through global average pooling and global maximum pooling. Then, the accurate channel attention weight is calculated through a carefully designed multi-layer perceptron and activation function. The formula is: Among them, M c is the channel attention map, H and W are the fused feature maps F merge The height and width of ,σ is the activation function, F merge It is the feature after cross-scale fusion. In the spatial dimension, the feature map is carefully average pooled and max pooled before being spliced. Then, it passes through a specific convolution layer and activation function to obtain the precise spatial attention weight. The formula is: Among them, M s is the spatial attention map, C is the feature map F merge The number of channels, It's F merge The cth channel of Conv 7×7 It is a 7×7 convolution, [] represents the concatenation operation, and the input is the feature F after cross-scale fusionmerge , multiply these two weights with the original features to generate mixed features: Among them, F att It is the feature enhanced by CBAM. Indicates element-by-element multiplication, F merge is the feature after cross-scale fusion, M c and M s The calculated channel and spatial attention maps significantly enhance the feature expression and obtain high-quality mixed features. Then, octave convolution is used to decompose the mixed features into high-frequency and low-frequency components. The high-frequency components retain the original resolution to accurately capture the subtle details of the nodules. The low-frequency components are downsampled by 2 times to effectively obtain semantic and contextual information. The formula is: Among them, F H is the high frequency component feature, F L is the low-frequency component feature, OctConv H and OctConv L They are octave convolution operations to extract high-frequency and low-frequency components, and the input is the CBAM enhanced feature F att , the high-frequency and low-frequency components are processed in depth with 3×3 convolution and then re-fused to generate frequency enhancement features: F freq =Conv 3×3 ([F H ;Upsample(F L )]), where F freq It is the feature after frequency enhancement, Conv 3×3 It is a 3×3 convolution, [;] represents the concatenation operation, and Upsample is the upsampling operation, which converts the low-frequency component F L Upsample to the appropriate size, F H and F L It is the high and low frequency features after the decomposition of the octave convolution, and is passed to the UNet decoder for preliminary segmentation, providing strong support for subsequent accurate analysis.

[0029] Since some of the hospital's high-end ultrasound equipment has the ability to capture video sequence images, for this type of dynamic data, the frequency-enhanced features are cleverly divided into multi-channel tensors according to scale. Using the time roll operation, the features of the current frame and the previous frame are accurately projected into the query matrix Q, key matrix K, and value matrix V, respectively. Valid key-value pairs are successfully constructed using the formula: Among them, Q t is the query matrix of the current frame t, K t-1 is the key matrix of the previous frame t-1, V t-1 is the value matrix of the previous frame t-1, Proj q 、Proj k 、Proj vThey are projection function, F freq,t and F freq,t-1 are the frequency enhancement features of the current frame and the previous frame respectively. A 7×7 neighborhood window is set to perform accurate attention weight calculation within the pixel neighborhood. The calculation formula is: Among them, A is the result of attention calculation, d k is the key matrix K t-1 dimension, B is the relative position bias matrix, Softmax is the normalization function, Q t , K t-1 、V t-1 It is a key-value pair matrix constructed, and makes full use of the relative position bias to accurately capture the local texture and displacement information of the nodule. A dedicated time branch is constructed for each scale, and the current query frame is carefully matched with the previous frame to calculate the attention. The fused features are carefully set with the Dropout operation to effectively prevent the occurrence of overfitting. Then, through the residual connection and LayerNorm normalization processing, a high-quality hybrid feature map is output. The fusion formula is: F time =LayerNorm(F freq,t +Dropout(A)), where F time is the fused temporal feature, LayerNorm is the layer normalization operation, Dropout is the random inactivation operation, F freq,t is the frequency enhancement feature of the current frame, and A is the result of neighborhood attention calculation, which realizes the efficient aggregation of temporal information and can clearly show the changes of nodules in the time dimension.

[0030] After the UNet decoder outputs the preliminary segmentation results, it performs refined morphological operations on them. A disk-shaped structural element of a specific size is used for dilation, which successfully connects adjacent nodule areas and perfectly fills the internal small holes. Then, an erosion operation is performed to completely remove small noise areas and protruding burrs, and the segmentation boundary is fully optimized. The formula is: F final =Erode(Dilate(F seg ,S),S), where F final is the final segmentation result, F seg is the preliminary segmentation result output by the UNet decoder, Dilate is the expansion operation, Erode is the corrosion operation, S is the structural element, and F segThe preliminary segmentation output of frequency-enhanced features from UNet ultimately yielded accurate, smooth, and detail-rich thyroid nodule segmentation results. With this result, doctors can more clearly and accurately observe the shape, size, location, and other information of the nodules, providing an intuitive and reliable reference for the subsequent formulation of personalized diagnosis and treatment plans. In actual applications, this method has helped doctors successfully diagnose many cases of tiny thyroid nodules that are easily missed by traditional methods. It has also provided key support for the judgment of nodule properties in some difficult cases, significantly improving the accuracy and efficiency of diagnosis.

[0031] Example 2

[0032] In a project exploring the growth mechanism and early diagnosis technology of thyroid nodules, a medical imaging research institute collected 2,000 thyroid ultrasound image data covering urban and rural areas, different age groups (18-80 years old), and different gender ratios (about 40% male and about 60% female). The data types include single-frame images and continuous 10-20 second video sequences, comprehensively covering all types of nodule cases, including benign, malignant and undetermined ones.

[0033] like Figure 2 As shown, the data was first preprocessed, and the grayscale of 2000 images was normalized one by one. For some images with large grayscale deviations due to equipment aging, histogram matching technology was used for further calibration to ensure that the grayscale of all images was uniform in the range of [0, 1]. In the median filtering stage, an adaptive window size strategy was adopted. The window was automatically increased to 5×5 for noise-intensive areas, while the window was maintained at 3×3 for detail-rich areas. While efficiently removing speckle noise and salt and pepper noise, key details such as burrs and calcifications on the edge of the nodule were fully preserved. Subsequently, a pre-trained ResNet50 backbone network cluster was deployed for feature extraction. Distributed computing was used to accelerate processing. After obtaining multi-scale features, transfer learning technology was introduced. The network was retrained based on the thyroid nodule feature library built by the research institution. The convolution layer parameters were optimized for special morphologies such as cystic changes and aspect ratios of the nodules to generate more targeted high-level features.

[0034] To address the challenge of multi-scale feature fusion, the number of 1×1 convolution kernels is dynamically allocated based on the entropy of the feature map. Convolution kernels are added to information-rich feature maps to retain details, while convolution kernels are reduced for redundant feature maps to avoid computational waste. When the fused features enter CBAM, a parallel computing architecture is adopted to simultaneously perform attention calculations in the channel and spatial dimensions. In the channel attention calculation, a dynamic threshold mechanism is introduced to automatically adjust the activation threshold of the multi-layer perceptron according to the overall energy of the feature map to accurately identify key channels related to nodules. In the spatial attention calculation stage, the initial weights of the convolution layer are optimized in combination with the edge detection operator, so that the model prioritizes the focus on the nodule boundary area. When decomposing the features through octave convolution, a semantic enhancement module is designed for the low-frequency components. The conditional random field model is used to post-process the downsampled low-frequency features to enhance the semantic association between the nodule and the surrounding tissue. The frequency-enhanced features generated by the fusion significantly improve the ability to express the overall distribution and structure of the nodule while retaining details such as tiny nodule protrusions and uneven internal echoes.

[0035] For video sequence data, the frequency enhancement features are first divided into 4-way tensors according to the spatial resolution, corresponding to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions of the original image, respectively. In the temporal roll operation, a bidirectional projection mechanism is introduced to not only construct a key-value pair between the current frame and the previous frame, but also establish an association with the next frame to form temporal context information. In the neighborhood attention calculation link, a dynamic window strategy is adopted to automatically adjust the sliding step size of the 7×7 window according to the nodule movement speed. For fast-moving nodules, the step size is increased to improve computational efficiency, while for slowly changing nodules, the step size is reduced to capture subtle displacements. At the same time, a displacement prediction model is trained based on historical segmentation results, and a relative position bias matrix is ​​dynamically generated to effectively compensate for nodule displacement deviations caused by physiological movements such as breathing and swallowing. The fusion results of each time branch are cross-scale fused through a feature pyramid structure. The final output hybrid feature map can clearly present the growth and deformation trajectory of the nodule in consecutive frames.

[0036] The researchers developed an interactive post-processing platform that supports researchers to manually adjust morphological operation parameters. For different types of nodules, the platform provides preset parameter templates: for nodules with clear boundaries, a disk-shaped structuring element with a smaller radius is used for expansion and erosion; for nodules with blurred boundaries, a diamond-shaped structuring element is switched to and the operation intensity is increased, ultimately obtaining accurate, smooth and detail-rich thyroid nodule segmentation results.

[0037] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make some changes or modifications to equivalent embodiments using the technical contents disclosed above. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A thyroid ultrasound image nodule segmentation method based on spatiotemporal features and frequency enhancement, characterized in that: The method comprises the following specific steps: Multi-scale feature extraction: After grayscale normalization and median filtering preprocessing of thyroid ultrasound images, a pre-trained backbone network is used to extract multi-scale features at different resolutions, and convolution operations are used to generate basic and advanced features. Frequency feature enhancement processing: After cross-scale fusion of multi-scale features and CBAM enhancement, the octave convolution is used to decompose them into high-frequency and low-frequency components. After processing them separately, they are re-fused to generate frequency-enhanced features and passed to the UNet decoder for preliminary segmentation; Spatiotemporal feature aggregation: The input features are divided by scale, key-value pairs are constructed through temporal roll operations, attention is calculated within the pixel neighborhood, information is captured using relative position bias, and temporal information is aggregated through multi-scale temporal branch fusion and normalization. Post-segmentation processing: Perform morphological operations of dilation and erosion on the segmentation results to remove noise areas, fill holes, optimize segmentation boundaries, and obtain the thyroid ultrasound image nodule segmentation results.

2. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 1, characterized in that: In the multi-scale feature extraction step, the pre-trained backbone network is used to extract features from the pre-processed image. The backbone network obtains multi-scale features at different resolutions through multi-layer convolution and pooling operations. During the convolution process, a convolution kernel of a specific size and an appropriate step size are used to extract image features layer by layer. The calculation formula is: Among them, F l Multi-scale features of different levels l extracted by the backbone network, I denoise is the image after median filtering, and its calculation formula is: I denoise (i,j)=median{I norm (m,n),(m,n)∈W(i,j)}, where I norrm (m,n) is the pixel value of the normalized image at coordinate (m,n), W(i,j) is the filter window centered at coordinate (i,j), median represents the median operation, Conv l is the l-th layer convolution operation, F deepest It is the deepest feature output by the backbone network, Conv h It is the convolution operation for further processing of the deepest features, F h It is a generated high-level feature. For the deepest features output by the backbone network, an additional convolution layer is introduced for processing. High-level features are generated through convolution operations to further enhance the feature representation capability.

3. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 1, wherein: In the frequency feature enhancement processing step, the multi-scale features are fused across scales. The number of channels of each scale feature is first adjusted by 1×1 convolution, and the adjusted scale features are spliced ​​and fused to obtain the fused features. The fused features are input into CBAM, and CBAM performs attention calculations on the features from the channel and spatial dimensions respectively. In the channel attention calculation, the global information of the feature map in the channel dimension is obtained by global average pooling and global maximum pooling, and then the channel attention weight is calculated through the multi-layer perceptron and activation function. In the spatial attention calculation, the feature map is averaged and max pooled in the channel dimension, and the results are spliced ​​and the spatial attention weight is obtained through the convolution layer and activation function. The weights of the weights and spatial attention are multiplied with the original features to enhance the feature expression and generate mixed features. The mixed features are decomposed into high-frequency and low-frequency components by octave convolution. The octave convolution divides the feature map into high-frequency and low-frequency parts. The high-frequency part retains the original resolution and is used to capture the detailed texture information in the image. The low-frequency part reduces the resolution by downsampling to capture semantic and contextual information. The high-frequency and low-frequency components are processed by 3×3 convolution respectively to further extract features. The processed high-frequency and low-frequency components are re-spliced ​​and fused to generate frequency-enhanced features. The frequency-enhanced features are passed to the UNet decoder for preliminary segmentation. The UNet decoder gradually restores the image resolution through upsampling and convolution operations, and performs classification prediction on each pixel.

4. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 3, wherein: In the frequency feature enhancement processing step, CBAM performs attention calculation on the features from the channel and spatial dimensions respectively. In the channel attention calculation, the global information of the feature map in the channel dimension is obtained through global average pooling and global maximum pooling, and then the channel attention weight is calculated through the multi-layer perceptron and activation function. The formula is: Among them, M c is the channel attention map, H and W are the fused features F merge The height and width of the generated feature map, σ is the activation function, F merge It is the feature after cross-scale fusion. In the spatial attention calculation, the feature map is average pooled and max pooled in the channel dimension. The results are concatenated and passed through the convolution layer and activation function to obtain the spatial attention weight. The formula is: Among them, M s is the spatial attention map, C is the feature map F merge The number of channels, It's F merge The cth channel of Conv 7×7 It is a 7×7 convolution, [] represents the concatenation operation, which multiplies the channel attention weight and spatial attention weight with the original feature to enhance the feature expression and generate a mixed feature: Among them, F att It is the feature enhanced by CBAM. Indicates element-by-element multiplication, F merge is the feature after cross-scale fusion, M c and M s are the calculated channel and spatial attention maps respectively.

5. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 3, wherein: In the frequency feature enhancement processing step, octave convolution is used to decompose the mixed features into high-frequency and low-frequency components. The high-frequency part retains the original resolution and is used to capture the detailed texture information in the image. The low-frequency part reduces the resolution by downsampling to capture semantic and contextual information. The formula is: Among them, F H is the high frequency component feature, F L is the low-frequency component feature, OctConv H and OctConv L They are octave convolution operations to extract high-frequency and low-frequency components, and the input is the CBAM enhanced feature F att , 3×3 convolution is used to process the high-frequency and low-frequency components respectively to further extract features, and the processed high-frequency and low-frequency components are re-joined and fused to generate frequency enhancement features: F freq =Conv 3×3 ([F H ;Upsample(F L )]), where F freq It is the feature after frequency enhancement, Conv 3×3 It is a 3×3 convolution, [;] represents the concatenation operation, and Upsample is the upsampling operation, which converts the low-frequency component F L Upsample to the appropriate size, F H and F L It is the high and low frequency features after the decomposition of the octave convolution.

6. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 1, characterized in that: In the spatiotemporal feature aggregation step, the neighborhood attention mechanism, the time Roll mechanism and the LayerNorm residual structure are used to construct the spatiotemporal feature aggregation process, the frequency enhancement features are divided into multi-way tensors according to the scale to retain the feature information of different spatial resolutions, and the time Roll operation is used to move the features of the current frame in the time dimension and project them into the query matrix Q. The features of the previous frame are also processed and projected into the key matrix K and the value matrix V, thereby constructing the key-value pair, and calculating the attention in the pixel neighborhood. By setting the neighborhood window of appropriate size, the attention weight is calculated in the neighborhood range of each pixel point, and at the same time, the relative position bias is used to capture the local texture and Displacement information. When calculating the attention weight, the relative position relationship between pixels is considered, so that the model can better capture the local structural changes in the image. A time branch is constructed for each scale. In each time branch, the current query frame is matched one-to-one with the historical frame, and the attention is calculated based on the constructed key-value pairs. The calculated attention results are fused, and the fused features are subjected to the Dropout operation to randomly discard some neurons to prevent overfitting. Then, after residual connection and LayerNorm normalization, the residual connection adds the original input features to the processed features to avoid network degradation, and outputs a mixed feature map to achieve efficient temporal information aggregation.

7. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 6, wherein: In the spatiotemporal feature aggregation step, the time roll operation is used to move the features of the current frame in the time dimension and project them into the query matrix Q. The features of the previous frame are also processed and projected into the key matrix K and the value matrix V, thereby constructing key-value pairs. The formula is: Among them, Q t is the query matrix of the current frame t, K t-1 is the key matrix of the previous frame t-1, V t-1 is the value matrix of the previous frame t-1, Proj q 、Proj k 、Proj v They are projection function, F freq,t and F freq,t-1 are the frequency enhancement features of the current frame and the previous frame respectively. Attention is calculated within the pixel neighborhood. By setting a neighborhood window of appropriate size, the attention weight is calculated within the neighborhood of each pixel. At the same time, the relative position bias is used to capture local texture and displacement information. When calculating the attention weight, the relative position relationship between pixels is considered, so that the model can better capture the local structural changes in the image. The calculation formula is: V t-1 , where A is the result of attention calculation, d k is the key matrix K t-1 dimension, B is the relative position bias matrix, Softmax is the normalization function, Q t , K t-1 、V t-1 is the constructed key-value pair matrix.

8. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 7, characterized in that: In the spatiotemporal feature aggregation step, the calculated attention results are fused, and the fusion formula is: time =LayerNorm(F freq,t +Dropout(A)), where F time is the fused temporal feature, LayerNorm is the layer normalization operation, Dropout is the random inactivation operation, F freq,t is the frequency enhancement feature of the current frame, and A is the neighborhood attention calculation result.

9. The method for segmenting thyroid nodules in ultrasound images based on spatiotemporal features and frequency enhancement according to claim 1, wherein: In the post-segmentation processing step, the segmentation results are subjected to morphological operations of expansion and corrosion to obtain the thyroid ultrasound image nodule segmentation results, and the formula is: F final =Erode(Dilate(F seg ,S),S), where F final is the final segmentation result, F seg is the preliminary segmentation result output by the UNet decoder, Dilate is the expansion operation, Erode is the corrosion operation, S is the structural element, and F seg Preliminary segmentation output from UNet for frequency-enhanced features.

Images