Method for applying spatio-temporal visual transformer in dynamic ultrasound instance segmentation

Abstract

The application relates to the technical field of medical image artificial intelligence, and aims to provide a method for applying a space-time visual Transformer in dynamic ultrasound instance segmentation. The application comprises the following steps: constructing a network structure of a space-time visual Transformer model; collecting dynamic video data of an ultrasound examination and constructing the data set; temporarily constructing a Transformer structure for decoding for a spatial dimension Transformer module, and then pre-training; training, optimizing and testing the space-time visual Transformer model by using a dynamic ultrasound data set; during dynamic ultrasound examination, inputting a current frame of real-time video into the model, then performing real-time application processing, and finally outputting an instance segmentation result of an image. The application can solve the instance segmentation task in the real-time scanning process of ultrasound, thereby simplifying the instance segmentation process and realizing end-to-end training optimization; the space-time features extracted based on the long-distance global attention mechanism of the Transformer can improve the instance segmentation accuracy; the pre-training by using the mask automatic encoding method is easier to optimize and converges faster.

Description

Technical Field

[0001] This invention relates to the field of medical image artificial intelligence technology, and in particular to a method for applying spatiotemporal vision Transformer in dynamic ultrasound instance segmentation. Background Technology

[0002] If we can automatically segment and measure diagnostically valuable areas such as lesions or tissues on ultrasound images, we can reduce the workload of ultrasound physicians, decrease the subjectivity and error in diagnosis or measurement, and improve the efficiency of ultrasound examinations. During real-time ultrasound scanning, if multiple lesions are present, it is necessary to analyze the nature of each lesion. When lesions are close together on certain sections, it is essential to correlate the preceding and following frame information for each lesion. Therefore, real-time instance segmentation on dynamic ultrasound has significant application value.

[0003] Transformer is a deep learning model that fully utilizes attention mechanisms. DETR is an end-to-end object detection algorithm based on the Transformer architecture. Compared to object detection methods using convolutional structures, DETR eliminates many manually designed structures, such as non-maximum suppression and anchor box generation. Relying on the cross-attention decoding mechanism in Transformer, a single network can solve instance segmentation tasks in real-time ultrasound scanning. Using a visual Transformer that simultaneously possesses spatial and temporal dimensions, relying on its long-range global attention mechanism, the accuracy of nodule (lesion) segmentation, identification, and measurement tasks in dynamic ultrasound examinations can be improved.

[0004] However, compared to convolutional neural networks, Transformers require strong data augmentation and regularization strategies, and need to be trained on large datasets for more iterations to achieve good performance. Currently, labeled ultrasound data constitutes a relatively small proportion of other conventional ultrasound image data. Therefore, achieving rapid end-to-end training based on limited ultrasound datasets is quite difficult for spatiotemporal Transformer models, leading to difficulties in real-time instance segmentation during dynamic ultrasound scans.

[0005] Therefore, solving the above problems is of great significance for improving the timeliness, objectivity and accuracy of dynamic ultrasound scanning. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for applying spatiotemporal vision Transformer in dynamic ultrasound instance segmentation.

[0007] To solve the technical problem, the solution of the present invention is:

[0008] A method for applying a spatiotemporal vision Transformer in dynamic ultrasound instance segmentation is provided, comprising the following steps:

[0009] (1) Constructing the network structure of the spatiotemporal visual Transformer model: The network structure includes a spatial Transformer module, a temporal Transformer module and a hybrid Transformer module connected in sequence; after the hybrid Transformer module, a pixel-level decoding module and a Transformer decoding module are connected in parallel; the pixel-level decoding module and the Transformer decoding module jointly output to the mask prediction module; the Transformer decoding module also synchronously outputs to the category prediction module.

[0010] (2) Collect dynamic video data of ultrasound examinations to form a basic dataset; randomly select some data from it, label them to form a dynamic ultrasound dataset, and divide it into training set, optimization set and test set;

[0011] (3) Temporarily construct a Transformer structure for decoding for the spatial dimension Transformer module; then use the basic dataset to pre-train the mask-based autoencoding method for the spatial dimension Transformer module;

[0012] (4) The spatiotemporal vision Transformer model was trained, tuned, and tested using a dynamic ultrasound dataset. The forward propagation process of the network included: the spatial dimension Transformer module encoded all image patches and extracted spatial dimension features; the temporal dimension Transformer module encoded the spatial dimension features and extracted spatiotemporal features; the hybrid Transformer module fine-tuned the extracted spatiotemporal features; the pixel-level decoding module improved the resolution of the finely tuned spatiotemporal feature map, and the Transformer decoding module decoded the finely tuned spatiotemporal features. The outputs of the two modules were sent to the mask prediction module for processing and the mask prediction result was output; the synchronous output of the Transformer decoding module was processed by the category prediction module and the category prediction result was output; finally, the category prediction and mask prediction results were post-processed to obtain the instance segmentation result.

[0013] (5) During dynamic ultrasound examination, the current frame of the real-time video is extracted and input into the spatiotemporal vision Transformer model; then, real-time application processing is performed according to the forward propagation process described in step (4), and finally the instance segmentation result of the image is output.

[0014] As a preferred embodiment of the present invention, in the spatiotemporal vision Transformer network structure, each module has the following structural or functional features:

[0015] The spatial dimension Transformer module is a visual Transformer base network with 12 Transformer encoding network layers. Each Transformer encoding network layer contains a multi-head self-attention structure and a multilayer perceptron structure. Residual connections are used before each structure and layer normalization is performed.

[0016] The temporal Transformer module is a visual Transformer base network with two Transformer encoding network layers.

[0017] Hybrid Transformer module: This is a visual Transformer base network with two Transformer coding network layers;

[0018] Transformer decoding module: It consists of two Transformer decoding network layers. Each Transformer decoding network layer consists of three parts: a multi-head self-attention structure, a multi-head cross-attention structure, and a multi-layer perceptron structure. Residual connections are used before each structure and layer normalization is performed.

[0019] Pixel-level decoding module: The resolution of the spatiotemporal feature map output by the hybrid Transformer module is adjusted by using two 4x bilinear interpolation upsampling operations. After each upsampling, a convolution operation is connected to adjust the number of output spatiotemporal feature maps.

[0020] Mask prediction module: First, a multilayer perceptron structure with two hidden layers is used to map the output of the Transformer decoding module to a mask embedding vector. Then, batch matrix multiplication is performed on the output of the pixel-level decoding module and the mask embedding vector. Finally, the mask prediction result is output.

[0021] Category prediction module: This is a multilayer perceptron structure with two hidden layers. Its function is to map the output of the Transformer decoding module to the category prediction result.

[0022] As a preferred embodiment of the present invention, in step (2), the operations for annotating the data sequentially include: nodule grouping, contour drawing, and hierarchical classification; wherein,

[0023] Nodule grouping refers to determining the total number of nodules in each case video and numbering them to distinguish different nodules; nodule delineation refers to delineating the outline of each nodule in the video for each frame after grouping, and saving the outline as a binary mask image; classification refers to determining the classification label of a nodule by combining all frames in the video that a nodule appears in, using the TI-RADS classification method.

[0024] As a preferred embodiment of the present invention, the pre-training in step (3) includes:

[0025] (3.1) First, a convolutional layer is used to convert the input ultrasound image into non-overlapping image patches, the number of which is the length of the original sequence. After position encoding of the image patches, a portion of the image patches are uniformly and randomly sampled and input into the spatial dimension Transformer module. Then, position encoding and a learnable shared mask vector are used to restore the output of the spatial dimension Transformer module to the original sequence length. Finally, a Transformer structure separately constructed for the spatial dimension Transformer module is used for decoding, and the predicted result of the decoding output has the same size as the input image.

[0026] (3.2) Normalize the remaining image patches that were not sampled, calculate the mean square error between the prediction result and the normalized pixel value, and use it as the reconstruction loss function;

[0027] (3.3) After training on all the basic datasets, a pre-trained model is obtained. The spatial dimension Transformer module is initialized using the pre-trained model.

[0028] As a preferred embodiment of the present invention, step (4) includes:

[0029] (4.1) The input ultrasound image is converted into non-overlapping image patches using a convolutional layer. After position encoding of all image patches, the spatial dimension Transformer module is used for encoding to extract spatial dimension features.

[0030] (4.2) The spatial features are dimensionally swapped, and the image patches at the same position in each frame are taken as a sequence with a sequence length equal to the number of frames in the video segment. After adding the learnable temporal position encoding, the temporal Transformer module is used for encoding. After encoding, the dimensions are restored, that is, all the image patches in each frame are taken as a sequence to obtain the spatiotemporal features. Finally, the hybrid Transformer module is used to fine-tune the spatiotemporal features.

[0031] (4.3) In the pixel-level decoding module, the number and resolution of the spatiotemporal feature maps are adjusted by using upsampling combined with convolution;

[0032] (4.4) In the Transformer decoding module, the cross-attention method is used to decode the spatiotemporal features finely tuned by the hybrid Transformer module. First, the spatiotemporal features and the learnable query vector are input into the cross-attention layer of the Transformer decoding module. After the output of the cross-attention layer is positionally encoded, multi-head self-attention operation is performed. Finally, the multilayer perceptron output is used and enters the next decoding unit.

[0033] (4.5) In the mask prediction module, a multilayer perceptron is used to map the output of the Transformer decoding module to a segmentation embedding vector; then, after batch matrix multiplication with the output of the pixel-level decoding module, the mask prediction result is output.

[0034] (4.6) In the category prediction module, a multilayer perceptron is used to map the synchronous output of the Transformer decoding module to the category prediction result;

[0035] (4.7) Post-process the outputs of the category prediction module and the mask prediction module: The category prediction module outputs the probability of each instance belonging to each category, and the mask prediction module outputs the mask probability image of each instance; calculate the maximum probability and its category for each instance output by the category prediction module, and select instances whose maximum probability value is greater than a set threshold and do not belong to the background category; then select the mask probability images corresponding to these instances from the output of the mask prediction module, multiply the instance's category probability value by the mask probability image to obtain a new mask probability image; calculate the ratio of the number of pixels with the maximum probability value in the new mask probability image to the number of pixels with non-zero pixel values ​​in the binarized mask probability image, and if it is greater than a set threshold, it is considered a valid nodule instance; find all instance segmentation results that meet the conditions and use them as the final output results.

[0036] As a preferred embodiment of the present invention, in step (4), supervised training is performed based on the dynamic ultrasound dataset. During training, the sampling frequency of different nodule video segments is adjusted, and a weighted loss function is used to improve the detection rate of relatively small nodules. The overall model performance is evaluated using the class average accuracy, and the model is tested on the test set after optimization.

[0037] As a preferred embodiment of the present invention, in step (5), during the real-time ultrasound inspection, the spatiotemporal vision Transformer model provides the instance segmentation result of the current frame in real time; when the temporal Transformer module encodes the current frame, it needs to use the feature maps output by the spatial Transformer module of the previous few frames; after the current frame is processed by the temporal Transformer module, the output of the spatial Transformer module of the first frame is no longer saved, but the feature map output by the spatial Transformer module of the current frame needs to be saved for the input of the temporal Transformer module of the next frame; in this way, the instance segmentation result of the video frame at the current moment is continuously output.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] 1. This invention solves the instance segmentation task in the real-time ultrasound scanning process by using a single spatiotemporal vision Transformer network, thereby simplifying the instance segmentation process and achieving end-to-end training optimization.

[0040] 2. This invention utilizes spatiotemporal features extracted based on the Transformer's long-range global attention mechanism, which can improve instance segmentation accuracy. Pre-training the spatial-dimensional Transformer module using an autoencoder mask method is easier to optimize and converges faster than directly using supervised training with the Transformer. Attached Figure Description

[0041] Figure 1 This is a flowchart of the forward propagation of the spatiotemporal vision Transformer network in this invention. Detailed Implementation

[0042] First, it should be noted that this invention relates to machine learning and big data modeling technologies, and is an application of computer technology in the field of medical diagnosis. The implementation of this invention involves the application of multiple software functional modules. The applicant believes that, after carefully reading the application documents and accurately understanding the implementation principles and objectives of this invention, and in conjunction with existing known technologies, those skilled in the art can fully utilize their software programming skills to implement this invention. All references in this application fall within this scope, and the applicant will not list them all further.

[0043] Those skilled in the art will understand that, besides implementing a portion of the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, enabling the system and its various devices, modules, and units to function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0044] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0045] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. These embodiments will enable those skilled in the art to gain a more comprehensive understanding of the present invention, but will not limit the invention in any way.

[0046] Taking the segmentation and grading of thyroid nodules via ultrasound as an example, this invention uses a spatiotemporal vision Transformer to segment thyroid nodules during dynamic ultrasound scanning. The specific steps are as follows:

[0047] Step 1: Construct the spatiotemporal vision Transformer network structure.

[0048] 1. Constructing the network structure of the spatiotemporal visual Transformer model: This network structure includes a spatial dimension Transformer module, a temporal dimension Transformer module, and a hybrid Transformer module connected in sequence; after the hybrid Transformer module, a pixel-level decoding module and a Transformer decoding module are connected in parallel; the pixel-level decoding module and the Transformer decoding module jointly output to the mask prediction module; the Transformer decoding module also synchronously outputs to the category prediction module.

[0049] 2. Each module has the following structural or functional characteristics:

[0050] The spatial dimension Transformer module is a visual Transformer base network with 12 Transformer encoding network layers. Each Transformer encoding network layer contains one multi-head self-attention structure and one multilayer perceptron structure. Residual connections are used before each structure and layer normalization is performed. The output dimension of each layer is 768. Each head of the multi-head self-attention structure has a dimension of 64, and there are 12 heads in total. The multilayer perceptron structure consists of two fully connected layers and uses GELU non-linear activation. The intermediate layer dimension is 3072.

[0051] The temporal Transformer module is a visual Transformer base network with two Transformer encoding network layers.

[0052] Hybrid Transformer module: This is a visual Transformer base network with two Transformer coding network layers;

[0053] Transformer decoding module: It consists of two Transformer decoding network layers. Each Transformer decoding network layer consists of three parts: a multi-head self-attention structure, a multi-head cross-attention structure, and a multi-layer perceptron structure. Residual connections are used before each structure and layer normalization is performed.

[0054] Pixel-level decoding module: The size of the finely tuned spatiotemporal feature map is relatively small, so two 4x bilinear interpolation upsampling operations are used to adjust the resolution of the spatiotemporal feature map output by the hybrid Transformer module; after each upsampling, a convolution operation is connected to adjust the number of output spatiotemporal feature maps;

[0055] Mask prediction module: First, a multilayer perceptron structure with two hidden layers is used to map the output of the Transformer decoding module to a mask embedding vector. Then, batch matrix multiplication is performed on the output of the pixel-level decoding module and the mask embedding vector. Finally, the mask prediction result is output.

[0056] Category prediction module: This is a multilayer perceptron structure with two hidden layers. Its function is to map the output of the Transformer decoding module to the category prediction result.

[0057] Step 2: Collect dynamic video data of thyroid ultrasound examinations to form a basic dataset; randomly select a portion of the data, label it to form a dynamic ultrasound dataset, and divide it into a training set, an optimization set and a test set.

[0058] 1. Collect dynamic video data of thyroid ultrasound on a case-by-case basis, gathering dynamic data from different medical institutions using different equipment types and from doctors' scans. After collection, perform video cleaning to remove unqualified videos, crop non-ultrasound areas, and remove patient-related private information. Establish sensitive information removal naming rules, rename the videos, and form the basic dataset.

[0059] 2. Labeling all the video data in the base dataset would be a massive undertaking. Therefore, a portion of the data is randomly selected from the base dataset for labeling, termed the "labeled dataset," which comprises approximately 1 / 10 of the base dataset. From this labeled dataset, a further random proportion is selected and divided into training, optimization, and testing sets.

[0060] 3. The data annotation process includes three stages: nodule grouping, contour drawing, and classification. "Nodule grouping" involves determining the total number of nodules in each case video and numbering them to distinguish them. "Nodule drawing" uses the grouped nodules as units to draw the contour of each nodule appearing in the video for each frame, saving the contour as a binary mask image. Finally, based on all frames of the nodule's appearance in the video, the TI-RADS classification method is used to determine the nodule's classification label, primarily categorized into Level 1, Level 2, Level 3, Level 4, and Level 5.

[0061] In this invention, the operations of nodule grouping, contour drawing, and grading are all performed by skilled physicians or technicians. The operation methods can be performed according to the methods in existing published literature, and this invention does not make any special requirements.

[0062] Step 3: Using the basic dataset, pre-train the mask-based autoencoding method for the spatial dimension Transformer module.

[0063] 1. First, a temporary Transformer structure for decoding is constructed for the spatial dimension Transformer module. This Transformer structure has two Transformer encoding network layers and is located after the spatial dimension Transformer module.

[0064] 2. Assuming the input image size is S, a convolutional layer is first used to convert the input ultrasound image into non-overlapping 32×32 image patches. The number of image patches is (S / 32)×(S / 32), which is the original sequence length. After positional encoding of the image patches, 25% of these image patches are uniformly and randomly sampled and input into a spatial-dimensional Transformer module. Then, positional encoding and a learnable shared mask vector are used to restore the output of the spatial-dimensional Transformer module to the original sequence length. Finally, a separately constructed Transformer structure is used for decoding, and the decoded prediction result has the same size as the input image.

[0065] 3. Normalize the remaining 75% of image patches that were not sampled, and calculate the mean square error between the predicted result and the normalized pixel value, using this as the loss function for the mask autoencoding method. The significance of this is that it uses only 25% of the image patches to predict the content of the remaining 75% of the image. Since the mask autoencoding method is an unsupervised training method, it can be trained on large amounts of unlabeled ultrasound datasets, and using only 25% of the image patches can significantly improve training speed. Initializing the spatial dimension Transformer with a pre-trained model followed by supervised fine-tuning will converge faster and produce better model performance than directly using the Transformer network for supervised training.

[0066] 4. After training on all the basic datasets, a pre-trained model is obtained. The spatial dimension Transformer module is initialized using the pre-trained model, and then the Transformer structure used for decoding is removed.

[0067] 5. Table 1 provides the parameter configuration for pre-training the spatial-dimensional Transformer using the mask autoencoding method in this example. The data augmentation method uses only random scaling and cropping, with the aspect ratio of the randomly cropped rectangular regions randomly and uniformly selected within the range of [3 / 4, 4 / 3] and the area randomly and uniformly selected within the range of [20%, 100%].

[0068] Table 1

[0069] Parameter name value Optimizer AdamW Base learning rate 1e-4 Weight decay 0.05 Optimizer impulse <![CDATA[β1=0.9,β2=0.95]]> Batch size 8 Learning rate decay strategy cosine decay Pre-iteration count 50000 Total number of iterations 800000 Data augmentation Random scaling cropping

[0070] Step 4: Train, tune, and test the spatiotemporal vision Transformer model using a dynamic ultrasound dataset.

[0071] 1. The size of the input image after preprocessing is S. A convolutional layer is used to convert the input image into non-overlapping 32×32 image patches. The spatial dimension Transformer module is used to encode all image patches and extract spatial dimension features, denoted as L×(N×f)×E, where L represents the sequence length, N is the training batch size, f represents the length of the selected video segment, and E is the dimension of each sequence. Here, S=640, L=400, and E=768.

[0072] 2. The spatial feature L×(N×f)×E undergoes dimensionality swapping to obtain f×(N×L)×E. This represents treating image patches at the same location in each frame as a sequence, with the sequence length equal to the number of frames f in the training video segment. After adding learnable temporal location encoding, the temporal Transformer module is used for encoding. Dimensional restoration is then performed, treating all image patches in each frame as a sequence. The restored feature is the spatiotemporal feature, denoted as L×(N×f)×E, which simultaneously possesses spatial and temporal dimensional information. f Finally, a hybrid Transformer module with two Transformer coding network layers was used to fine-tune the spatiotemporal features.

[0073] 3. In the pixel-level decoding module, upsampling combined with convolution is used to adjust the number and resolution of the output spatiotemporal feature maps. The spatiotemporal features are converted into feature map form, with a feature map size 1 / 32 of the input image size. First, a fully connected layer is used to... f Adjusted to E d After performing a dimension swap, we get Then, two 4x upsampling and convolution operations are performed, and the final output feature map is (N×f)×E. d ×(S / 2)×(S / 2), where the feature map resolution is 1 / 2 of the input image.

[0074] 4. A Transformer decoding module with cross-attention is used to decode the spatiotemporal features finely tuned by the hybrid Transformer module. The spatiotemporal features and the learnable query vector are input into the cross-attention layer of the Transformer decoding module. The output of the cross-attention layer is positionally encoded, then subjected to multi-head self-attention operation, and finally output using a multilayer perceptron and fed into the next decoding unit. In the mask prediction module, a multilayer perceptron is used to map the output of the Transformer decoding module to a segmentation embedding vector. This vector is then multiplied by a batch matrix with the output of the pixel-level decoding module to output the mask prediction result. In the category prediction module, a multilayer perceptron is used to map the output of the Transformer decoding module to the category prediction result.

[0075] 5. Post-process the outputs of the category prediction module and the mask prediction module to obtain the final instance segmentation result for each thyroid nodule. The category prediction module outputs the probability of each instance belonging to each category, and the mask prediction module outputs the mask probability image for each instance. Calculate the maximum probability and its category for each instance output by the category prediction module. Select instances with a maximum probability value greater than 0.8 that do not belong to the background category. Select the corresponding mask probability images from the mask prediction module output for these instances. Multiply the instance's category probability value by the mask probability image to obtain a new mask probability image. Calculate the number of pixels occupying the maximum probability value in the new mask probability image, denoted as s. m Then, the mask probability image is binarized using a threshold of 0.5, and the number of pixels with non-zero pixel values ​​is calculated, denoted as s. t If s m / s t If the value is greater than 0.8, the mask image and its category are considered a valid nodule instance. Find all instance segmentation results that meet the criteria and output them.

[0076] 6. Supervised training, tuning, and testing of the spatiotemporal vision Transformer model on a dynamic ultrasound dataset (annotated dataset) were performed. Since the number of video frames containing nodules is relatively small and the nodule sizes vary, the sampling frequency of different nodule video segments needed to be adjusted during training. A loss function weighted equally by Lowaz and cross-entropy was used to improve the detection rate of relatively small nodules. Class average accuracy was used to evaluate model performance. The model was tested on the tuning set, and appropriate initial learning rate, learning rate decay strategy, pre-iteration count, and total number of iterations were selected. The trained model was then tested on the test set after tuning. Table 2 provides the parameter configuration for fine-tuning the spatiotemporal vision Transformer model in this example. The data augmentation methods used included scaling, motion blur, brightness, contrast, and sharpness adjustments.

[0077] Table 2

[0078] Parameter name value Optimizer AdamW Base learning rate 5e-5 Weight decay 0.05 Optimizer impulse <![CDATA[β1=0.9,β2=0.999]]> Batch size 4 Learning rate decay strategy cosine decay Pre-iteration count 10000 Total number of iterations 300000 Layer-by-layer learning rate decay 0.75 Scale [0.5,1.5] Motion blur intensity [0,15] Brightness, contrast and sharpness [0.1,1.9]

[0079] Step 5: During dynamic ultrasound examination, extract the current frame of the real-time video and input it into the spatiotemporal vision Transformer model; then perform real-time application inference according to the processing flow in Step 4, and finally output the image segmentation result.

[0080] During real-time ultrasound examination, the spatiotemporal vision Transformer network provides the instance segmentation result of the current frame in real time. When the temporal Transformer module encodes the f-th frame (the current frame), it needs to use the feature maps output by the spatial Transformer modules of the previous f-1 frames. After the f-th frame is processed by the temporal Transformer module, the output of the spatial Transformer module of the first frame is no longer saved, but the feature map output by the spatial Transformer module of the f-th frame needs to be saved for use as input to the temporal Transformer module of the next frame. In this way, the instance segmentation result of the current video frame is continuously output.

Claims

1. A method for applying a spatiotemporal vision Transformer in dynamic ultrasound instance segmentation, characterized in that, Includes the following steps: (1) Constructing the network structure of the spatiotemporal visual Transformer model: The network structure includes a spatial Transformer module, a temporal Transformer module and a hybrid Transformer module connected in sequence; after the hybrid Transformer module, a pixel-level decoding module and a Transformer decoding module are connected in parallel; the pixel-level decoding module and the Transformer decoding module jointly output to the mask prediction module; the Transformer decoding module also synchronously outputs to the category prediction module. (2) Collect dynamic video data of ultrasound examinations to form a basic dataset; randomly select some data from it, label them to form a dynamic ultrasound dataset, and divide it into training set, optimization set and test set; (3) Temporarily construct a Transformer structure for decoding for the spatial dimension Transformer module; then use the basic dataset to pre-train the mask-based autoencoding method for the spatial dimension Transformer module; (4) The spatiotemporal vision Transformer model was trained, tuned and tested using a dynamic ultrasound dataset. The forward propagation process of the network includes: the spatial dimension Transformer module encodes all image patches and extracts spatial dimension features. The temporal Transformer module encodes spatial features to extract spatiotemporal features; the hybrid Transformer module fine-tunes the extracted spatiotemporal features; a pixel-level decoding module improves the resolution of the fine-tuned spatiotemporal feature map, and a Transformer decoding module decodes the fine-tuned spatiotemporal features. The outputs of both modules are sent to the mask prediction module for processing, outputting the mask prediction result; the synchronous output of the Transformer decoding module is processed by the class prediction module to output the class prediction result; finally, the class prediction and mask prediction results are post-processed to obtain the instance segmentation result. (5) During dynamic ultrasound examination, the current frame of the real-time video is extracted and input into the spatiotemporal vision Transformer model; then, real-time application processing is performed according to the forward propagation process described in step (4), and finally the instance segmentation result of the image is output.

2. The method according to claim 1, characterized in that, In the spatiotemporal vision Transformer network structure, each module has the following structural or functional characteristics: The spatial dimension Transformer module is a visual Transformer base network with 12 Transformer encoding network layers. Each Transformer encoding network layer contains a multi-head self-attention structure and a multilayer perceptron structure. Residual connections are used before each structure and layer normalization is performed. The temporal Transformer module is a visual Transformer base network with two Transformer encoding network layers. Hybrid Transformer module: This is a visual Transformer base network with two Transformer coding network layers; Transformer decoding module: It consists of two Transformer decoding network layers. Each Transformer decoding network layer consists of three parts: a multi-head self-attention structure, a multi-head cross-attention structure, and a multi-layer perceptron structure. Residual connections are used before each structure and layer normalization is performed. Pixel-level decoding module: The resolution of the spatiotemporal feature map output by the hybrid Transformer module is adjusted by using two 4x bilinear interpolation upsampling operations. After each upsampling, a convolution operation is connected to adjust the number of output spatiotemporal feature maps. Mask prediction module: First, a multilayer perceptron structure with two hidden layers is used to map the output of the Transformer decoding module to a mask embedding vector. Then, batch matrix multiplication is performed on the output of the pixel-level decoding module and the mask embedding vector. Finally, the mask prediction result is output. Category prediction module: This is a multilayer perceptron structure with two hidden layers. Its function is to map the output of the Transformer decoding module to the category prediction result.

3. The method according to claim 1, characterized in that, In step (2), the data annotation operations include, in sequence: nodule grouping, contour drawing, and hierarchical classification; among which, Nodule grouping refers to determining the total number of nodules in each case video and numbering them to distinguish different nodules; nodule delineation refers to delineating the outline of each nodule in the video for each frame after grouping, and saving the outline as a binary mask image; classification refers to determining the classification label of a nodule by combining all frames in the video that a nodule appears in, using the TI-RADS classification method.

4. The method according to claim 1, characterized in that, The pre-training mentioned in step (3) includes: (3.1) First, a convolutional layer is used to convert the input ultrasound image into non-overlapping image patches, the number of which is the length of the original sequence. After position encoding of the image patches, a portion of the image patches are uniformly and randomly sampled and input into the spatial dimension Transformer module. Then, position encoding and a learnable shared mask vector are used to restore the output of the spatial dimension Transformer module to the original sequence length. Finally, a Transformer structure separately constructed for the spatial dimension Transformer module is used for decoding, and the predicted result of the decoding output has the same size as the input image. (3.2) Normalize the remaining image patches that were not sampled, calculate the mean square error between the prediction result and the normalized pixel value, and use it as the reconstruction loss function; (3.3) After training on all the basic datasets, a pre-trained model is obtained. The spatial dimension Transformer module is initialized using the pre-trained model.

5. The method according to claim 1, characterized in that, Step (4) includes: (4.1) The input ultrasound image is converted into non-overlapping image patches using a convolutional layer. After position encoding of all image patches, the spatial dimension Transformer module is used for encoding to extract spatial dimension features. (4.2) The spatial features are dimensionally swapped, and the image patches at the same position in each frame are taken as a sequence with a sequence length equal to the number of frames in the video segment. After adding the learnable temporal position encoding, the temporal Transformer module is used for encoding. After encoding, the dimensions are restored, that is, all the image patches in each frame are taken as a sequence to obtain the spatiotemporal features. Finally, the hybrid Transformer module is used to fine-tune the spatiotemporal features. (4.3) In the pixel-level decoding module, the number and resolution of the spatiotemporal feature maps are adjusted by using upsampling combined with convolution; (4.4) In the Transformer decoding module, the cross-attention method is used to decode the spatiotemporal features finely tuned by the hybrid Transformer module. First, the spatiotemporal features and the learnable query vector are input into the cross-attention layer of the Transformer decoding module. After the output of the cross-attention layer is positionally encoded, multi-head self-attention operation is performed. Finally, the multilayer perceptron output is used and enters the next decoding unit. (4.5) In the mask prediction module, a multilayer perceptron is used to map the output of the Transformer decoding module to a segmentation embedding vector; then, after batch matrix multiplication with the output of the pixel-level decoding module, the mask prediction result is output. (4.6) In the category prediction module, a multilayer perceptron is used to map the synchronous output of the Transformer decoding module to the category prediction result; (4.7) Post-process the outputs of the category prediction module and the mask prediction module: The category prediction module outputs the probability of each instance belonging to each category, and the mask prediction module outputs the mask probability image of each instance; calculate the maximum probability and its category for each instance output by the category prediction module, and select instances whose maximum probability value is greater than a set threshold and do not belong to the background category; then select the mask probability images corresponding to these instances from the output of the mask prediction module, multiply the instance's category probability value by the mask probability image to obtain a new mask probability image; calculate the ratio of the number of pixels with the maximum probability value in the new mask probability image to the number of pixels with non-zero pixel values ​​in the binarized mask probability image, and if it is greater than a set threshold, it is considered a valid nodule instance; find all instance segmentation results that meet the conditions and use them as the final output results.

6. The method according to claim 1, characterized in that, In step (4), supervised training is performed based on the dynamic ultrasound dataset. During training, the sampling frequency of different nodule video segments is adjusted, and a weighted loss function is used to improve the detection rate of relatively small nodules. The overall model performance is evaluated using the class average accuracy, and the model is tested on the test set after optimization.

7. The method according to claim 1, characterized in that, In step (5), during the real-time ultrasound inspection, the spatiotemporal vision Transformer model provides the instance segmentation result of the current frame in real time. When the temporal Transformer module encodes the current frame, it needs to use the feature maps output by the spatial Transformer module of the previous few frames. After the current frame is processed by the temporal Transformer module, the output of the spatial Transformer module of the first frame is no longer saved, but the feature map output by the spatial Transformer module of the current frame needs to be saved for the input of the temporal Transformer module of the next frame. In this way, the instance segmentation result of the video frame at the current moment is continuously output.

Images