Ultrasonic sequence tracking method based on knowledge distillation assisted by optical flow and feature accumulation

Abstract

The present invention relates to a knowledge distillation ultrasound sequence tracking method based on optical flow assistance and feature accumulation, and belongs to the field of ultrasound image tracking. It includes the following steps: obtaining ultrasound sequence data, and dividing the data into a training set and a test set; constructing a knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation, the model includes a teacher network and a student network; the teacher network includes a feature encoder, a visual converter, a feature accumulation module and a prediction layer; the student network is a lightweight network based on the visual converter, including a projection layer, a visual converter and a prediction layer; the ultrasound sequence in the training set is respectively input into the teacher network and the student network for image analysis; the model is trained by the total loss of the teacher network and the student network to obtain a trained model; the ultrasound sequence in the test set is input into the trained model to obtain the final tracking result of the target. The present invention can improve the accuracy of ultrasound image target tracking.

Description

Technical Field

[0001] The present invention belongs to the field of ultrasonic image tracking, and in particular relates to a knowledge distillation ultrasonic sequence tracking method based on optical flow assistance and feature accumulation. Background Art

[0002] Image-guided radiotherapy is a treatment method that uses real-time imaging technology to locate and adjust the lesion. Ultrasound (US) imaging is a method suitable for observing tumors, detecting various organs and tissues, and diagnosing various diseases due to its advantages such as lack of radiation, real-time monitoring, high precision, and low cost. So far, ultrasound is mainly used for diseases such as prostate cancer, liver cancer, and tumor metastasis.

[0003] However, the patient's respiratory movement may affect the accuracy of ultrasound-guided radiotherapy, especially during radiotherapy of the liver, where the movement of the liver may be as long as several centimeters due to the patient's breathing rate. In order to minimize movement, patients must be required to receive treatment while holding their breath. An important disadvantage of the breath-holding condition is that it may require patients to make repeated attempts, especially when the patient strives to maintain breathing throughout the process, which is likely to increase the patient's discomfort and may affect the accuracy of the results. And in current clinical conditions, many SBRT (stereotactic body radiation therapy) patients must be treated under free breathing conditions for various reasons. In terms of diagnosis and treatment, in order to minimize the impact of breathing on image-guided radiotherapy, during treatment, doctors generally use a larger imaging range for the movement of these organs, for example, the range of motion of the lungs is about 1-17.5 mm, the liver is about 10-54 mm, the kidneys are about 10-40 mm, the diameter of the kidneys is about 10-40 mm, and the pancreas is about 20-40 mm. But the disadvantage is that they reduce the advantages of guided radiotherapy and introduce certain errors and limitations. In existing ultrasound surgery, visual tracking is the most common method. However, due to the high noise, tissue occlusion and instability of human breathing rate in ultrasound images, visual tracking alone is often disturbed. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a knowledge distillation ultrasonic sequence tracking method based on optical flow assistance and feature accumulation.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a knowledge distillation ultrasound sequence tracking method based on optical flow assistance and feature accumulation, comprising the following steps:

[0007] S1. Obtain ultrasound sequence data, and divide the ultrasound sequence data into a training set and a test set;

[0008] S2. Construct a knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation, the model includes a teacher network and a student network; the teacher network includes a feature encoder, a visual converter, a feature accumulation module and a prediction layer; the student network is a lightweight network based on the visual converter, including a projection layer, a visual converter and a prediction layer; the ultrasound sequence in the training set is respectively input into the teacher network and the student network for image analysis;

[0009] S3. Train the knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation through the total loss of the teacher network and the student network to obtain a trained model;

[0010] S4. Input the ultrasonic sequence in the test set into the trained model to obtain the final tracking result of the target.

[0011] Furthermore, the feature encoder of the teacher network in step S2:

[0012] The feature encoder includes a linear projection layer and a feature enhancement module; the feature enhancement module includes a convolutional network, a Monte Carlo attention module and a convergence transformer; the convolutional network includes a A convolutional layer and a Convolutional layer, the convergence transformer includes a 3×3 convolutional layer, a 1×1 convolutional layer, two visual converters and two 1×1 convolutional layers in sequence;

[0013] The ultrasound sequence in the training set is input into the Siamese network to extract the template image and search images , the template image and search images Split into patches and flatten them to get template sequences and search sequence ,in represents the resolution of each patch, Represents the number of patches in the template area, Represents the number of patches in the search area; the template sequence and search sequence The sequence is projected into the latent space through a linear projection layer Get the template token and search token , the formula is as follows:

[0014] ,

[0015] ,

[0016] in, The symbols represent the parameters of the linear projection layer. and Represent the learnable position embeddings of the template sequence and the search sequence respectively; the template token and search token After the feature enhancement module is used to extract features, the template image features are obtained. and search image features , is the dimension of the feature map.

[0017] Furthermore, the visual converter of the teacher network in step S2:

[0018] The visual converter includes a multi-head attention layer, a candidate elimination module and an optical flow auxiliary module; the candidate elimination module includes a sorter, a target position mask generator, a mask fusion, a normalization layer and a multi-layer perceptron, and the optical flow auxiliary module includes an optical flow extractor, an optical flow mask generator and a feature extractor;

[0019] The template image features and search image features After linear transformation of the multi-head attention layer, we get the query matrix Q, key matrix K and value matrix V, which are expressed as follows:

[0020] ,

[0021] ,

[0022] ,

[0023] in, Represent the weight matrices learned by the query matrix Q, key matrix K, and value matrix V, respectively. represents the template image features, Represents the search image features; for the query matrix , key matrix Sum Matrix Perform separate operations and process each head Dimensional submatrix, for each head Calculate the attention score, concatenate the outputs of all heads, and map them back to the original dimension through a linear layer to obtain the feature map processed by the multi-head self-attention layer , the formula is as follows:

[0024] ,

[0025] ,

[0026] ,

[0027] ,

[0028] ,

[0029] in , Represents the number of heads, represents the weight matrix of the multi-head attention layer, represents the linear layer function connecting all head attention scores, represents the operation of calculating the attention score, Represents the feature map processed by the multi-head self-attention layer; in the candidate elimination module, the bounding box mask is generated based on the bounding box information of the annotated file in the training set ultrasound sequence , the feature map processed by the multi-head self-attention layer The sorter scores the similarity between the template area and the search area. Before obtaining The target position mask generator then outputs the filtered mask , the formula is as follows:

[0030] ,

[0031] in, Indicates the reserved area. Before The similarity score of the regions is high. represents an adjustable parameter; the bounding box mask and filter mask The mask fuser performs pixel-by-pixel AND operations to obtain the tracking target mask. , the tracking target mask After processing by the normalization layer and the multi-layer perceptron, the position characteristics of the tracked target are obtained. ;

[0032] Extract the search image from the Siamese network The optical flow extractor extracts the optical flow and obtains the forward optical flow of adjacent video frames. and backward optical flow ; Forward optical flow and backward optical flow Converted into a forward optical flow image through the optical flow mask generator and the backward optical flow image ; The forward optical flow image and the backward optical flow image Passing the threshold Filter to remove areas where motion is not obvious, and then generate a forward optical flow mask for the tracked target and the backward optical flow mask of the tracked target ; Finally, the forward optical flow mask of the tracked target and the backward optical flow mask of the tracked target Perform pixel-by-pixel AND operation to obtain the optical flow mask of the tth frame , the optical flow mask of the tth frame and Search image for frame Connect them and send them to the feature extractor to obtain the motion features of the target , the formula is as follows:

[0033] ,

[0034] ,

[0035] ,

[0036] ,

[0037] ,

[0038] in, represents the optical flow extractor, Represents the operation of converting optical flow into optical flow image, represents the optical flow mask generator, represents pixel-by-pixel AND operation, Representative Frame search for motion features in images, is the image concatenation operation, Represents a feature extractor, which is a transformer encoder.

[0039] Furthermore, the feature accumulation module and prediction layer of the teacher network in step S2:

[0040] The feature accumulation module includes an encoder, a feature storage library, and a decoder; the encoder includes a first residual network, a convolutional attention module, a second residual network, and a 1×1 convolutional layer; the decoder includes a 1×1 convolutional layer, a similarity calculation module, a weighted aggregation module, and a reshaping module;

[0041] Encoder stage: The first Frame search image to The motion characteristics of the target in each frame of the frame search image , location features and search image features After the first residual network is used for feature fusion, the fusion features of each frame of the search image are obtained. , the fusion feature After the convolutional attention module is used for feature enhancement, the enhanced features are obtained. , the fusion feature and Enhanced Features The hint value of each frame search image is obtained through the second residual network At the same time, each frame searches for image features After a 1×1 convolutional layer, the hint key of each frame search image is obtained , the prompt key and prompt value Form a pair of prompt key-value pairs , stored in the feature repository, which stores the Frame to The search image history features of the frame are updated using a first-in-first-out strategy. The shapes of , the formula is as follows:

[0042] ,

[0043] ,

[0044] ,

[0045] ,

[0046] ,

[0047] ,

[0048] ,

[0049] in, The size of the representative memory is an adjustable parameter. represents the first residual network, Representation based on fusion features Calculated channel attention weights, Representation based on fusion features The calculated spatial attention weights, represents the multi-layer perceptron processing function in the convolutional attention module, represents the convolution operation in the convolutional attention module, and Respectively represent the maximum pooling and average pooling along the spatial dimension, and Respectively represent the maximum pooling and average pooling in the channel dimension, represents the Sigmoid function, and denote the element-wise multiplication of channels and pixels, respectively, represents the second residual network, represents a 1×1 convolutional layer;

[0050] Decoder stage: Search image features of frame search images go through The convolutional layer gets the query hint key , calculate the query hint key through the similarity calculation module The similarity score with all the hint keys in the feature repository is used to calculate the attention score , the attention score The attention matrix Pass and memory prompt value Weighted aggregation to obtain aggregate features , the aggregated features The reshaping module is used to reshape the features to obtain the reshaped features ;

[0051] The remodeling features and query prompt keys Connect to form the Prompt information feature map of frame search image ; The prompt information feature map Enter the prediction layer to get the classification score map of the target , local offset and the normalized bounding box size ,in, , , , and Respectively represent the height and width of the search image feature, is the search sequence The patch size is 2, and 2 represents the height and width dimensions; the position with the highest score in the classification score map is the target position , and then get the final target bounding box , the formula is as follows:

[0052]

[0053] in, Indicates the target location The horizontal axis of Indicates the target location The vertical coordinate of Represents the horizontal coordinate of the upper left corner of the prediction box, Represents the ordinate of the upper left corner of the prediction box, Indicates the width of the prediction box, Indicates the height of the prediction box.

[0054] Furthermore, in step S2, the student network:

[0055] The visual converter of the student network includes a normalization layer, a multi-head attention layer, a normalization layer, and a multi-layer perceptron;

[0056] The template image and search images After being processed by the mapping layer, the output features of the mapping layer are obtained, and the output features of the mapping layer are sent to the visual converter for feature extraction to obtain the prompt information feature map of the tracking target. , the prompt information feature map of the tracking target After being processed by the prediction layer, the predicted bounding box is obtained, and then the center position of the target is determined.

[0057] Furthermore, the total loss of the teacher network in step S3 includes classification loss and regression loss:

[0058] In terms of classification loss, the Gaussian Weighted Focal loss function is used, and the formula is as follows:

[0059] ,

[0060] in, and denote the first hyperparameter and the second hyperparameter respectively, Represents the classification score map of the target, Represents the ground truth heat map generated using the Gaussian kernel function;

[0061] The regression loss includes the bounding box prediction loss and the bounding box overlap loss. The bounding box prediction loss is calculated using The loss formula is as follows:

[0062]

[0063] in, represents the predicted bounding box, Represents the true target bounding box; the bounding box overlap loss uses the intersection-over-union (IoU) loss, and the formula is as follows:

[0064] ,

[0065] in, Represents the distance between the center point of the predicted bounding box and the center point of the true target bounding box, represents the diagonal length of the minimum outer frame that encloses these two boxes, represents the intersection of the predicted bounding box and the true target bounding box, represents the union of the predicted bounding box and the true target bounding box, represents the generalized intersection and ratio;

[0066] The total loss of the teacher network is:

[0067] ,

[0068] in, represents the weight of the weighted focal loss function, express The weight of the loss function, Represents the weight of the generalized intersection-over-union loss function.

[0069] Furthermore, the total loss of the student network in step S3 is Including trailing loss and layered distillation loss :

[0070] Tracking loss of the student network Same total loss as the teacher network:

[0071] ,

[0072] The hierarchical distillation loss includes similarity matrix knowledge distillation loss, output feature map knowledge distillation loss and response map knowledge distillation loss; the similarity matrix is ​​the weight matrix calculated by the multi-head self-attention layer in the student network and the teacher network, and the formula of the similarity matrix knowledge distillation loss is as follows:

[0073] ,

[0074] in, represents the i-th layer similarity matrix of the student network, represents the j-th layer similarity matrix of the teacher network; the output feature map is the prompt information feature map of the teacher network and the student network, and the formula of the output feature map knowledge distillation loss is as follows:

[0075] ,

[0076] in, represents the output feature map of the student network, Represents the output feature map of the teacher network; the response map of the teacher tracking network output For target positioning, the response map For distillation, the formula of the response graph knowledge distillation loss is expressed as follows:

[0077] ,

[0078] in, represents the temperature coefficient, and Represents the output response map of the student network and the teacher network for target positioning, divided by the temperature coefficient To normalize the response mapping of the teacher and student networks;

[0079] The total loss of the student network is:

[0080] ,

[0081] in, Represents the knowledge distillation loss function of the similarity matrix The weight of Represents the knowledge distillation loss function of the output feature map The weight of Represents the response graph knowledge distillation loss function The weight of .

[0082] In a second aspect, the present invention further provides a knowledge distillation ultrasonic sequence tracking device based on optical flow assistance and feature accumulation, comprising:

[0083] Data acquisition unit: used for acquiring ultrasound sequence data and dividing the ultrasound sequence data into a training set and a test set;

[0084] Model construction unit: used to construct a knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation, the model includes a teacher network and a student network; the teacher network includes a feature encoder, a visual converter, a feature accumulation module and a prediction layer; the student network is a lightweight network based on the visual converter, including a projection layer, a visual converter and a prediction layer; the ultrasound sequence in the training set is respectively input into the teacher network and the student network for image analysis;

[0085] Training unit: used to train the knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation through the total loss of the teacher network and the student network to obtain a trained model;

[0086] Target tracking unit: used to input the ultrasonic sequence in the test set into the trained model to obtain the final tracking result of the target.

[0087] In a third aspect, the present invention also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the knowledge distillation ultrasonic sequence tracking method based on optical flow assistance and feature accumulation described in the first aspect is performed.

[0088] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the knowledge distillation ultrasonic sequence tracking method based on optical flow assistance and feature accumulation as described in the first aspect.

[0089] The advantages of the present invention are:

[0090] In the teacher network, the present invention uses a feature enhancement module to enhance the network's ability to learn image features; introduces a candidate elimination module to allow the transformer encoder to focus on learning the features of the target; uses an optical flow auxiliary module to collect the optical flow information of the video to construct an optical flow mask to extract the motion features of the target; constructs a feature repository to generate prompt information using past and current image information, thereby strengthening the use of temporal context information; these modules are plug-and-play components to improve the performance of the model. In the student network, the projection layer and transformer encoder are used to extract features, and the final tracking results are generated through the prediction layer. This lightweight model can perform real-time target tracking, and at the same time introduces knowledge distillation to optimize the tracking performance of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0092] Figure 1 is a flow chart of the method of the present invention;

[0093] Figure 2 A network structure diagram of the method of the present invention;

[0094] Figure 3 Graphs showing experimental results of the method of the present invention and other methods. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0096] Example 1

[0097] In this embodiment, Figure 1 As shown, the present invention provides a knowledge distillation ultrasound sequence tracking method based on optical flow assistance and feature accumulation, and the specific steps include:

[0098] S1. Obtain ultrasound sequence data, and divide the ultrasound sequence data into a training set and a test set;

[0099] Specifically, the ultrasound sequence data was obtained from a public free dataset CLUST-2D. It was obtained by four volunteers under free breathing conditions, and the sequences were divided into five groups (CIL, ETH, ICR, MED-1, MED-2). The training set and test set include a total of 63 2D ultrasound liver sequences, 24 ultrasound sequences in the training dataset, of which there are 53 groundtruth (manually labeled), and 39 ultrasound sequences in the test dataset, of which there are 85 groundtruth (manually labeled).

[0100] S2. Construct a knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation, the model includes a teacher network and a student network; the teacher network includes a feature encoder, a visual converter, a feature accumulation module and a prediction layer; the student network is a lightweight network based on the visual converter, including a projection layer, a visual converter and a prediction layer; the ultrasound sequence in the training set is respectively input into the teacher network and the student network for image analysis;

[0101] Specifically, the feature encoder includes a linear projection layer and a feature enhancement module; the feature enhancement module includes a convolutional network, a Monte Carlo attention module and a convergence transformer; the convolutional network includes a A convolutional layer and a Convolutional layer, the convergence transformer includes a 3×3 convolutional layer, a 1×1 convolutional layer, two visual converters and two 1×1 convolutional layers in sequence;

[0102] The ultrasound sequence in the training set is input into the Siamese network to extract the template image and search images , the template image and search images Split into patches and flatten them to get template sequences and search sequence ,in represents the resolution of each patch, Represents the number of patches in the template area, Represents the number of patches in the search area; the template sequence and search sequence The sequence is projected into the latent space through a linear projection layer Get the template token and search token , the formula is as follows:

[0103] ,

[0104] ,

[0105] in, The symbols represent the parameters of the linear projection layer. and Represent the learnable position embeddings of the template sequence and the search sequence respectively; the template token and search token After the feature enhancement module is used to extract features, the template image features are obtained. and search image features , is the dimension of the feature map.

[0106] Specifically, the visual converter includes a multi-head attention layer, a candidate elimination module and an optical flow auxiliary module; the candidate elimination module includes a sorter, a target position mask generator, a mask fusion, a normalization layer and a multi-layer perceptron, and the optical flow auxiliary module includes an optical flow extractor, an optical flow mask generator and a feature extractor;

[0107] The template image features and search image features After linear transformation of the multi-head attention layer, we get the query matrix Q, key matrix K and value matrix V, which are expressed as follows:

[0108] ,

[0109] ,

[0110] ,

[0111] in, Represent the weight matrices learned by the query matrix Q, key matrix K, and value matrix V, respectively. represents the template image features, Represents the search image features; for the query matrix , key matrix Sum Matrix Perform separate operations and process each head Dimensional submatrix, for each head Calculate the attention score, concatenate the outputs of all heads, and map them back to the original dimension through a linear layer to obtain the feature map processed by the multi-head self-attention layer , the formula is as follows:

[0112] ,

[0113] ,

[0114] ,

[0115] ,

[0116] ,

[0117] in , Represents the number of heads, represents the weight matrix of the multi-head attention layer, represents the linear layer function connecting all head attention scores, represents the operation of calculating the attention score, Represents the feature map processed by the multi-head self-attention layer; in the candidate elimination module, the bounding box mask is generated based on the bounding box information of the annotated file in the training set ultrasound sequence , the feature map processed by the multi-head self-attention layer The sorter scores the similarity between the template area and the search area. Before obtaining The target position mask generator then outputs the filtered mask , the formula is as follows:

[0118] ,

[0119] in, Indicates the reserved area. Before The similarity score of the regions is high. represents an adjustable parameter; the bounding box mask and filter mask The mask fuser performs pixel-by-pixel AND operations to obtain the tracking target mask. , the tracking target mask After processing by the normalization layer and the multi-layer perceptron, the position characteristics of the tracked target are obtained. ;

[0120] Extract the search image from the Siamese network The optical flow extractor extracts the optical flow and obtains the forward optical flow of adjacent video frames. and backward optical flow ; Forward optical flow and backward optical flow Converted into a forward optical flow image through the optical flow mask generator and the backward optical flow image ; The forward optical flow image and the backward optical flow image Passing the threshold Filter to remove areas where motion is not obvious, and then generate a forward optical flow mask for the tracked target and the backward optical flow mask of the tracked target ; Finally, the forward optical flow mask of the tracked target and the backward optical flow mask of the tracked target Perform pixel-by-pixel AND operation to obtain the optical flow mask of the tth frame , the optical flow mask of the tth frame and Search image for frame Connect them and send them to the feature extractor to obtain the motion features of the target , the formula is as follows:

[0121] ,

[0122] ,

[0123] ,

[0124] ,

[0125] ,

[0126] in, represents the optical flow extractor, Represents the operation of converting optical flow into optical flow image, represents the optical flow mask generator, represents pixel-by-pixel AND operation, Representative Frame search for motion features in images, is the image concatenation operation, Represents a feature extractor, which is a transformer encoder.

[0127] Specifically, the feature accumulation module includes an encoder, a feature storage library, and a decoder; the encoder includes a first residual network, a convolutional attention module, a second residual network, and a 1×1 convolutional layer; the decoder includes a 1×1 convolutional layer, a similarity calculation module, a weighted aggregation module, and a reshaping module;

[0128] Encoder stage: The first Frame search image to The motion characteristics of the target in each frame of the frame search image , location features and search image features After the first residual network is used for feature fusion, the fusion features of each frame of the search image are obtained. , the fusion feature After the convolutional attention module is used for feature enhancement, the enhanced features are obtained. , the fusion feature and Enhanced Features The hint value of each frame search image is obtained through the second residual network At the same time, each frame searches for image features After a 1×1 convolutional layer, the hint key of each frame search image is obtained , the prompt key and prompt value Form a pair of prompt key-value pairs , stored in the feature repository, which stores the Frame to The search image history features of the frame are updated using a first-in-first-out strategy. The shapes of , the formula is as follows:

[0129] ,

[0130] ,

[0131] ,

[0132] ,

[0133] ,

[0134] ,

[0135] ,

[0136] in, The size of the representative memory is an adjustable parameter. represents the first residual network, Representation based on fusion features Calculated channel attention weights, Representation based on fusion features The calculated spatial attention weights, represents the multi-layer perceptron processing function in the convolutional attention module, represents the convolution operation in the convolutional attention module, and Respectively represent the maximum pooling and average pooling along the spatial dimension, and Respectively represent the maximum pooling and average pooling in the channel dimension, represents the Sigmoid function, and denote the element-wise multiplication of channels and pixels, respectively, represents the second residual network, represents a 1×1 convolutional layer;

[0137] Decoder stage: Search image features of frame search images go through The convolutional layer gets the query hint key , calculate the query hint key through the similarity calculation module The similarity score with all the hint keys in the feature repository is used to calculate the attention score , the attention score The attention matrix Pass and memory prompt value Weighted aggregation to obtain aggregate features , the aggregated features The reshaping module is used to reshape the features to obtain the reshaped features ;

[0138] Specifically, the remodeling features and query prompt keys Connect to form the Prompt information feature map of frame search image ; The prompt information feature map Enter the prediction layer to get the classification score map of the target , local offset and the normalized bounding box size ,in, , , , and Respectively represent the height and width of the search image feature, is the search sequence The patch size is 2, and 2 represents the height and width dimensions; the position with the highest score in the classification score map is the target position , and then get the final target bounding box , the formula is as follows:

[0139]

[0140] in, Indicates the target location The horizontal axis of Indicates the target location The vertical coordinate of Represents the horizontal coordinate of the upper left corner of the prediction box, Represents the ordinate of the upper left corner of the prediction box, Indicates the width of the prediction box, Indicates the height of the prediction box.

[0141] Specifically, the visual converter of the student network includes a normalization layer, a multi-head attention layer, a normalization layer, and a multi-layer perceptron; the template image and search images After being processed by the mapping layer, the output features of the mapping layer are obtained, and the output features of the mapping layer are sent to the visual converter for feature extraction to obtain the prompt information feature map of the tracking target. , the prompt information feature map of the tracking target After being processed by the prediction layer, the predicted bounding box is obtained, and then the center position of the target is determined.

[0142] S3. Train the knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation through the total loss of the teacher network and the student network to obtain a trained model;

[0143] Specifically, the total loss of the teacher network includes classification loss and regression loss: the Gaussian Weighted Focal loss function is used for classification loss, and the formula is as follows:

[0144] ,

[0145] in, and denote the first hyperparameter and the second hyperparameter respectively, Represents the classification score map of the target, Represents the ground truth heat map generated using the Gaussian kernel function; regression loss includes bounding box prediction loss and bounding box overlap loss. The bounding box prediction loss is used The loss formula is as follows:

[0146]

[0147] in, represents the predicted bounding box, Represents the true target bounding box; the bounding box overlap loss uses the intersection-over-union (IoU) loss, and the formula is as follows:

[0148] ,

[0149] in, Represents the distance between the center point of the predicted bounding box and the center point of the true target bounding box, represents the diagonal length of the minimum outer frame that encloses these two boxes, represents the intersection of the predicted bounding box and the true target bounding box, represents the union of the predicted bounding box and the true target bounding box, represents the generalized intersection-union ratio; the total loss of the teacher network is:

[0150] ,

[0151] in, represents the weight of the weighted focal loss function, express The weight of the loss function, Represents the weight of the generalized intersection-over-union loss function.

[0152] Specifically, the total loss of the student network Including trailing loss and layered distillation loss : Tracking loss of the student network Same total loss as the teacher network:

[0153] ,

[0154] The hierarchical distillation loss includes similarity matrix knowledge distillation loss, output feature map knowledge distillation loss and response map knowledge distillation loss; the similarity matrix is ​​the weight matrix calculated by the multi-head self-attention layer in the student network and the teacher network, and the formula of the similarity matrix knowledge distillation loss is as follows:

[0155] ,

[0156] in, represents the i-th layer similarity matrix of the student network, represents the j-th layer similarity matrix of the teacher network; the output feature map is the prompt information feature map of the teacher network and the student network, and the formula of the output feature map knowledge distillation loss is as follows:

[0157] ,

[0158] in, represents the output feature map of the student network, Represents the output feature map of the teacher network; the response map of the teacher tracking network output For target positioning, the response map For distillation, the formula of the response graph knowledge distillation loss is expressed as follows:

[0159] ,

[0160] in, represents the temperature coefficient, and Represents the output response map of the student network and the teacher network for target positioning, divided by the temperature coefficient To normalize the response mapping of the teacher and student networks; the total loss of the student network is:

[0161] ,

[0162] in, Represents the knowledge distillation loss function of the similarity matrix The weight of Represents the knowledge distillation loss function of the output feature map The weight of Represents the response graph knowledge distillation loss function The weight of .

[0163] S4. Input the ultrasonic sequence in the test set into the trained model to obtain the final tracking result of the target.

[0164] Example 2

[0165] In this embodiment, an ablation experiment is performed on the method of the present invention on the data set obtained by the present invention. The four modules, namely the candidate elimination module, the optical flow auxiliary module, the feature accumulation module and the feature enhancement module, are combined with each other, and only three of them are selected each time to explore the influence of each module on the tracking results.

[0166] Table 1 Comparison of ablation experiment results

[0167]

[0168] As shown in Table 1, by comparing the average tracking error, standard deviation and TE95th (95th percentile of tracking error) of each group, it can be seen that each module has a certain effect on the tracking result, and the tracking performance decreases after removing any module. From Table 1, it can be seen that the average tracking error, standard deviation and TE95th of the model with these four modules combined are all minimized, indicating that the method of the present invention has good stability and high accuracy.

[0169] The CIL-05 ultrasound sequence of the CLUST2015 dataset was used to test the method of the present invention and three other advanced tracking methods (SiamRPN method, LCT method, KCF method). The SiamRPN method is a target tracking algorithm based on a Siamese network (SiameseNetwork), which combines a region proposal network (RPN) to achieve faster and more accurate target tracking; the LCT method decomposes long-term target tracking into translation estimation and scale change estimation of the target, and configures a detector to achieve re-detection; the KCF method uses a kernel correlation filter to achieve fast and accurate tracking of the target, and uses the circulant matrix technology in the frequency domain to convert the filtering operation into a dot multiplication operation, thereby speeding up the calculation speed. Figure 3 As shown, Figure 3 The horizontal axis is the frame number of the video image, and the vertical axis is the center coordinate value of the tracked blood vessel position. Figure 3 The vertical axis of the upper middle figure is the X coordinate value of the center position of the tracking blood vessel. Figure 3 The vertical axis of the lower middle figure is the Y coordinate value of the center position of the tracked blood vessel. Compared with the true value Ground truch, it can be seen from the tracking effect that the model of the method of the present invention is closer to the true value, has a strong accuracy, is relatively stable, and has good robustness.

[0170] Example 3

[0171] This embodiment provides a knowledge distillation ultrasonic sequence tracking device based on optical flow assistance and feature accumulation, including:

[0172] Data acquisition unit: used for acquiring ultrasound sequence data and dividing the ultrasound sequence data into a training set and a test set;

[0173] Model construction unit: used to construct a knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation, the model includes a teacher network and a student network; the teacher network includes a feature encoder, a visual converter, a feature accumulation module and a prediction layer; the student network is a lightweight network based on the visual converter, including a projection layer, a visual converter and a prediction layer; the ultrasound sequence in the training set is respectively input into the teacher network and the student network for image analysis;

[0174] Training unit: used to train the knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation through the total loss of the teacher network and the student network to obtain a trained model;

[0175] Target tracking unit: used to input the ultrasonic sequence in the test set into the trained model to obtain the final tracking result of the target.

[0176] Example 4

[0177] This embodiment provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the above-mentioned knowledge distillation ultrasonic sequence tracking method based on optical flow assistance and feature accumulation is executed;

[0178] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] Example 5

[0180] This embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned knowledge distillation ultrasound sequence tracking method based on optical flow assistance and feature accumulation is executed;

[0181] Storage media include: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media that can store programs.

[0182] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A knowledge distillation ultrasound sequence tracking method based on optical flow assistance and feature accumulation, characterized in that: The following steps are involved: S1. Obtain ultrasound sequence data, and divide the ultrasound sequence data into a training set and a test set; S2. Construct a knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation, the model includes a teacher network and a student network; the teacher network includes a feature encoder, a visual converter, a feature accumulation module and a prediction layer; the student network is a lightweight network based on the visual converter, including a projection layer, a visual converter and a prediction layer; the ultrasound sequence in the training set is respectively input into the teacher network and the student network for image analysis; S21. The specific operation of the feature encoder of the teacher network is: The feature encoder includes a linear projection layer and a feature enhancement module; The feature enhancement module includes a convolutional network, a Monte Carlo attention module and a convergence transformer; the convolutional network includes a A convolutional layer and a Convolutional layer, the convergence transformer includes a 3×3 convolutional layer, a 1×1 convolutional layer, two visual converters and two 1×1 convolutional layers in sequence; The ultrasound sequence in the training set is input into the Siamese network to extract the template image and search images , the template image and search images Split into patches and flatten them to get template sequences and search sequence ,in represents the resolution of each patch, Represents the number of patches in the template area, Represents the number of patches in the search area; the template sequence and search sequence The sequence is projected into the latent space through a linear projection layer Get the template token and search token , the formula is as follows: ， ， in, The symbols represent the parameters of the linear projection layer. and Represent the learnable position embeddings of the template sequence and the search sequence respectively; the template token and search token After the feature enhancement module is used to extract features, the template image features are obtained. and search image features , is the dimension of the feature map; S22. The specific operation of the visual converter of the teacher network is: The visual converter includes a multi-head attention layer, a candidate elimination module and an optical flow auxiliary module; the candidate elimination module includes a sorter, a target position mask generator, a mask fusion, a normalization layer and a multi-layer perceptron, and the optical flow auxiliary module includes an optical flow extractor, an optical flow mask generator and a feature extractor; The template image features and search image features After linear transformation of the multi-head attention layer, we get the query matrix Q, key matrix K and value matrix V, which are expressed as follows: ， ， ， in, Represent the weight matrices learned by the query matrix Q, key matrix K, and value matrix V, respectively. represents the template image features, Represents the search image features; for the query matrix , key matrix Sum Matrix Perform separate operations and process each head Dimensional submatrix, for each head Calculate the attention score, concatenate the outputs of all heads, and map them back to the original dimension through a linear layer to obtain the feature map processed by the multi-head self-attention layer , the formula is as follows: ， ， ， ， ， in , Represents the number of heads, represents the weight matrix of the multi-head attention layer, represents the linear layer function connecting all head attention scores, represents the operation of calculating the attention score, Represents the feature map processed by the multi-head self-attention layer; in the candidate elimination module, the bounding box mask is generated based on the bounding box information of the annotated file in the training set ultrasound sequence , the feature map processed by the multi-head self-attention layer The sorter scores the similarity between the template area and the search area. Before obtaining The target position mask generator then outputs the filtered mask , the formula is as follows: ， in, Indicates the reserved area. Before The similarity score of the regions is high. represents an adjustable parameter; the bounding box mask and filter mask The mask fuser performs pixel-by-pixel AND operations to obtain the tracking target mask. , the tracking target mask After processing by the normalization layer and the multi-layer perceptron, the position characteristics of the tracked target are obtained. ; Extract the search image from the Siamese network The optical flow extractor extracts the optical flow and obtains the forward optical flow of adjacent video frames. and backward optical flow ; Forward optical flow and backward optical flow Converted into a forward optical flow image through the optical flow mask generator and the backward optical flow image ; The forward optical flow image and the backward optical flow image Passing the threshold Filter to remove areas where motion is not obvious, and then generate a forward optical flow mask for the tracked target and the backward optical flow mask of the tracked target ; Finally, the forward optical flow mask of the tracked target and the backward optical flow mask of the tracked target Perform pixel-by-pixel AND operation to obtain the optical flow mask of the tth frame , the optical flow mask of the tth frame and Search image for frame Connect them and send them to the feature extractor to obtain the motion features of the target , the formula is as follows: ， ， ， ， ， in, represents the optical flow extractor, Represents the operation of converting optical flow into optical flow image, represents the optical flow mask generator, represents pixel-by-pixel AND operation, Representative Frame search for motion features in images, is the image concatenation operation, represents a feature extractor, which is a transformer encoder; S23. The specific operations of the feature accumulation module and prediction layer of the teacher network are: The feature accumulation module includes an encoder, a feature storage library, and a decoder; the encoder includes a first residual network, a convolutional attention module, a second residual network, and a 1×1 convolutional layer; the decoder includes a 1×1 convolutional layer, a similarity calculation module, a weighted aggregation module, and a reshaping module; Encoder stage: The first Frame search image to The motion characteristics of the target in each frame of the frame search image , location features and search image features After the first residual network is used for feature fusion, the fusion features of each frame of the search image are obtained. , the fusion feature After the convolutional attention module is used for feature enhancement, the enhanced features are obtained. , the fusion feature and Enhanced Features The hint value of each frame search image is obtained through the second residual network At the same time, each frame searches for image features After a 1×1 convolutional layer, the hint key of each frame search image is obtained , the prompt key and prompt value Form a pair of prompt key-value pairs , stored in the feature repository, which stores the Frame to The search image history features of the frame are updated using a first-in-first-out strategy. The shapes of , the formula is as follows: ， ， ， ， ， ， ， in, The size of the representative memory is an adjustable parameter. represents the first residual network, Representation based on fusion features Calculated channel attention weights, Representation based on fusion features The calculated spatial attention weights, represents the multi-layer perceptron processing function in the convolutional attention module, represents the convolution operation in the convolutional attention module, and Respectively represent the maximum pooling and average pooling along the spatial dimension, and Respectively represent the maximum pooling and average pooling in the channel dimension, represents the Sigmoid function, and denote the element-wise multiplication of channels and pixels, respectively, represents the second residual network, represents a 1×1 convolutional layer; Decoder stage: Search image features of frame search images go through The convolutional layer gets the query hint key , calculate the query hint key through the similarity calculation module The similarity score with all the hint keys in the feature repository is used to calculate the attention score , the attention score The attention matrix Pass and memory prompt value Weighted aggregation to obtain aggregate features , the aggregated features The reshaping module is used to reshape the features to obtain the reshaped features ; The remodeling features and query prompt keys Connect to form the Prompt information feature map of frame search image ; The prompt information feature map Enter the prediction layer to get the classification score map of the target , local offset and the normalized bounding box size ,in, , , , and Respectively represent the height and width of the search image feature, is the search sequence The patch size is 2, and 2 represents the height and width dimensions; the position with the highest score in the classification score map is the target position , and then get the final target bounding box , the formula is as follows: in, Indicates the target location The horizontal axis of Indicates the target location The vertical coordinate of Represents the horizontal coordinate of the upper left corner of the prediction box, Represents the ordinate of the upper left corner of the prediction box, Indicates the width of the prediction box, Indicates the height of the prediction box; S24. The specific operations of the student network are: The visual converter of the student network includes a normalization layer, a multi-head attention layer, a normalization layer, and a multi-layer perceptron; the template image and search images After being processed by the mapping layer, the output features of the mapping layer are obtained, and the output features of the mapping layer are sent to the visual converter for feature extraction to obtain the prompt information feature map of the tracking target. , the prompt information feature map of the tracking target The predicted bounding box is obtained after being processed by the prediction layer, and then the center position of the target is determined; S3. Train the knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation through the total loss of the teacher network and the student network to obtain a trained model; S4. Input the ultrasonic sequence in the test set into the trained model to obtain the final tracking result of the target.

2. The method for ultrasonic sequence tracking based on knowledge distillation of optical flow assistance and feature accumulation according to claim 1 is characterized in that: The total loss of the teacher network in step S3 includes classification loss and regression loss: In terms of classification loss, the Gaussian Weighted Focal loss function is used, and the formula is as follows: ， in, and denote the first hyperparameter and the second hyperparameter respectively, Represents the classification score map of the target, Represents the ground truth heat map generated using the Gaussian kernel function; The regression loss includes the bounding box prediction loss and the bounding box overlap loss. The bounding box prediction loss is calculated using The loss formula is as follows: ， in, represents the predicted bounding box, Represents the true target bounding box; the bounding box overlap loss uses the intersection-over-union (IoU) loss, and the formula is as follows: ， in, Represents the distance between the center point of the predicted bounding box and the center point of the true target bounding box, represents the diagonal length of the minimum outer frame that encloses these two boxes, represents the intersection of the predicted bounding box and the true target bounding box, represents the union of the predicted bounding box and the true target bounding box, represents the generalized intersection and ratio; The total loss of the teacher network is: ， in, represents the weight of the weighted focal loss function, express The weight of the loss function, Represents the weight of the generalized intersection-over-union loss function.

3. The method for ultrasonic sequence tracking based on knowledge distillation of optical flow assistance and feature accumulation according to claim 2 is characterized in that: The total loss of the student network in step S3 Including trailing loss and layered distillation loss : Tracking loss of the student network Same total loss as the teacher network: ， The hierarchical distillation loss includes similarity matrix knowledge distillation loss, output feature map knowledge distillation loss and response map knowledge distillation loss; the similarity matrix is ​​the weight matrix calculated by the multi-head self-attention layer in the student network and the teacher network, and the formula of the similarity matrix knowledge distillation loss is as follows: ， in, represents the i-th layer similarity matrix of the student network, represents the j-th layer similarity matrix of the teacher network; the output feature map is the prompt information feature map of the teacher network and the student network, and the formula of the output feature map knowledge distillation loss is as follows: ， in, represents the output feature map of the student network, Represents the output feature map of the teacher network; the response map of the teacher tracking network output For target positioning, the response map For distillation, the formula of the response graph knowledge distillation loss is expressed as follows: ， in, represents the temperature coefficient, and Represents the output response map of the student network and the teacher network for target positioning, divided by the temperature coefficient To normalize the response mapping of the teacher and student networks; The total loss of the student network is: ， in, Represents the knowledge distillation loss function of the similarity matrix The weight of Represents the knowledge distillation loss function of the output feature map The weight of Represents the response graph knowledge distillation loss function The weight of .

4. A knowledge distillation ultrasonic sequence tracking device based on optical flow assistance and feature accumulation, performing the steps of the knowledge distillation ultrasonic sequence tracking method based on optical flow assistance and feature accumulation as claimed in claim 1, characterized in that: include: Data acquisition unit: used for acquiring ultrasound sequence data and dividing the ultrasound sequence data into a training set and a test set; Model construction unit: used to construct a knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation, the model includes a teacher network and a student network; the teacher network includes a feature encoder, a visual converter, a feature accumulation module and a prediction layer; the student network is a lightweight network based on the visual converter, including a projection layer, a visual converter and a prediction layer; the ultrasound sequence in the training set is respectively input into the teacher network and the student network for image analysis; Training unit: used to train the knowledge distillation ultrasound sequence tracking model based on optical flow assistance and feature accumulation through the total loss of the teacher network and the student network to obtain a trained model; Target tracking unit: used to input the ultrasonic sequence in the test set into the trained model to obtain the final tracking result of the target.

5. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the knowledge distillation ultrasonic sequence tracking method based on optical flow assistance and feature accumulation as described in any one of claims 1 to 3 is performed.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the knowledge distillation ultrasound sequence tracking method based on optical flow assistance and feature accumulation as described in any one of claims 1 to 3.

Images