Training method of ultrasonic video basic model based on edge mask and time sequence difference

Abstract

The invention discloses an edge mask and time sequence difference-based ultrasonic video basic model training method, which comprises the following steps of: reducing speckle noise in an ultrasonic image through wavelet transform, masking edge regions of two front frames and two rear frames by using a canny algorithm, completely masking the two middle frames, and performing time sequence difference on the two middle frames by using a motion difference attention module to obtain a time sequence difference-based ultrasonic video basic model. Initial feature reconstruction is carried out on the first two frames and the last two frames after masking; and finally, the features are sent into a decoder together to reconstruct a video sequence and calculate loss, and a final training weight is obtained through multiple iterative training.

Description

Technical Field

[0001] The present invention relates to computer vision processing and deep learning technology, and in particular to a training method for an ultrasound video basic model based on edge masking and temporal difference. Background Art

[0002] Medical ultrasound imaging technology, as a non-invasive, radiation-free examination method, plays a core role in clinical diagnosis and has been widely used in multiple medical fields. However, traditional medical ultrasound imaging technology faces several challenges, such as the quality of ultrasound videos being significantly affected by operator skill, limited image resolution within ultrasound videos, and the high degree of subjectivity in ultrasound video interpretation. In recent years, the rapid development of artificial intelligence technology, particularly the breakthroughs in deep learning algorithms for image and video recognition and processing, has revolutionized the field of medical ultrasound imaging. Deep learning, a method that mimics the human brain's ability to analyze and learn data, can learn features from ultrasound videos using large amounts of training data and can be used to automatically identify, segment, and classify medical ultrasound videos.

[0003] With the development of deep learning, downstream tasks such as the automatic recognition, segmentation, and classification of medical ultrasound videos have made significant progress. However, in deep learning, most research begins with pre-training on natural videos, then migrates to downstream tasks such as medical ultrasound videos through methods such as transfer learning and contrastive learning. Due to the domain gap between natural videos and medical ultrasound videos, simple transfer learning often yields poor results. With the development of foundational models, this problem is expected to be resolved.

[0004] The Ultrasound Video Base Model is a deep learning model specifically designed for processing and analyzing ultrasound video data. Developed using self-supervised pre-training on a large-scale, unlabeled dataset, the model extracts valuable information from the video data. This information can be generalized to various downstream tasks, significantly improving model development efficiency. Compared to pre-training on natural videos and then transferring the model to downstream tasks such as medical ultrasound videos through transfer learning or contrastive learning, applying the weights pre-trained using the general Ultrasound Video Base Model to downstream ultrasound tasks can significantly reduce the gap between fields. With the continuous advancement of deep learning technology, the Ultrasound Video Base Model is expected to become a valuable tool in the field of ultrasound video analysis.

[0005] However, ultrasound video data is characterized by high noise, temporal variations, and unstable image quality, making it difficult for traditional image processing methods to effectively handle it. Therefore, it is particularly important to develop a foundational ultrasound video model that can address these challenges. This foundational ultrasound video model utilizes advanced deep learning techniques, including convolutional neural networks (CNNs) and Transformers. These techniques can effectively extract the spatiotemporal features of ultrasound video data and leverage the powerful feature extraction and generalization capabilities of deep learning models to improve the model's performance when processing complex data.

[0006] The ultrasound video basic model faces a series of challenges and difficulties during its design and implementation, which require innovative technologies and methods to address. Noise and artifacts: Ultrasound video data often contains a large amount of noise and artifacts, which can interfere with the model's learning and inference processes. Temporal variation: Different frames in the same video sequence may experience significant temporal variation, requiring the model to be able to exploit this variation and extract stable features. Summary of the Invention

[0007] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide a training method for an ultrasound video basic model based on edge masking and temporal difference. The present invention combines wavelet transform and edge noise detection algorithm to suppress noise in ultrasound video, and uses temporal difference attention to generate high-quality features to better achieve reconstruction of video sequences.

[0008] Technical solution: A training method for an ultrasound video basic model based on edge masking and temporal difference of the present invention comprises the following steps:

[0009] Step (1): input a video sequence from the ultrasound video dataset, and extract six frames from the video sequence, which are denoted as F t-2 ,F t-1 ,F t～1 ,F t～2 ,F t+1 ,F t+2 , unify the size of these six frames of images to C×H×W, where C is the number of channels, H is the height of the image frame, and W is the width of the image frame;

[0010] Step (2): Apply wavelet transform to the six frames of images obtained by step (1) to suppress the noise. The obtained image frames are recorded as

[0011] Step (3): For the six image frames obtained in step (2), the first two frames and the last two frames are subjected to edge masking by using the Canny algorithm to find the boundary sensitive parts. The image frames after edge masking are recorded as The two middle frames are fully masked to obtain the corresponding image frames. This fully masking of only the two middle frames requires the model to not only understand the temporal relationships of the video but also accurately infer and reconstruct the masked content in the absence of direct visual information. This approach effectively enables the model to deeply explore the temporal and spatial correlations in the video sequence, thereby demonstrating higher performance in video understanding and analysis.

[0012] The specific method of the above edge mask is: first, Use the canny algorithm to perform edge masking and obtain a masked image F t-2 ′ ; Then, in the image frame and Mapped with F t-2 ′ The same mask area, then the image frame and The unmasked areas in the image are input into the feature encoder for feature extraction. The feature encoder adopts the Vision Transformer encoder. The extracted features are recorded as Where i represents the i-th unmasked pixel block, i∈0,1,2…;

[0013] Step (4), based on the fully masked image frame, the features obtained in step (3) are The initial feature reconstruction is performed through the temporal difference module. The specific process is as follows:

[0014] First, respectively and and and Perform differential calculation to obtain motion features, that is, Then, the obtained motion features are subjected to average pooling, maximum pooling and convolution operations, and then and and Connect them together and perform residual operation to obtain the initial features of the two middle frames, that is,

[0015] Step (5) Output the features of step (4) And the features output from step (4) Reconstructing the image frame through a feature decoder, wherein the feature decoder adopts a VisionTransformer decoder;

[0016] Step (6) calculates the self-supervised loss of reconstruction and obtains the final reconstruction result and weight through multiple iterative training.

[0017] Furthermore, the specific method of edge masking in step (3) is:

[0018] First, a Gaussian filter is used to smooth the corresponding image to reduce noise, as shown in formula (1):

[0019]

[0020] Where D represents the Gaussian function, σ represents the standard deviation of the Gaussian function, and x, y are the coordinates of the pixel relative to the center of the filter;

[0021] Then, the Sobel operator is used to calculate the gradients in the horizontal and vertical directions, and the gradient magnitude G and direction θ are further obtained. The gradient calculation formulas are as follows:

[0022]

[0023] θ(x,y)=arctan2(d y (x,y),d x (x,y)) (3)

[0024] d x (x,y) is the gradient in the horizontal direction, d y (x,y) is the gradient in the vertical direction;

[0025] Next, the obtained gradient magnitude is compared with the gradient magnitudes of its adjacent pixels along the gradient direction, and only the local maximum gradient value is retained, as shown in formula (4):

[0026]

[0027] Finally, the edge mask is determined by two thresholds, as shown in formula (5):

[0028]

[0029] Furthermore, the specific calculation method of the timing difference module in step (4) is shown in formula (6) and formula (7):

[0030]

[0031] Where ⊕ represents the concatenation operation, pool represents the average pooling and maximum pooling operations, and θ represents the convolution operation.

[0032] Furthermore, the self-supervised loss function and noise loss function reconstructed in step (6) are defined as follows:

[0033]

[0034] Where, N in formula (8) is the number of pixels, is the j-th pixel value of the reconstructed t-th frame image, is the j-th pixel value of the original t-th frame image, β in formula (9) represents the local window of the image after image reconstruction, and represents the average value of pixels within the window, and Represents the variance of pixels within the window, σ αβ represents the covariance, γ1 and γ2 are small constants;

[0035] The final training goal is as follows:

[0036] L=L S-sup +λL Noise (α,β) (10)

[0037] Here, λ is the equilibrium parameter.

[0038] Beneficial Effects: This invention uses the temporal information captured by the previous and next frames to calculate the difference between them, obtaining motion information between the previous and next frames of the video. This allows the network to focus more on the parts with motion changes, and uses edge masks to make the construction of image frames more challenging, thereby enhancing the network's expressive performance to a certain extent. Compared with the existing technology, this invention has the following advantages:

[0039] (1) The present invention realizes the noise reduction processing of ultrasound through wavelet transform to reduce the influence of noise.

[0040] (2) The present invention uses edge masking to make the masked area more inclined to the edge area, making video reconstruction a more challenging and meaningful self-supervised task.

[0041] (3) The temporal difference module of the present invention fully utilizes the temporal information of the previous and next frames, making the modeling of image frames more challenging. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0043] Figure 2 Schematic diagram of the model framework of the present invention;

[0044] Figure 3 This is a rendering of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0046] like Figure 1 and Figure 2As shown, a training method of an ultrasound video basic model based on edge mask and temporal difference of the present invention includes the following steps:

[0047] Step (1): input a video sequence from the ultrasound video dataset, and extract six frames from the video sequence, which are denoted as F t-2 ,F t-1 ,F t～1 ,F t～2 ,F t+1 ,F t+2 , unify the size of these six frames of images to C×H×W, where C is the number of channels, H is the height of the image frame, and W is the width of the image frame;

[0048] Step (2): Apply wavelet transform to the six frames of images obtained by step (1) to suppress the noise. The obtained image frames are recorded as

[0049] Step (3): For the six image frames obtained in step (2), the first two frames and the last two frames are subjected to edge masking by using the Canny algorithm to find the boundary sensitive parts. The image frames after edge masking are recorded as Use the full mask on the two frames in the middle to get the corresponding image frames; then use the full mask on the two frames to get the corresponding image frames. and The unmasked regions are fed into the feature encoder for feature extraction. Subsequently, these six image frames are fed into the model in the format of N*C*H*W, where N represents the number of input image frames, and C, H, and W represent the number of image channels, height, and width, respectively. Although the two middle frames are completely masked and lack any feature information, they are not removed individually to ensure consistency between model input and output. Ultimately, the model reconstructs N*C*H*W image frames of the same format. This processing approach not only maintains the integrity of the data stream but also simplifies the operational process.

[0050] The specific method of the above edge mask is: first, Use the canny algorithm to perform edge masking and obtain a masked image F t-2 ′ ; Then, in the image frame and Mapped with F t-2 ′ The same mask area, then the image frame and The unmasked areas in the image are input into the feature encoder for feature extraction; the extracted features are recorded as Where i represents the i-th unmasked pixel block, i∈0,1,2…; in this step, only the image frame Ft-3 Using the Canny algorithm for edge masking, the other three frames only need to map the same masked area, which can prevent the current frame from simply learning features at the same location from adjacent frames. This operation is very beneficial for the reconstruction of image frames (especially key features) and also reduces the amount of calculation to a certain extent.

[0051] Step (4): the features obtained in step (3) Through the temporal difference module, the initial features of the completely masked image features of the two middle frames are reconstructed. The specific process is as follows:

[0052] Step (4.1), respectively and and and Perform differential calculation to obtain motion features, that is,

[0053] Step (4.2) performs average pooling, maximum pooling and convolution operations on the obtained motion features, and then and and Connect them together and perform residual operation to obtain the initial features of the two middle frames, that is,

[0054] The calculation formulas are shown in formulas (6) and (7):

[0055]

[0056] in represents the concatenation operation, pool represents the average pooling and maximum pooling operations, and θ represents the convolution operation;

[0057] Step (5) output the features of step (4) And the features output from step (4) Reconstructing the image frame through a feature decoder, wherein the feature decoder adopts a VisionTransformer decoder;

[0058] Step (6) calculates the self-supervised loss of reconstruction and obtains the final reconstruction result and weight through multiple iterative training.

[0059] The specific method of edge masking in step (3) of this embodiment is:

[0060] First, a Gaussian filter is used to smooth the corresponding image to reduce noise, as shown in formula (1):

[0061]

[0062] Where D represents the Gaussian function, σ represents the standard deviation of the Gaussian function, and x, y are the coordinates of the pixel relative to the center of the filter;

[0063] Then, the Sobel operator is used to calculate the gradients in the horizontal and vertical directions, and the gradient magnitude Grd and direction θ are further obtained. The gradient calculation formulas are as follows:

[0064]

[0065] θ(x,y)=arctan2(d y (x,y),d x (x,y)) (3)

[0066] d x (x,y) is the gradient in the horizontal direction, d y (x,y) is the gradient in the vertical direction;

[0067] Next, the obtained gradient magnitude is compared with the gradient magnitudes of its adjacent pixels along the gradient direction, and only the local maximum gradient value is retained, as shown in formula (4):

[0068]

[0069] Finally, the edge mask is determined by two thresholds, as shown in formula (5):

[0070]

[0071] The self-supervised loss function and noise loss function reconstructed in step (6) of this embodiment are defined as follows:

[0072]

[0073] Where, N in formula (8) is the number of pixels, is the j-th pixel value of the reconstructed t-th frame image, is the j-th pixel value of the original t-th frame image, β in formula (9) represents the local window of the image after image reconstruction, and represents the average value of pixels within the window, and Represents the variance of pixels within the window, σ αβ represents the covariance, γ1 and γ2 are small constants;

[0074] The final training goal is as follows:

[0075] L=L S-sup +λL Noise (α,β) (10)

[0076] Here, λ is the equilibrium parameter.

[0077] like Figure 3 As shown, this embodiment extracts six image frames (such as Figure 3 The first row of the image), the effect after wavelet transformation is shown in the second row, the middle two frames are completely masked and the first two frames / last two frames are edge masked, as shown in Figure 3 As shown in the third row of [ ], this facilitates in-depth exploration of temporal and spatial correlations in ultrasound video sequences. This paper fully masks the two middle frames because, without full masking, the images themselves provide useful feature information during feature reconstruction, rather than relying solely on the feature information of the four preceding and following frames. Full masking further emphasizes the role of temporal information.

[0078] Depend on Figure 3 It can be seen that after wavelet transform and edge masking, the noise in the image is greatly reduced, which is conducive to the subsequent mining of temporal and spatial correlations in ultrasound video sequences.

Claims

1. A training method for an ultrasound video basic model based on edge masking and temporal difference, characterized in that: The following steps are involved: Step (1): input a video sequence from the ultrasound video dataset, and extract six frames from the video sequence, which are denoted as F t-3 ,F t-1 ,F t～1 ,F t～2 ,F t+1 ,F t+2 , unify the size of these six frames of images to C×H×W, where C is the number of channels, H is the height of the image frame, and W is the width of the image frame; Step (2): Apply wavelet transform to the six frames of images obtained by step (1) to suppress the noise. The obtained image frames are recorded as Step (3): For the six image frames obtained in step (2), the first two frames and the last two frames are subjected to edge masking by using the Canny algorithm to find the boundary sensitive parts. The image frames after edge masking are recorded as Use full mask on the two middle frames to get the corresponding image frames; The specific method of the above edge mask is: first, Use the canny algorithm to perform edge masking and obtain a masked image F t-2 ′ ; Then, in the image frame and Mapped with F t-2 'The same mask area, then the image frame and The unmasked areas in the image are input into the feature encoder for feature extraction; the extracted features are recorded as Where i represents the i-th unmasked pixel block, i∈0,1,2…; Step (4), based on the fully masked image frame, the features obtained in step (3) are The initial feature reconstruction is performed through the temporal difference module. The specific process is as follows: First, respectively and and and Perform differential calculation to obtain motion characteristics, namely: Then the motion features Perform average pooling, maximum pooling, and convolution operations, and then and Connect together and and Connect them together, and then perform residual operation on the two connected results to obtain the initial features of the two middle frames, that is, and Step (5) Output the features of step (4) And the features output from step (3) Reconstruct the image frame through the feature decoder; Step (6) calculates the self-supervised loss of reconstruction and obtains the final reconstruction result and weight through multiple iterative training.

2. The method for training an ultrasound video basic model based on edge mask and temporal difference according to claim 1, wherein: The specific method of edge masking in step (3) is: First, a Gaussian filter is used to smooth the corresponding image to reduce noise, as shown in formula (1): Where D represents the Gaussian function, σ represents the standard deviation of the Gaussian function, and x, y are the coordinates of the pixel relative to the center of the filter; Then, the Sobel operator is used to calculate the gradients in the horizontal and vertical directions, and the gradient magnitude Grd and direction θ are further obtained. The gradient calculation formulas are as follows: d x (x,y) is the gradient in the horizontal direction, d y (x,y) is the gradient in the vertical direction; Next, the obtained gradient magnitude is compared with the gradient magnitudes of its adjacent pixels along the gradient direction, and only the local maximum gradient value is retained, as shown in formula (4): Finally, the edge mask is determined by two thresholds, as shown in formula (5):

3. The method for training an ultrasound video basic model based on edge mask and temporal difference according to claim 1, wherein: The calculation formulas of the timing difference module in step (4) are shown in formulas (6) and (7): in, represents the concatenation operation, pool represents the average pooling and maximum pooling operations, and θ represents the convolution operation.

4. The method for training an ultrasound video basic model based on edge mask and temporal difference according to claim 1, wherein: The self-supervised loss function and noise loss function reconstructed in step (6) are defined as follows: In formula (8), N is the number of pixels, is the j-th pixel value of the reconstructed t-th frame image, is the j-th pixel value of the original t-th frame image; In formula (9), α represents the local window of the image before wavelet transformation, and β represents the local window of the image after image reconstruction. and represents the average value of pixels within the window, and Represents the variance of pixels within the window, σ αβ represents the covariance, γ1 and γ2 refer to small constants; The final training goal is as follows: L=L S-s +λL Noise (a,b) (10) Here, λ is the equilibrium parameter.

Images