A method, system, device and medium for target enhancement processing of ultrasound images

Abstract

This invention discloses a method, system, device, and medium for ultrasound image target enhancement processing. The method includes: acquiring N consecutive frames of ultrasound images containing a target object and their corresponding N consecutive motion information; acquiring N target semantic features corresponding to the N consecutive frames of ultrasound images and multi-scale intermediate features corresponding to the Nth frame of ultrasound images; acquiring N-1 target mask features corresponding to the first N-1 frames of ultrasound images; analyzing the correlation between the N target semantic features to obtain global matching cost and attention features; acquiring the motion temporal features reflected by the N consecutive motion information; fusing the global matching cost, attention features, N-1 target mask features, and motion temporal features to obtain the target mask features corresponding to the Nth frame of ultrasound images; and analyzing the target mask features and multi-scale intermediate features corresponding to the Nth frame of ultrasound images to obtain a target mask image. This invention can achieve accurate tracking and positioning of target objects.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method, system, device, and medium for enhancing targets in ultrasound images. Background Technology

[0002] In modern clinical medicine, ultrasound imaging is commonly used to track, locate, and detect relevant targets of interest (such as lesions, blood vessels, and tumor cells) in specific areas of a patient. However, due to the disadvantages of ultrasound imaging, such as low resolution and poor signal-to-noise ratio, the relevant targets of interest are not easily visible in ultrasound images. In particular, when medical staff hold the ultrasound probe for hands-on testing, the probe may shake directly, or the ultrasound probe may shake indirectly due to the sudden movement of the patient during the testing process. These factors can easily lead to a deterioration in the imaging quality of the ultrasound image, making it difficult to accurately track and distinguish the relevant targets of interest. Summary of the Invention

[0003] This invention provides a method, system, device, and medium for enhancing ultrasound image targets, in order to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0004] In a first aspect, a method for target enhancement processing of ultrasound images is provided, the method comprising:

[0005] An ultrasonic probe is used to acquire N consecutive frames of ultrasonic images containing the target object, and an inertial element mounted on the ultrasonic probe is used to acquire the corresponding N consecutive motion information.

[0006] The N consecutive frames of ultrasound images are parsed using a semantic encoder to obtain the corresponding N target semantic features and the multi-scale intermediate features corresponding to the Nth frame of ultrasound image;

[0007] The first N-1 frames of ultrasound images are preprocessed and then input into a mask encoder for parsing to obtain the corresponding N-1 target mask features.

[0008] The correlations among the N target semantic features are analyzed to obtain the global matching cost and attention features;

[0009] The motion temporal features are obtained by analyzing the N consecutive motion information using an LSTM model.

[0010] The global matching cost, the attention feature, the N-1 target mask features, and the motion temporal features are fused to obtain the target mask features corresponding to the Nth frame ultrasound image;

[0011] The target mask features and multi-scale intermediate features corresponding to the Nth frame of ultrasound image are analyzed using a decoder to obtain the corresponding target mask image and visualize it.

[0012] Furthermore, for each ultrasound image in the first N-1 frames of ultrasound images, preprocessing includes:

[0013] Obtain the mask image corresponding to each frame of ultrasound image, and then overlay the channels of each frame of ultrasound image with the mask image.

[0014] Furthermore, the process of analyzing the correlations among the N target semantic features to obtain the global matching cost and attention features includes:

[0015] Based on the L2 norm, the relevance matching calculation is performed on the Nth target semantic feature and the first N-2 target semantic features to obtain the global matching cost;

[0016] Channel stacking, two-dimensional matching, and convolution operations are sequentially performed on the (N-1)th target semantic feature and the Nth target semantic feature to obtain the local matching cost;

[0017] Perform disparity prediction on the local matching cost to obtain the short-time motion field between the (N-1)th frame ultrasound image and the Nth frame ultrasound image;

[0018] Perform multiple convolution operations on the short-time motion field to obtain short-time motion features;

[0019] The short-term motion features are used to reweight the Nth target semantic feature, and then the reweighted result is added to the Nth target semantic feature to obtain the attention feature.

[0020] Further, the fusion processing of the global matching cost, the attention features, the N-1 target mask features, and the motion temporal features to obtain the target mask features corresponding to the Nth frame ultrasound image includes:

[0021] The global matching cost is normalized, and then the normalization result is multiplied by the first N-2 target mask features to obtain the global matching features;

[0022] Multiply the attention feature by the (N-1)th target mask feature to obtain the local matching feature;

[0023] The global matching features, the local matching features, and the motion temporal features are superimposed and convolved to obtain the target mask features corresponding to the Nth frame ultrasound image.

[0024] Furthermore, the semantic encoder includes a first input layer, a first convolutional layer, a first block network layer, a second block network layer, a third block network layer, a fourth block network layer, and a first output layer connected in sequence.

[0025] When the semantic encoder is used to parse the Nth frame of ultrasound image, the first intermediate feature output by the second Block network layer, the second intermediate feature output by the third Block network layer, and the third intermediate feature output by the fourth Block network layer are collectively referred to as the multi-scale intermediate features.

[0026] Furthermore, the decoder includes a second input layer, a first upsampling layer, a first feature fusion layer, a second upsampling layer, a second feature fusion layer, a third upsampling layer, a third feature fusion layer, a fourth upsampling layer, and a second output layer;

[0027] The first upsampling layer is used to upsample the target mask features corresponding to the Nth frame ultrasound image provided by the second input layer, and then the first feature fusion layer is used to fuse the upsampling result with the third intermediate feature to obtain the first mask feature map with enhanced target edge.

[0028] The first mask feature map is upsampled using the second upsampling layer, and then the upsampling result is fused with the second intermediate feature using the second feature fusion layer to obtain a second mask feature map with enhanced target edges.

[0029] The second mask feature map is upsampled using the third upsampling layer, and then the upsampling result is fused with the first intermediate feature using the third feature fusion layer to obtain a third mask feature map with enhanced target edges.

[0030] The third mask feature map is upsampled using the fourth upsampling layer to obtain the final target mask image, which is then output via the second output layer.

[0031] Furthermore, the internal operation of the first feature fusion layer includes:

[0032] The third intermediate feature is convolved, and the convolution result is added to the upsampling result to obtain the initial mask feature map;

[0033] Max pooling is performed on the third intermediate feature to obtain the first pooling result;

[0034] The third intermediate feature is subjected to average pooling to obtain the second pooling result;

[0035] The first pooling result and the second pooling result are sequentially subjected to channel stacking, convolution, normalization and inversion operations. The output result is then multiplied by the initial mask feature map and then subjected to convolution to obtain the first mask feature map.

[0036] Secondly, an ultrasound image target enhancement processing system is provided, the system comprising:

[0037] The data acquisition module is used to acquire N consecutive frames of ultrasound images containing the target object using an ultrasound probe, and simultaneously acquire corresponding N consecutive motion information using an inertial element mounted on the ultrasound probe.

[0038] The semantic feature parsing module is used to parse the N consecutive frames of ultrasound images using a semantic encoder to obtain the corresponding N target semantic features and the multi-scale intermediate features corresponding to the Nth frame of ultrasound image.

[0039] The mask feature parsing module is used to input the first N-1 frames of ultrasound images after preprocessing into the mask encoder for parsing, and obtain the corresponding N-1 target mask features;

[0040] The correlation analysis module is used to analyze the correlation between the N target semantic features to obtain the global matching cost and attention features;

[0041] The motion feature parsing module is used to parse the N consecutive motion information using an LSTM model to obtain motion temporal features;

[0042] A multi-feature fusion module is used to fuse the global matching cost, the attention feature, the N-1 target mask features and the motion temporal features to obtain the target mask features corresponding to the Nth frame ultrasound image;

[0043] The decoding processing module is used to analyze the target mask features and multi-scale intermediate features corresponding to the Nth frame ultrasound image using a decoder, so as to obtain the corresponding target mask image and visualize it.

[0044] Thirdly, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the ultrasound image target enhancement processing method as described in the first aspect.

[0045] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the ultrasound image target enhancement processing method as described in the first aspect.

[0046] This invention has at least the following beneficial effects: By performing correlation matching analysis on the N target semantic features corresponding to N consecutive frames of ultrasound images from both global and local perspectives, and then using the correlation information obtained from the analysis and the motion temporal features reflected by the ultrasound probe during the acquisition of N consecutive frames of ultrasound images to fuse and learn the N-1 target mask features corresponding to the first N-1 frames of ultrasound images to generate the target mask features corresponding to the Nth frame of ultrasound images, the difficulty in target tracking caused by ultrasound probe jitter can be overcome. Finally, by combining the multi-scale intermediate features corresponding to the Nth frame of ultrasound images, a decoder is used to perform multiple target edge enhancement processing on the target mask features corresponding to the Nth frame of ultrasound images, which can make the edge region of the target object more clearly highlighted in the final output target mask image, thereby achieving accurate tracking and positioning of the target object to meet the actual needs of clinical medicine. Attached Figure Description

[0047] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0048] Figure 1 This is a flowchart illustrating an ultrasound image target enhancement processing method according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the composition of an ultrasound image target enhancement processing system according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the hardware structure of the computer device in an embodiment of this disclosure. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," "third," "fourth," etc., used in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed and are inherent to these processes, methods, products, or apparatuses.

[0053] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an ultrasound image target enhancement processing method provided in an embodiment of the present invention. The method includes the following:

[0054] Step S110: Use an ultrasonic probe to acquire N consecutive frames of ultrasonic images containing the target object, and simultaneously use an inertial element installed on the ultrasonic probe to acquire the corresponding N consecutive motion information, where N is a positive integer and N is greater than 1;

[0055] Step S120: Use a semantic encoder to parse the N consecutive frames of ultrasound images to obtain the corresponding N target semantic features and the multi-scale intermediate features corresponding to the Nth frame of ultrasound image;

[0056] Step S130: After preprocessing, the first N-1 frames of ultrasound images are input into the mask encoder for parsing to obtain the corresponding N-1 target mask features;

[0057] Step S140: Analyze the correlation between the N target semantic features to obtain the global matching cost and attention features;

[0058] Step S150: Use the LSTM model to analyze the N consecutive motion information to obtain motion temporal features;

[0059] Step S160: The global matching cost, the attention feature, the N-1 target mask features and the motion temporal features are fused to obtain the target mask features corresponding to the Nth frame ultrasound image;

[0060] Step S170: Use a decoder to analyze the target mask features and multi-scale intermediate features corresponding to the Nth frame ultrasound image to obtain the corresponding target mask image and visualize it.

[0061] In this embodiment of the invention, the N consecutive ultrasound images mentioned in step S110 are denoted as {P1, P2, ..., P...} in chronological order of acquisition time. N-2 ,P N-1 ,P N}, and the N consecutive motion information data are denoted as {S1, S2, ..., S} in chronological order of acquisition time. N-2 ,S N-1 ,S N The N consecutive ultrasound images and the N consecutive motion information have a corresponding time synchronization relationship, that is: the first ultrasound image P1 is acquired at the same time as the first motion information S1, the second ultrasound image P2 is acquired at the same time as the second motion information S2, and so on.

[0062] In this embodiment of the invention, the semantic encoder mentioned in step S120 is specifically composed of a first input layer, a first convolutional layer, a first block network layer, a second block network layer, a third block network layer, a fourth block network layer, and a first output layer connected in sequence. The first convolutional layer is responsible for extracting local features in the image and preserving the spatial structure information of the image. The four block network layers adopt a self-attention mechanism to capture the long-distance dependencies of the image and perform high-level abstraction and integration of the extracted features.

[0063] More specifically, the first convolutional layer includes a first convolutional sub-layer, a second convolutional sub-layer, and a third convolutional sub-layer connected in sequence. The first convolutional sub-layer uses a 3×3 convolutional kernel with a stride of 2. The second and third convolutional sub-layers both use a 3×3 convolutional kernel with a stride of 1. In practical applications, the data output by the first convolutional sub-layer should be processed by the GELU (Gaussian Error Linear Units) activation function and BN (Batch Normalization) before being input to the second convolutional sub-layer. The data output by the second convolutional sub-layer should be processed by the GELU activation function and BN before being input to the third convolutional sub-layer. The data output by the third convolutional sub-layer should be processed by the GELU activation function and BN before being input to the first Block network layer.

[0064] More specifically, the above four Block network layers are all composed of a Patch Embedding layer and multiple Block structures connected sequentially. The Patch Embedding layer uses a 2×2 convolutional kernel with a stride of 2. This invention does not limit the number of Block structures deployed inside each Block network layer, nor does it require that the above four Block network layers should have the same number of Block structures. Adjustments can be made according to actual application requirements.

[0065] Each Block structure consists of an LPU (Local Perception Unit) layer, a Depthwise convolutional layer, a first adder, a first LayerNorm layer, an LMHSA (Lightweight Multi-Head Self-Attention Mechanism) layer, a second adder, a second LayerNorm layer, and an FNN (Feed-Forward) layer. The network consists of a feedforward neural network (FNN) layer and a third adder. The Depthwise convolutional layer uses a 3×3 kernel with a stride of 1, and the FNN layer uses an inverse residual structure. In practical applications, the Depthwise convolutional layer processes the data output by the LPU layer. The first adder adds the data output by the Depthwise convolutional layer and the data output by the LPU layer. The first LayerNorm layer processes the sum of the first adder's output. The LMHSA layer processes the data output by the first LayerNorm layer. The second adder adds the sum of the first adder's output and the data output by the LMHSA layer. The second LayerNorm layer processes the sum of the second adder's output. The FNN layer processes the data output by the second LayerNorm layer. The third adder adds the sum of the second adder's output and the data output by the FNN layer.

[0066] In this embodiment of the invention, step S120 is implemented as follows: the first frame ultrasound image P1 is input to the semantic encoder for parsing to obtain the first target semantic feature F1; the second frame ultrasound image P2 is input to the semantic encoder for parsing to obtain the second target semantic feature F2; and so on, to obtain N consecutive frames of ultrasound images {P1, P2, ..., P...} N-2 ,P N-1 ,P NThe corresponding N target semantic features are {F1, F2, ..., F} N-2 ,F N-1 ,F N Furthermore, when the Nth frame ultrasound image P... N When the data is input to the semantic encoder for parsing, it is necessary to additionally obtain the data output by the second Block network layer and record it as the first intermediate feature C1, the data output by the third Block network layer and record it as the second intermediate feature C2, and the data output by the fourth Block network layer and record it as the third intermediate feature C3, and then output the Nth frame ultrasound image P. N The corresponding multi-scale intermediate features are {C1, C2, C3}.

[0067] In this embodiment of the invention, the mask encoder mentioned in step S130 above adopts a ResNet18 network based on a lightweight CNN (Convolutional Neural Network) architecture. Specifically, it is composed of a third input layer, a second convolutional layer, a residual layer, a global average pooling layer, a fully connected layer, and a third output layer connected in sequence. The second convolutional layer is composed of four convolutional sub-layers connected in sequence, each convolutional sub-layer using a 3×3 convolutional kernel and a ReLU (Rectified Linear Unit) activation function. The residual layer is composed of eight residual blocks connected in sequence, each residual block consisting of two convolutional sub-layers and a skip connection.

[0068] In this embodiment of the invention, step S130 is implemented as follows: The mask image MP1 corresponding to the first frame ultrasound image P1 is obtained; then, the first frame ultrasound image P1 and the mask image MP1 are superimposed in the channel dimension and input to the mask encoder for parsing to obtain the first target mask feature IM1; the mask image MP2 corresponding to the second frame ultrasound image P2 is obtained; then, the second frame ultrasound image P2 and the mask image MP2 are superimposed in the channel dimension and input to the mask encoder for parsing to obtain the second target mask feature IM2; and so on, obtaining the first N-1 frames of ultrasound images {P1, P2, ..., P...} N-2 ,P N-1 The corresponding N-1 target mask features are {IM1,IM2,...,IM...} N-2 IM N-1}

[0069] In this embodiment of the invention, the specific implementation process of step S140 includes the following:

[0070] Step S141: Transfer the Nth frame ultrasound image P N Compared with the first N-2 frames of ultrasound images {P1,P2,...,P N-2 Long-term global relevance matching between} is equivalent to converting it into the Nth target semantic feature F. N With the first N-2 target semantic features {F1,F2,...,F N-2 Long-term global correlation matching between}, and then using the L2 norm constraint principle to calculate the global matching cost is:

[0071]

[0072] In the formula, Cost Ref Let ||·||2 be the mathematical notation for the L2 norm, and let the Set function be used to set the first N-2 target semantic features {F1, F2, ..., F...}. N-2 Represented in matrix form;

[0073] Step S142: Transfer the Nth frame ultrasound image P N Compared with the (N-1)th frame ultrasound image P N-1 Short-term local correlation matching between them is equivalent to converting the Nth target semantic feature F N With the (N-1)th target semantic feature F N-1 The short-term local correlation matching between them is then performed, followed by channel stacking, two-dimensional matching, and convolution operations to calculate the local matching cost:

[0074]

[0075] In the formula, Cost m The local matching cost is defined as Conv, where Conv represents a 1×1 convolution operation, M represents a two-dimensional matching network, || represents channel stacking, p = (x, y) represents the position coordinates of the feature point, and u is the displacement hypothesis, also known as a displacement search box with length U and width V, representing the local matching cost for the (N-1)th target semantic feature F. N-1 Each feature point A and the Nth target semantic feature F N For each feature point B in the target semantic feature F, the position of each feature point A corresponds to the position of each feature point B. The center point of the displacement search box is the position of each feature point A itself. Then, the displacement search box is placed at the (N-1)th target semantic feature F. N-1 The small subset of feature points covered in the text are all matched with each feature point B for local correlation.

[0076] Step S143: Calculate the local matching cost (Cost). m Perform parallax prediction to obtain the Nth frame ultrasound image P N Compared with the (N-1)th frame ultrasound image PN-1 The short-term sports fields between are:

[0077]

[0078] In the formula, Vec represents the short-time motion field expressed in two channels, and σ represents the two-dimensional soft-argmin operation, which is mainly used to estimate the Nth target semantic feature F. N With the (N-1)th target semantic feature F N-1 The best matching relationship between corresponding feature points;

[0079] Step S144: Process the short-time motion field Vec using multiple convolutions to obtain short-time motion features F. SS Specifically, this is achieved by performing five 3×3 convolution operations on the short-term motion field Vec, followed by a single 1×1 convolution operation on the output, to obtain the short-term motion feature F. SS Furthermore, the stride of each convolution operation is set to 1;

[0080] Step S145: Transfer the short-time motion feature F SS A single 1×1 convolution operation is performed to complete the feature dimensionality reduction process. The Sigmoid activation function is then used to optimize the dimensionality-reduced short-time motion features F. SS Perform smoothing processing, and then apply the smoothed short-term motion features F SS As spatial attention weights, a broadcast mechanism is simultaneously used for the Nth target semantic feature F. N The weighting is then performed, and finally the weighted result is combined with the Nth target semantic feature F. N Add them together to obtain the attention feature F. MA .

[0081] In this embodiment of the invention, the above step S150 is implemented as follows: the continuous N motion information {S1, S2, ..., S...} N-2 ,S N-1 ,S N Each piece of motion information in the sequence {S1, S2, ..., S} specifically includes the three-dimensional spatial coordinate data and triaxial acceleration data of the ultrasonic probe at the corresponding acquisition time. The sequence of N consecutive motion information {S1, S2, ..., S} is then used to represent these data. N-2 ,S N-1 ,S N The sequence is input as a data sequence into an existing LSTM (Long Short Term Memory) model to capture the temporal information in the data sequence and learn the features that change over time, thereby outputting the motion temporal features F. LSTM .

[0082] In this embodiment of the invention, the specific implementation process of step S160 includes the following:

[0083] Step S161: Use the softmax function to calculate the global matching cost. Ref Normalization is performed to obtain the smoothed global matching cost W. Ref ;

[0084] Step S162: Select the first N-2 target mask features {IM1,IM2,...,IM...} N-2 The smoothed global matching cost W Ref Multiply the results to obtain the global matching features:

[0085] F MEM =Set{IM1,IM2,...,IM N-2}×W Ref ;

[0086] In the formula, F MEM For the global matching features, the Set function is used to set the first N-2 target mask features {IM1,IM2,...,IM...} N-2 Represented in matrix form;

[0087] Step S163: Add the (N-1)th target mask feature IM N-1 With the attention feature F MA Perform multiplication to obtain the local matching features F SM ;

[0088] Step S164: Transfer the motion timing feature F LSTM The global matching feature F MEM and the local matching feature F SM Perform fusion processing to obtain the Nth frame ultrasound image P N The corresponding target mask features are:

[0089] F M =Conv(F MEM ||F SM ||F LSTM )

[0090] In the formula, F M The target mask features are defined as follows: Conv represents a 1×1 convolution operation, and || represents channel stacking.

[0091] In this embodiment of the invention, the decoder mentioned in step S170 is specifically composed of a second input layer, a first upsampling layer, a first feature fusion layer, a second upsampling layer, a second feature fusion layer, a third upsampling layer, a third feature fusion layer, a fourth upsampling layer, and a second output layer connected sequentially. The specific implementation process of step S170 includes the following:

[0092] Step S171: Upon receiving the Nth frame ultrasound image P through the second input layer... N The corresponding target mask feature F M Then, it is directly transmitted to the first upsampling layer;

[0093] Step S172: Apply the target mask feature F through the first upsampling layer. M Perform a 2x upsampling process to obtain the first target mask feature F. M1 ;

[0094] Step S173: Combine the third intermediate feature C3 and the first target mask feature F through the first feature fusion layer. M1 Perform fusion processing to obtain the first mask feature map F P1 ;

[0095] Step S174: Apply the second upsampling layer to the first mask feature map F P1 Perform a 2x upsampling process to obtain the second target mask feature F. M2 ;

[0096] Step S175: Combine the second intermediate feature C2 and the second target mask feature F through the second feature fusion layer. M2 The fusion process is performed to obtain the second mask feature map F. P2 ;

[0097] Step S176: Apply the third upsampling layer to the second mask feature map F P2 Perform a 2x upsampling process to obtain the third target mask feature F. M3 ;

[0098] Step S177: The first intermediate feature C1 and the third target mask feature F are processed by the third feature fusion layer. M3 The fusion process is performed to obtain the third mask feature map F. P3 ;

[0099] Step S178: The third mask feature map F is processed by the fourth upsampling layer. P3 Perform a 4x upsampling process to obtain the Nth frame ultrasound image P. N The corresponding target mask image;

[0100] Step S179: The target mask image is directly output through the second output layer, and then the target mask image is visualized on a specific display device to facilitate timely viewing and analysis by staff.

[0101] It should be noted that, from the first mask feature map F P1 To the second mask feature map F P2 Then, the third mask feature map F P3 This can reflect the gradual enhancement effect of the edge region of the target object at different scales.

[0102] More specifically, the implementation process of step S173 above includes the following:

[0103] Step S173.1: Perform a 3×3 convolution operation on the third intermediate feature C3, and then combine this convolution result with the first target mask feature F. M1 The features are added together to obtain the first initial mask feature map F′. P1 ;

[0104] Step S173.2: Perform max pooling on the third intermediate feature C3 to obtain the first pooling result Pool1;

[0105] Step S173.3: Perform average pooling on the third intermediate feature C3 to obtain the second pooling result Pool2;

[0106] Step S173.4: After superimposing the first pooling result Pool1 and the second pooling result Pool2 along the channel dimension, firstly, a 7×7 convolution operation is performed on the superimposed result. Secondly, the convolution result is normalized using the Sigmoid activation function. Then, the normalized result is subtracted from 1 to complete the inversion operation. Finally, the inverted result is compared with the first initial mask feature map F′. P1 The multiplication is performed, and finally a 3×3 convolution operation is performed on the multiplication result to obtain the first mask feature map F. P1 .

[0107] More specifically, the implementation process of step S175 above includes the following:

[0108] Step S175.1: Perform a 3×3 convolution operation on the second intermediate feature C2, and then combine this convolution result with the second target mask feature F. M2 The features are added together to obtain the second initial mask feature map F′. P2 ;

[0109] Step S175.2: Perform max pooling on the second intermediate feature C2 to obtain the third pooling result Pool3;

[0110] Step S175.3: Perform average pooling on the second intermediate feature C2 to obtain the fourth pooling result Pool4;

[0111] Step S175.4: After superimposing the third pooling result Pool3 and the fourth pooling result Pool4 along the channel dimension, firstly, a 7×7 convolution operation is performed on the superimposed result. Secondly, the convolution result is normalized using the Sigmoid activation function. Then, the normalized result is subtracted by 1 to complete the inversion operation. Finally, the inverted result is combined with the second initial mask feature map F′. P2 The results are multiplied, and then a 3×3 convolution is performed on the multiplication result to obtain the second mask feature map F. P2 .

[0112] More specifically, the implementation process of step S177 above includes the following:

[0113] Step S177.1: Perform a 3×3 convolution operation on the first intermediate feature C1, and then combine this convolution result with the third target mask feature F. M3 The features are added together to obtain the third initial mask feature map F′. P3 ;

[0114] Step S177.2: Perform max pooling on the first intermediate feature C1 to obtain the fifth pooling result Pool5;

[0115] Step S177.3: Perform average pooling on the first intermediate feature C1 to obtain the sixth pooling result Pool6;

[0116] Step S177.4: After superimposing the fifth pooling result Pool5 and the sixth pooling result Pool6 along the channel dimension, firstly, a 7×7 convolution operation is performed on the superimposed result. Secondly, the convolution result is normalized using the Sigmoid activation function. Then, the normalized result is subtracted by 1 to complete the inversion operation. Finally, the inverted result is combined with the third initial mask feature map F′. P3 The results are multiplied, and then a 3×3 convolution operation is performed on the multiplication result to obtain the third mask feature map F. P3 .

[0117] It should be noted that the ultrasound image target enhancement processing method proposed in this invention can be applied to the field of breast lesion detection. Medical personnel hold the ultrasound probe and scan the patient's breast region to obtain a corresponding breast ultrasound image. This breast ultrasound image contains the target object—a breast lesion. Simultaneously, the motion information of the ultrasound probe during the scanning process is acquired through the inertial element, followed by a series of subsequent image analysis processes. Of course, this invention can also be applied to detecting other types of target objects in other areas of the patient's body, such as blood vessels, tumor cells, etc., and this invention does not impose specific limitations on these applications.

[0118] In addition, the inertial element can be a six-axis sensor of model ICM20602, an inertial measurement sensor of model MPU6000, or other related types of sensors.

[0119] In this embodiment of the invention, correlation matching analysis is performed on the N target semantic features corresponding to N consecutive frames of ultrasound images from both global and local perspectives. Then, the correlation information obtained from the analysis and the motion temporal features reflected by the ultrasound probe during the acquisition of N consecutive frames of ultrasound images are used to fuse and learn the N-1 target mask features corresponding to the first N-1 frames of ultrasound images to generate the target mask features corresponding to the Nth frame of ultrasound images. This can overcome the difficulty of target tracking caused by ultrasound probe jitter. Finally, combined with the multi-scale intermediate features corresponding to the Nth frame of ultrasound images, a decoder is used to perform multiple target edge enhancement processing on the target mask features corresponding to the Nth frame of ultrasound images. This makes the edge region of the target object more clearly highlighted in the final output target mask image, thereby achieving accurate tracking and positioning of the target object to meet the actual needs of clinical medicine.

[0120] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the composition of an ultrasound image target enhancement processing system provided in an embodiment of the present invention. The system includes:

[0121] The data acquisition module 210 is used to acquire N consecutive ultrasound images through an ultrasound probe, wherein each ultrasound image contains a target object. An inertial element is pre-installed on the ultrasound probe, and N consecutive motion information associated with the N consecutive ultrasound images is acquired synchronously through the inertial element.

[0122] The semantic feature parsing module 220 is used to input the N consecutive ultrasound images into the semantic encoder for parsing, so as to obtain N target semantic features corresponding to the N consecutive ultrasound images and the multi-scale intermediate features corresponding to the Nth ultrasound image.

[0123] The mask feature parsing module 230 is used to preprocess the first N-1 frames of ultrasound images, and then input the preprocessed first N-1 frames of ultrasound images into the mask encoder for parsing, so as to obtain N-1 target mask features corresponding to the first N-1 frames of ultrasound images.

[0124] The correlation analysis module 240 is used to analyze the correlation between the N target semantic features in order to obtain the attention features and the global matching cost;

[0125] The motion feature parsing module 250 is used to input the N consecutive motion information into the LSTM model for parsing in order to obtain motion temporal features;

[0126] The multi-feature fusion module 260 is used to fuse the N-1 target mask features, the attention features, the motion temporal features and the global matching cost to obtain the target mask features corresponding to the Nth frame ultrasound image;

[0127] The decoding processing module 270 is used to input the multi-scale intermediate features and target mask features corresponding to the Nth frame ultrasound image into the decoder for parsing, so as to obtain the target mask image corresponding to the Nth frame ultrasound image, and then perform visualization processing on the target mask image.

[0128] The content of the above method embodiments is applicable to this system embodiment. The functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are the same as those in the above method embodiments. Therefore, they will not be repeated here.

[0129] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements an ultrasound image target enhancement processing method as described in the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium by which a device (e.g., a computer, mobile phone, etc.) stores or transmits information in a readable form, and can be a read-only memory, a disk, or an optical disk, etc.

[0130] also, Figure 3 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. The computer device includes components such as a processor 320, a memory 330, an input unit 340, and a display unit 350. Those skilled in the art will understand that... Figure 3 The illustrated device structure is not intended to limit all devices and may include more or fewer components than shown, or combine certain components. The memory 330 can be used to store the computer program 310 and various functional modules. The processor 320 runs the computer program 310 stored in the memory 330, thereby performing various functional applications and data processing of the device. The memory can be internal memory or external memory, or include both internal and external memory. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, USB flash drives, magnetic tapes, etc. The memory 330 disclosed in the embodiments of this invention includes, but is not limited to, these types of memory. The memory 330 disclosed in the embodiments of this invention is only an example and not a limitation.

[0131] Input unit 340 is used to receive signal input and user-input keywords. Input unit 340 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive the corresponding connection device according to a pre-set program; other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as play control buttons, power buttons, etc.), trackballs, mice, joysticks, etc. Display unit 350 can be used to display user-input information or information provided to the user, as well as various menus of the terminal device. Display unit 350 may be in the form of a liquid crystal display, organic light-emitting diode, etc. Processor 320 is the control center of the terminal device, connecting various parts of the entire device through various interfaces and lines, performing various functions and processing data by running or executing software programs and / or modules stored in memory 330, and calling data stored in memory 330.

[0132] As one embodiment, the computer device includes a processor 320, a memory 330, and a computer program 310, wherein the computer program 310 is stored in the memory 330 and configured to be executed by the processor 320, and the computer program 310 is configured to perform an ultrasound image target enhancement processing method according to the above embodiment.

[0133] Although the description of this application has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment. Rather, it should be considered as effectively covering the intended scope of this application by referring to the appended claims and taking into account the prior art, which provides for a broad possible interpretation of these claims. Furthermore, the foregoing description of this application with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this application that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A method for enhancing a target in an ultrasound image, characterized in that, The method includes: An ultrasonic probe is used to acquire N consecutive frames of ultrasonic images containing the target object, and an inertial element mounted on the ultrasonic probe is used to acquire the corresponding N consecutive motion information. The N consecutive frames of ultrasound images are parsed using a semantic encoder to obtain the corresponding N target semantic features and the multi-scale intermediate features corresponding to the Nth frame of ultrasound image; The first N-1 frames of ultrasound images are preprocessed and then input into a mask encoder for parsing to obtain the corresponding N-1 target mask features. The correlations among the N target semantic features are analyzed to obtain the global matching cost and attention features; The motion temporal features are obtained by analyzing the N consecutive motion information using an LSTM model. The global matching cost, the attention feature, the N-1 target mask features, and the motion temporal features are fused to obtain the target mask features corresponding to the Nth frame ultrasound image; The target mask features and multi-scale intermediate features corresponding to the Nth frame ultrasound image are analyzed using a decoder to obtain the corresponding target mask image and visualize it. The step of analyzing the correlation between the N target semantic features to obtain the global matching cost and attention features includes: Based on the L2 norm, the relevance matching calculation is performed on the Nth target semantic feature and the first N-2 target semantic features to obtain the global matching cost; Channel stacking, two-dimensional matching, and convolution operations are sequentially performed on the (N-1)th target semantic feature and the Nth target semantic feature to obtain the local matching cost; Perform disparity prediction on the local matching cost to obtain the short-time motion field between the (N-1)th frame ultrasound image and the Nth frame ultrasound image; Perform multiple convolution operations on the short-time motion field to obtain short-time motion features; The short-term motion features are used to reweight the Nth target semantic feature, and then the reweighted result is added to the Nth target semantic feature to obtain the attention feature.

2. The ultrasound image target enhancement processing method according to claim 1, characterized in that, For each ultrasound image in the first N-1 frames of ultrasound images, preprocessing includes: Obtain the mask image corresponding to each frame of ultrasound image, and then overlay the channels of each frame of ultrasound image with the mask image.

3. The ultrasound image target enhancement processing method according to claim 1, characterized in that, The process of fusing the global matching cost, the attention features, the N-1 target mask features, and the motion temporal features to obtain the target mask features corresponding to the Nth frame of the ultrasound image includes: The global matching cost is normalized, and then the normalization result is multiplied by the first N-2 target mask features to obtain the global matching features; Multiply the attention feature by the (N-1)th target mask feature to obtain the local matching feature; The global matching features, the local matching features, and the motion temporal features are superimposed and convolved to obtain the target mask features corresponding to the Nth frame ultrasound image.

4. The ultrasound image target enhancement processing method according to claim 1, characterized in that, The semantic encoder includes a first input layer, a first convolutional layer, a first block network layer, a second block network layer, a third block network layer, a fourth block network layer, and a first output layer connected in sequence. When the semantic encoder is used to parse the Nth frame of ultrasound image, the first intermediate feature output by the second Block network layer, the second intermediate feature output by the third Block network layer, and the third intermediate feature output by the fourth Block network layer are collectively referred to as the multi-scale intermediate features.

5. The ultrasound image target enhancement processing method according to claim 4, characterized in that, The decoder includes a second input layer, a first upsampling layer, a first feature fusion layer, a second upsampling layer, a second feature fusion layer, a third upsampling layer, a third feature fusion layer, a fourth upsampling layer, and a second output layer; The first upsampling layer is used to upsample the target mask features corresponding to the Nth frame ultrasound image provided by the second input layer, and then the first feature fusion layer is used to fuse the upsampling result with the third intermediate feature to obtain the first mask feature map with enhanced target edge. The first mask feature map is upsampled using the second upsampling layer, and then the upsampling result is fused with the second intermediate feature using the second feature fusion layer to obtain a second mask feature map with enhanced target edges. The second mask feature map is upsampled using the third upsampling layer, and then the upsampling result is fused with the first intermediate feature using the third feature fusion layer to obtain a third mask feature map with enhanced target edges. The third mask feature map is upsampled using the fourth upsampling layer to obtain the final target mask image, which is then output via the second output layer.

6. The ultrasound image target enhancement processing method according to claim 5, characterized in that, The internal operation of the first feature fusion layer includes: The third intermediate feature is convolved, and the convolution result is added to the upsampling result to obtain the initial mask feature map; Max pooling is performed on the third intermediate feature to obtain the first pooling result; The third intermediate feature is subjected to average pooling to obtain the second pooling result; The first pooling result and the second pooling result are sequentially subjected to channel stacking, convolution, normalization and inversion operations. The output result is then multiplied by the initial mask feature map and then subjected to convolution to obtain the first mask feature map.

7. An ultrasound image target enhancement processing system, characterized in that, The system includes: The data acquisition module is used to acquire N consecutive frames of ultrasound images containing the target object using an ultrasound probe, and simultaneously acquire corresponding N consecutive motion information using an inertial element mounted on the ultrasound probe. The semantic feature parsing module is used to parse the N consecutive frames of ultrasound images using a semantic encoder to obtain the corresponding N target semantic features and the multi-scale intermediate features corresponding to the Nth frame of ultrasound image. The mask feature parsing module is used to input the first N-1 frames of ultrasound images after preprocessing into the mask encoder for parsing, and obtain the corresponding N-1 target mask features; The correlation analysis module is used to analyze the correlation between the N target semantic features to obtain the global matching cost and attention features; The motion feature parsing module is used to parse the N consecutive motion information using an LSTM model to obtain motion temporal features; A multi-feature fusion module is used to fuse the global matching cost, the attention feature, the N-1 target mask features and the motion temporal features to obtain the target mask features corresponding to the Nth frame ultrasound image; The decoding processing module is used to analyze the target mask features and multi-scale intermediate features corresponding to the Nth frame ultrasound image using a decoder, to obtain the corresponding target mask image and visualize it. The step of analyzing the correlation between the N target semantic features to obtain the global matching cost and attention features includes: Based on the L2 norm, the relevance matching calculation is performed on the Nth target semantic feature and the first N-2 target semantic features to obtain the global matching cost; Channel stacking, two-dimensional matching, and convolution operations are sequentially performed on the (N-1)th target semantic feature and the Nth target semantic feature to obtain the local matching cost; Perform disparity prediction on the local matching cost to obtain the short-time motion field between the (N-1)th frame ultrasound image and the Nth frame ultrasound image; Perform multiple convolution operations on the short-time motion field to obtain short-time motion features; The short-term motion features are used to reweight the Nth target semantic feature, and then the reweighted result is added to the Nth target semantic feature to obtain the attention feature.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The processor executes the computer program to implement the ultrasound image target enhancement processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the ultrasound image target enhancement processing method as described in any one of claims 1 to 6.

Images