Ultrasound image processing method, electronic device, and storage medium

Abstract

The application provides an ultrasound image processing method, an electronic device and a storage medium, and belongs to the technical field of ultrasound image processing. The method comprises the following steps: acquiring an initial ultrasound image, and dividing the initial ultrasound image into a plurality of block sub-images; extracting global context information and local detail features of the plurality of block sub-images; generating a target feature map based on the global context information and the local detail features; generating a type label of each pixel in the initial ultrasound image by performing a convolution operation on the target feature map, wherein the type label comprises one or more of nerve, blood vessel, muscle, fascia and bone; constructing a segmentation mask of the initial ultrasound image based on the type label of each pixel, and superimposing the segmentation mask on the initial ultrasound image to generate a target ultrasound image. The application can effectively improve the visualization effect of the ultrasound image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ultrasound image processing, and in particular to an ultrasound image processing method, an electronic device and a storage medium. BACKGROUND

[0002] Ultrasound-guided regional anesthesia is an effective anesthesia and analgesia measure. The technique requires an anesthesiologist to accurately identify a nerve and its adjacent tissue structure through a regional anesthesia ultrasound image, and to inject local anesthetic around the nerve under ultrasound guidance to achieve the purpose of regional anesthesia. However, in the actual application of traditional ultrasound-guided regional anesthesia, due to the limitations of various factors such as noise and bone structure obstruction in ultrasound imaging, the obtained ultrasound images often have problems such as blurred anatomical edges and insufficient contrast, making it difficult for doctors to accurately identify the target nerve and surrounding tissue structure from the ultrasound images with poor visual effects.

[0003] Currently, some studies have attempted to improve ultrasound image quality through image processing techniques, such as using traditional filtering or enhancement algorithms for image optimization. However, these methods are often designed for specific ultrasound imaging problems and are difficult to cope with complex and variable features in different organs and pathological states. For example, in abdominal ultrasound, the artifacts produced by fat tissue and air cavities cannot be effectively processed. Moreover, although these methods can reduce noise, they tend to lose small structures and detailed information during the denoising process, affecting diagnostic accuracy. For example, in liver nodule detection, excessive smoothing may mask the true morphology of early micro-tumors, interfering with the accurate judgment of doctors.

[0004] Therefore, there is currently a technical problem of poor visualization of ultrasound imaging. SUMMARY

[0005] The purpose of the present application is to provide an ultrasound image processing method, an electronic device and a storage medium to solve the above problems.

[0006] To achieve the above purpose, in a first aspect, the present application provides an ultrasound image processing method, which comprises:

[0007] obtaining an initial ultrasound image and dividing the initial ultrasound image into a plurality of block sub-images;

[0008] extracting global context information and local detail features of the plurality of block sub-images;

[0009] generating a target feature map based on the global context information and the local detail features;

[0010] By performing a convolution operation on the target feature map, a type label is generated for each pixel in the initial ultrasound image. The type label includes one or more of the following: nerve, blood vessel, muscle, fascia, and bone.

[0011] A segmentation mask for the initial ultrasound image is constructed based on the type label of each pixel, and the segmentation mask is superimposed on the initial ultrasound image to generate the target ultrasound image.

[0012] In some implementations, the extraction of global context information and local detail features from the plurality of sub-image blocks includes:

[0013] The aforementioned sub-image blocks are mapped to vector representations of a preset dimension;

[0014] Add the positional encoding of the corresponding sub-image to the vector representation of each sub-image;

[0015] Based on the positional encoding corresponding to each sub-image block, the global context information of the several sub-image blocks is calculated through a multi-head self-attention mechanism;

[0016] Based on the global context information, a feedforward network is used to perform a nonlinear transformation on the vector representation of each sub-image to obtain the local detail features of the several sub-images.

[0017] In some implementations, generating the target feature map based on the global context information and the local detail features includes:

[0018] The global context information and the local detail features are fused to generate a fused feature map;

[0019] The fused feature map is input into the encoder network to generate a multi-level initial feature map;

[0020] By using skip connections, the initial low-level feature maps are passed to the corresponding layers of the decoder network to obtain low-level features.

[0021] Upsampling is performed on the initial high-level feature map to obtain high-level features;

[0022] The low-level features and the high-level features are fused to generate a target feature map.

[0023] In some implementations, after constructing a segmentation mask for the initial ultrasound image based on the type label of each pixel and overlaying the segmentation mask onto the initial ultrasound image to generate the target ultrasound image, the method further includes:

[0024] Based on a preset set of regional anesthesia standard images, a subset of standard images corresponding to the target ultrasound image is determined.

[0025] The images in the subset of standard images are fused to obtain a fused standard image;

[0026] The fused standard image and the target ultrasound image are matched to determine the scanning accuracy of the target ultrasound image.

[0027] In some embodiments, after matching the fused standard image and the target ultrasound image to determine the scanning accuracy of the target ultrasound image, the method further includes:

[0028] When the scanning accuracy is lower than a preset matching threshold, a set of continuous target ultrasound images corresponding to consecutive frames of the initial ultrasound image is obtained;

[0029] Based on the type label distribution of each frame in the continuous target ultrasound image set, the key anatomical structures of each frame in the continuous target ultrasound image set are determined;

[0030] Based on the location parameters of each of the key anatomical structures, a virtual anatomical model is constructed;

[0031] Based on the virtual anatomical model and the fused standard image, the target scanning parameters of the ultrasound probe are determined;

[0032] The actual physical parameters of the ultrasound probe are obtained, and a scanning guidance path is output based on the actual physical parameters and the target scanning parameters.

[0033] In some implementations, the location information includes center coordinates, volume information, direction vector, and depth information. Before constructing the virtual anatomical model based on the location parameters of each of the key anatomical structures, the following further steps are included:

[0034] Based on the boundary information of each of the key anatomical structures, calculate the center coordinates and volume information of each of the key anatomical structures;

[0035] The orientation vector of each key anatomical structure is determined by fitting the principal axis of each key anatomical structure separately.

[0036] The depth information of each of the key anatomical structures is determined based on the original grayscale values ​​of the target ultrasound image.

[0037] In some embodiments, constructing a virtual anatomical model based on the location parameters of each of the key anatomical structures further includes:

[0038] The center coordinates, volume information, orientation vector and depth information of each key anatomical structure in each frame of the continuous target ultrasound image set are integrated to generate a set of spatial parameters for each key anatomical structure.

[0039] Based on the set of spatial parameters, each of the key anatomical structures is mapped into three-dimensional space to form a spatial distribution representation of each of the key anatomical structures;

[0040] The spatial distribution representation of each of the key anatomical structures is rendered to generate a virtual anatomical model.

[0041] In some implementations, after constructing a segmentation mask for the initial ultrasound image based on the type label of each pixel and overlaying the segmentation mask onto the initial ultrasound image to generate the target ultrasound image, the method further includes:

[0042] The target anatomical structure is determined based on the type label distribution in the target ultrasound image;

[0043] The target pressure range of the ultrasound probe is determined based on the tissue characteristics of the target anatomical structure.

[0044] The real-time pressure of the ultrasound probe is acquired, and a correction prompt is output when the real-time pressure exceeds the target pressure range.

[0045] Secondly, this application proposes an electronic device, comprising:

[0046] One or more processors;

[0047] Memory, used to store one or more programs.

[0048] When the one or more programs are executed by the one or more processors, the one or more processors perform the ultrasound image processing method as described above.

[0049] Thirdly, this application proposes a storage medium storing executable instructions that, when executed by a processor, cause the processor to perform the ultrasound image processing method as described above.

[0050] Compared with the prior art, the beneficial effects of this application include:

[0051] Firstly, by combining global contextual information and local detailed features, the anatomical relationships of the target nerve and surrounding tissues can be captured more accurately, thus significantly improving the accuracy of image segmentation. Secondly, pixel-level annotation, compared to traditional region- or edge-based processing methods, can describe the distribution of different tissues in greater detail. Thirdly, during the generation of the target ultrasound image, by overlaying the segmentation mask onto the initial ultrasound image, not only can the information of the initial ultrasound image be preserved, but key anatomical structures (such as nerves and blood vessels) can also be displayed through visual markers, significantly improving the visualization effect of ultrasound imaging. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0053] Figure 1 This is a flowchart illustrating an ultrasound image processing method in one embodiment;

[0054] Figure 2 This is a flowchart illustrating the process of extracting global context information and local detail features of the several sub-image blocks in one embodiment;

[0055] Figure 3 This is a schematic diagram of the process of generating a target feature map based on the global context information and the local detail features in one embodiment;

[0056] Figure 4 This is a flowchart illustrating the ultrasound image processing method in another embodiment;

[0057] Figure 5 This is a flowchart illustrating the ultrasound image processing method in yet another embodiment;

[0058] Figure 6 This is a flowchart illustrating the process of constructing a virtual anatomical model based on the positional parameters of each of the key anatomical structures in one embodiment.

[0059] Figure 7 This is a schematic diagram of the electronic device involved in the ultrasound image processing method in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0062] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0063] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0064] As mentioned above, some studies have attempted to improve ultrasound image quality through image processing techniques, such as using traditional filtering or enhancement algorithms for image optimization. However, these methods are often designed for specific ultrasound imaging problems and struggle to handle the complex and varied features of different organs and pathological conditions. For example, in abdominal ultrasound, artifacts caused by adipose tissue and air cavities cannot be effectively processed. Furthermore, while these methods can reduce noise, they are prone to losing minute structural and detailed information during denoising, affecting diagnostic accuracy. For instance, in liver nodule detection, excessive smoothing may mask the true morphology of early-stage small tumors, interfering with the doctor's accurate judgment. Therefore, there is currently a technical problem of poor visualization effects in ultrasound imaging. To address this, this application proposes an ultrasound image processing method, electronic device, and storage medium that uses the TransUNet deep learning model to perform real-time enhancement and segmentation of ultrasound images, effectively improving the visualization effects of ultrasound images.

[0065] like Figure 1 As shown in the embodiments of this application, an ultrasound image processing method includes the following steps:

[0066] Step S10: Obtain an initial ultrasound image and divide the initial ultrasound image into several sub-images.

[0067] In this embodiment, the initial ultrasound image refers to the unprocessed raw ultrasound image (usually a two-dimensional grayscale image) acquired through ultrasound equipment. This image serves as input to a pre-trained regional anesthesia ultrasound image recognition model. This model is a deep learning model combining Transformer and U-Net architectures, specifically designed for real-time enhancement and segmentation of ultrasound images. Specifically, the regional anesthesia ultrasound image recognition model includes a Transformer-based encoder and a U-Net-based decoder. The Transformer-based encoder consists of multiple stacked Transformer layers (also called encoder layers). Each Transformer layer contains two main sub-layers: a self-attention mechanism and a feedforward network. The self-attention mechanism in the Transformer helps the model better understand the relationship between key anatomical structures such as nerves and blood vessels and surrounding tissues in the ultrasound image, capturing long-range dependencies that are difficult for traditional convolutional neural networks to detect. The feedforward network in the Transformer extracts local detail features from the image. The U-Net-based decoder integrates global contextual information and local detail features. Through skip connections (concatenating feature maps from different levels in the encoder with corresponding feature maps from the decoder), it continuously integrates local detail information and global contextual information to predict the category label (e.g., nerve, blood vessel) for each pixel, thus generating a high-resolution segmentation mask. This combination enables the model to more meticulously depict the boundaries and shapes of key anatomical structures when processing ultrasound image enhancement and segmentation tasks, building upon the acquired global information.

[0068] Prior to step S10, a set of regional anesthesia ultrasound sample images can be obtained, manually segmented and labeled by a team of experts in the field of regional anesthesia. Specifically, the experts will precisely and manually label the ultrasound sample images based on the anatomical structures in the ultrasound sample images, such as nerves, blood vessels, and muscles, segmenting and labeling different tissue structures to form labeled training data. Subsequently, these labeled ultrasound sample images are used to train a regional anesthesia ultrasound image recognition model. Through deep learning algorithms, the model can learn the characteristics and patterns of different tissue structures, thereby automatically segmenting and recognizing ultrasound images in subsequent applications.

[0069] After acquiring the initial ultrasound image, before inputting it into the pre-trained regional anesthesia ultrasound image recognition model, the initial ultrasound image can undergo some preprocessing, such as normalization, resolution adjustment, noise reduction, or contrast enhancement. Normalization refers to scaling the pixels of the initial ultrasound image to a fixed range (e.g., 0 to 1). Resolution adjustment refers to adjusting the resolution of the initial ultrasound image to suit the input size requirements of the model.

[0070] The processed initial ultrasound image is input into a pre-trained regional anesthesia ultrasound image recognition model. Based on the received initial ultrasound image, the model, during the encoder stage, segments the initial ultrasound image into several sub-images. Each sub-image primarily contains all or local features of one or more sites / organs. Adjacent sub-images may overlap, allowing information about a site / organ to be fully contained within one or more sub-images.

[0071] Step S20: Extract the global context information and local detail features of the several sub-image blocks.

[0072] In this embodiment, the regional anesthesia ultrasound model extracts global contextual information and local detail features from several sub-image blocks using a Transformer-based encoder. Global contextual information refers to the overall features of the entire initial ultrasound image, reflecting the relationships and overall structure between different sub-image blocks. In ultrasound images, global contextual information helps the regional anesthesia ultrasound image recognition model understand the spatial relationships and interactions between key anatomical structures such as nerves and blood vessels and surrounding tissues. For example, through global contextual information, the model can identify the relative positions of nerves and blood vessels, as well as their layout relationships with other tissues such as bones and muscles. This information is crucial for accurately segmenting and identifying key anatomical structures in ultrasound images because it provides the overall semantic and structural background of the image. Local detail features refer to the detailed features from each small sub-image block of the initial ultrasound image, such as edges, textures, and grayscale variations. In ultrasound images, local detail features help the regional anesthesia ultrasound image recognition model accurately identify and segment the boundaries and internal features of key anatomical structures such as nerves and blood vessels. For example, through local detail features, the model can identify the subtle branches of nerve fibers, as well as the wall thickness and internal blood flow of blood vessels. This detailed information is crucial for accurately segmenting and identifying key anatomical structures in ultrasound images, as it provides fine structural and textural information about the image.

[0073] In this embodiment, global contextual information and local detail features complement each other, jointly providing rich image information for the regional anesthesia ultrasound image recognition model. Global contextual information helps the regional anesthesia ultrasound image recognition model understand the overall structure and layout of the image, while local detail features ensure that the model can perform detailed analysis and recognition of every part of the image. This combination enables the regional anesthesia ultrasound image recognition model to achieve better performance in complex ultrasound image recognition and segmentation tasks.

[0074] In some implementations, such asFigure 2 As shown, step S20 includes:

[0075] Step S21: Map the plurality of sub-image blocks into vector representations of a preset dimension.

[0076] Mapping each sub-image to a vector representation of a predetermined dimension is a process of compressing the pixel values ​​of the sub-image into a fixed-length vector. For example, several sub-images (e.g., each sub-image is 16×16 pixels in size) can be mapped to vectors of a predetermined dimension (e.g., 64-dimensional).

[0077] In some implementations, for each sub-image, a small convolutional neural network (CNN) can be used as a projector to map the two-dimensional image patch into a one-dimensional vector. This projector can extract basic features of the sub-image, such as edges and textures, by learning the weights of different convolutional kernels, and combine these features into a fixed-dimensional vector representation through fully connected layers.

[0078] In step S21, by mapping the sub-image from a high-dimensional pixel space to a low-dimensional vector space, the dimensionality of the data is reduced, thus lowering the complexity of subsequent calculations. Simultaneously, this process also performs preliminary feature extraction on the sub-image, transforming pixel-level information into vector representations that reflect the basic features of the sub-image. These vector representations contain important features such as the local texture and grayscale of the sub-image, providing foundational data for subsequent self-attention mechanisms and nonlinear transformations.

[0079] Step S22: Add the position code of the corresponding sub-image to the vector representation of each sub-image block.

[0080] In this embodiment, positional encoding is a signal or vector used to represent the spatial location of a sub-image within the initial ultrasound image. Since the Transformer model inherently lacks the ability to perceive the order or spatial location of elements in the input sequence, positional information needs to be explicitly added to the vector representation of the sub-image. Positional encoding can be a fixed-positional encoding based on sine and cosine functions, or a trainable positional encoding obtained through learning.

[0081] For example, for the i-th sub-image, a two-dimensional positional encoding vector composed of sine and cosine functions can be generated, where the frequency varies proportionally with position. This positional encoding vector maintains the same dimension as the vector representation of the sub-image, and then the positional encoding is combined with the vector representation of the sub-image through simple element-wise addition or concatenation operations. In this way, the vector representation of each sub-image not only contains its own feature information but also its spatial position information in the original image.

[0082] In some implementations, the two-dimensional position coordinates of the sub-image can be calculated first based on its row and column numbers in the initial ultrasound image. These coordinates are then mapped to position encoding vectors of the corresponding dimensions using sine and cosine functions. For example, for the position (row, col), it can be decomposed into two dimensions: row and column, and the sine and cosine values ​​can be calculated separately. These values ​​are then concatenated into a long vector as the position code.

[0083] In step S22, by adding positional encoding, additional dimensional information is added to the originally identical sub-image features, enabling the regional anesthesia ultrasound image recognition model to distinguish sub-images that were originally difficult to distinguish due to similar texture or grayscale features, thereby improving the overall distinguishability of features and the representational ability of the regional anesthesia ultrasound image recognition model.

[0084] Step S23: Based on the positional encoding corresponding to each sub-image block, the global context information of the several sub-image blocks is calculated through a multi-head self-attention mechanism.

[0085] In this embodiment, the multi-head self-attention mechanism is one of the core components of the regional anesthesia ultrasound image recognition model, used to calculate the correlation between different sub-images. During the extraction of global contextual information, the multi-head self-attention mechanism interacts the vector representation of each sub-image with the vector representations of all other sub-images, thereby obtaining global information about the entire initial ultrasound image.

[0086] In some implementations, based on the positional encoding corresponding to each sub-image, a set of query vectors, key vectors, and value vectors corresponding to each sub-image are generated through a multi-head self-attention mechanism. The attention weights between different sub-images are obtained by calculating the dot product similarity between the query vector and the key vector. The attention weights reflect the relevance between different sub-images, while the value vectors represent the feature content of the sub-images. The global context information of each sub-image is obtained by weighted summation of the attention weights and value vectors.

[0087] For example, suppose there are N sub-images, each represented by a d-dimensional vector. The multi-head self-attention mechanism divides these vectors into H heads, calculating attention weights and weighted summations in each head. This is equivalent to extracting different aspects of global information from multiple different subspaces, and finally combining the information from all heads to form a richer and more comprehensive global contextual information vector.

[0088] In step S23, considering that certain important anatomical structures may be distributed in different regions of the initial ultrasound image, with complex long-distance dependencies between them—for example, a nerve may be associated with distant blood vessels or muscle tissue—the multi-head self-attention mechanism can capture these global dependencies without distance limitations, enabling the regional anesthesia ultrasound image recognition model to understand the structure and layout of the entire initial ultrasound image. By integrating attention information from different sub-images, the regional anesthesia ultrasound image recognition model can generate feature representations with a global perspective, which is of great significance for the detection of medium-to-large targets (such as the brachial plexus) and image segmentation tasks in the initial ultrasound image.

[0089] Step S24: Based on the global context information, the vector representation of each sub-image is nonlinearly transformed through a feedforward network to obtain the local detail features of the several sub-images.

[0090] In this implementation, the feedforward network consists of two fully connected layers and a non-linear activation function. In the first fully connected layer, the input vector representation (including global context information processed by the self-attention mechanism) is expanded to a higher dimension and then transformed by a non-linear activation function (such as ReLU or GELU) to introduce non-linear features. Next, in the second fully connected layer, the features are shrunk back to the original dimension. Throughout this transformation process, the feedforward network can learn more complex feature maps, thereby better capturing local detail features in the sub-image.

[0091] For example, assuming the input vector has a dimension of d_model, it is expanded to a dimension of 4d_model in the first fully connected layer of the feedforward network. Then, the ReLU activation function is used to non-linearly project and fold the features in the high-dimensional space. Finally, the second fully connected layer shrinks the features back to the d_model dimension, resulting in local detail features after non-linear transformation.

[0092] In step S24, because the anatomical structures in ultrasound images vary in shape and have complex textures, simple linear transformations may not be able to fully express these features. By using the nonlinear transformation of a feedforward network, the features can be adjusted and combined more finely, thereby extracting richer local detail features.

[0093] Step S30: Generate a target feature map based on the global context information and the local detail features.

[0094] Individual global contextual information and local detail features can only reflect a portion of the features of an initial ultrasound image. Global information provides the overall structure and semantics, while local details focus on fine textures and edges. By integrating global contextual information and local detail features into a target feature map using a U-Net-based decoder, not only can macroscopic anatomical layouts be included, but subtle tissue texture variations can also be preserved. This is crucial for accurately identifying and segmenting complex structures in ultrasound images.

[0095] In some implementations, such as Figure 3 As shown, step S30 includes:

[0096] Step S31: The global context information and the local detail features are fused to generate a fused feature map.

[0097] In this embodiment, global contextual information and local detail features are two different levels of features, which respectively contain the overall structure and local details of the image. To generate a richer feature representation, global contextual information and local detail features are fused to generate a fused feature map.

[0098] In some implementations, fusion can be achieved through simple element-wise weighted summation or feature concatenation. For example, assuming the dimension of global context information is d_g and the dimension of local detail features is d_l, a fused feature map can be generated through simple element-wise weighted summation, i.e., fused feature map = global context information × w_g + local detail features × w_l, where w_g and w_l are learnable weights used to adjust the relative importance of the two features.

[0099] In some implementations, more sophisticated fusion strategies can be used, such as learning how to fuse the two types of features through a small neural network. For example, global contextual information and local detail features can be concatenated and then a fused feature map can be generated through several fully connected or convolutional layers.

[0100] Step S32: Input the fused feature map into the encoder network to generate a multi-level initial feature map.

[0101] In this embodiment, the U-Net-based decoder uses an image segmentation network structure, comprising an encoder network and a decoder network. The encoder network is a multi-layer neural network consisting of multiple convolutional and pooling layers, used to extract and compress features from the input fused feature map at multiple levels, thereby generating initial feature maps at different levels.

[0102] In some implementations, the encoder network performs multiple convolutions and downsampling operations on the fused feature map to progressively extract features at different levels. For example, the first convolutional block may contain two or three convolutional layers to extract low-level local features, and then a pooling layer reduces the spatial resolution of the fused feature map by half. The second convolutional block further extracts deeper features and then reduces the spatial resolution of the fused feature map again through another pooling layer. This process is repeated multiple times until a multi-level initial feature map is generated.

[0103] In this way, the encoder network can generate multi-level initial feature maps, with different levels of initial feature maps having an inverse relationship between spatial resolution and semantic level. For example, low-level initial feature maps may have a low semantic level but high spatial resolution (containing rich local texture and edge information), while high-level initial feature maps may contain higher-level semantic information, such as the overall shape and distribution of anatomical structures, but have low spatial resolution.

[0104] In step S32, the multi-level initial feature maps provide a rich feature foundation for the subsequent feature fusion and decoding processes. The low-level initial feature maps can be used to recover image details, while the high-level initial feature maps can be used to guide the overall segmentation and recognition.

[0105] Step S33: Through skip connections, the low-level initial feature map is passed to the corresponding layer of the decoder network to obtain low-level features.

[0106] In this embodiment, skip connections are a technique that directly passes low-level initial feature maps from the encoder network to the corresponding layers of the decoder network in order to preserve and utilize low-level detailed features.

[0107] Step S34: Upsample the high-level initial feature map to obtain high-level features.

[0108] High-level initial feature maps typically have low spatial resolution but high semantic level. To generate the final segmentation mask or target feature map, these high-level feature maps need to be upsampled to restore a spatial resolution close to that of the initial ultrasound image.

[0109] In some implementations, upsampling can be performed using methods such as bilinear interpolation, nearest-neighbor interpolation, and transposed convolution. Bilinear interpolation is an interpolation-based upsampling method that increases spatial resolution by inserting new pixel values ​​into the high-level initial feature map; these pixel values ​​are calculated using linear interpolation. Nearest-neighbor interpolation fills new spatial locations by copying the nearest pixel value. Transposed convolution is a convolution-based upsampling method that learns a convolution kernel to perform upsampling, resulting in stronger expressive power. For example, a transposed convolutional layer can be used to upsample the high-level initial feature map while increasing the number of channels, thereby generating high-level features with higher spatial resolution and more channels.

[0110] In step S34, because the spatial resolution of the high-level initial feature map is low, it cannot be directly used to generate a high-resolution segmentation mask or target feature map. By upsampling, the spatial resolution of the high-level initial feature map can be restored to a level similar to that of the initial ultrasound image, thereby providing a more refined feature representation for subsequent processing.

[0111] Step S35: Fuse the low-level features and the high-level features to generate a target feature map.

[0112] By fusing low-level features that mainly contain local details and high-level features that mainly contain global semantic information, similar to a secondary purification and enhancement of features, more potential connections hidden in global and local information can be discovered, making the features richer and more discriminative.

[0113] In some implementations, fusion can be achieved through element-wise weighted summation, feature concatenation, and multilayer perceptrons (MLPs). For example, low-level and high-level features can be summed element-wise with weights learned by a small neural network. Alternatively, the two types of features can be concatenated and then fused through several convolutional or fully connected layers.

[0114] In the U-Net architecture of this embodiment, the upsampled high-level features and the low-level features passed through skip connections are concatenated and then processed by a small convolutional network. This small convolutional network may contain several convolutional layers and non-linear activation functions to learn how to fuse the two types of features.

[0115] Step S40: By performing a convolution operation on the target feature map, a type label for each pixel in the initial ultrasound image is generated.

[0116] In this embodiment, type labels refer to tags used in ultrasound images to identify the tissue structure type to which each pixel belongs. These tags typically include nerves, blood vessels, muscles, fascia, and bones. Type labels are used to segment and identify ultrasound images, assigning each pixel to the corresponding tissue structure category, thereby achieving accurate annotation and segmentation of the ultrasound image.

[0117] It's important to understand that the regional anesthesia ultrasound image recognition model learned the features and patterns of different tissue structures (such as nerves, blood vessels, and muscles) during the training phase using a large number of manually labeled ultrasound sample images. These features and patterns are encoded in the weights of the convolution kernel, enabling the regional anesthesia ultrasound image recognition model to automatically extract and recognize these features during convolution operations.

[0118] In some implementations, when convolutional layers are used to convolve the target feature map, multiple convolutional kernels (filters) within the convolutional layer slide across the target feature map, extracting a feature vector for each pixel. After the final convolutional layer, the feature vector for each pixel is passed to a classifier. The classifier is typically a fully connected layer or a convolutional layer, used to output the probability distribution for each pixel. The classifier labels the pixel with the type that has the highest probability of belonging to various types (such as nerve, blood vessel, muscle, fascia, and bone) by calculating the probability distribution of each pixel belonging to various types. For example, if a pixel has the highest probability of belonging to a nerve, then that pixel will be labeled as a nerve.

[0119] Step S50: Construct a segmentation mask for the initial ultrasound image based on the type label of each pixel, and overlay the segmentation mask onto the initial ultrasound image to generate the target ultrasound image.

[0120] In this embodiment, the segmentation mask is a color marker that matches the size of the initial ultrasound image and is used to mark specific regions or structures in the image, such as nerves in red (specific RGB values) and blood vessels in blue.

[0121] In some implementations, image fusion techniques such as opacity blending or intensity adjustment can be used when fusing the segmentation mask with the initial ultrasound image. Opacity blending, by specifying the transparency of the mask, makes it cover the original image in a semi-transparent form, thereby simultaneously displaying the texture details of the original image and the key areas marked by the mask.

[0122] Taking transparent overlay as an example, when the mask transparency is set to 50%, doctors can clearly see the grayscale information of the initial ultrasound image, while also observing the segmentation results of the color annotations. This effectively avoids the complete occlusion of the original image by the mask information, achieving an organic combination of the two visually.

[0123] In step S50, pixel-level type labels are presented in an intuitive image form to facilitate subsequent visualization and analysis, clearly identifying the distribution of various anatomical structures in the initial ultrasound image, and providing doctors with precise information on the location of lesions and key structures.

[0124] In some implementations, the target anatomical structure is determined based on the type label distribution in the target ultrasound image. The target pressure range of the ultrasound probe is determined based on the tissue characteristics of the target anatomical structure; for example, the pressure range may be smaller for softer nerve tissue to avoid damage. The real-time pressure of the ultrasound probe is acquired, and a correction prompt is output when the real-time pressure exceeds the target pressure range. Here, type label distribution refers to the distribution of type labels for each pixel in the target ultrasound image; these labels are used to identify different tissue structures, such as nerves, blood vessels, and muscles. Target anatomical structure refers to the anatomical structure that requires focused attention in the target ultrasound image, usually determined based on the type label distribution, such as nerves and blood vessels. Tissue characteristics refer to the physical and physiological characteristics of the target anatomical structure, such as soft tissue and hard tissue. Target pressure range refers to the pressure range that the ultrasound probe needs to achieve during pressure application to ensure accurate capture of the characteristics of the target anatomical structure. Real-time pressure refers to the pressure value measured in real-time by the ultrasound probe during actual operation. By determining the target pressure range, the pressure application process of the ultrasound probe can be optimized, improving the accuracy and safety of pressure application.

[0125] In the ultrasound image processing method proposed in this application, firstly, by combining global contextual information and local detailed features, the anatomical relationships of the target nerve and surrounding tissues can be captured more accurately, thereby significantly improving the accuracy of image segmentation. Secondly, the pixel-level annotation method, compared with traditional region- or edge-based processing methods, can describe the distribution of different tissues in more detail. Thirdly, in the process of generating the target ultrasound image, by superimposing a segmentation mask onto the initial ultrasound image, not only can the information of the initial ultrasound image be preserved, but key anatomical structures (such as nerves and blood vessels) can also be displayed through visual markers, significantly improving the visualization effect of ultrasound imaging.

[0126] In one embodiment, such as Figure 4 As shown, after step S50, the method further includes:

[0127] Step S60: Based on a preset regional anesthesia standard image set, determine the standard image subset corresponding to the target ultrasound image.

[0128] In this embodiment, the preset regional anesthesia standard image set is a group of high-quality ultrasound images annotated and verified by experts. These images cover various common regional anesthesia scenarios and anatomical structures. Each regional anesthesia scenario or anatomical structure has a corresponding series of continuous ultrasound image frames, constructing a dynamic standard image range. This is to capture anatomical changes at different time points and ensure that the standard images can reflect the anatomical state at different time points. Each standard image has clear anatomical edges, good contrast, and accurate annotations, serving as a target for evaluation and comparison to assess the quality and accuracy of the target ultrasound image. By comparing with the standard image set, the scanning accuracy and quality of the target ultrasound image can be determined.

[0129] The target ultrasound image is compared with images in the regional anesthesia standard image set using a feature matching algorithm. Key features can include the image's edges, texture, shape, etc., as well as key anatomical structures within the image (such as nerves, blood vessels, muscles, etc.).

[0130] Based on the comparison results (similarity calculation results), the standard image subset is selected from the images that are most similar to the target ultrasound image (above a preset similarity threshold or a preset number of images with positive similarity ranking). The number of images in the standard image subset is an integer greater than one, and the specific number depends on the application scenario and computing resources.

[0131] Step S70: Perform fusion processing on each image in the standard image subset to obtain a fused standard image.

[0132] Multi-scale deep learning (such as neural network models like U-net) is employed to fuse images from a standard image subset. By fusing images from different time points, anatomical structural information can be integrated to generate a fused standard image containing richer features. This includes not only static anatomical structures but also dynamically changing information, such as the movement or deformation of nerves and blood vessels.

[0133] In step S70, fusing standard images provides more comprehensive anatomical information, which helps to more accurately assess the scanning accuracy of the target ultrasound image. By including features from different time points, fusing standard images can better reflect the dynamic changes of anatomical structures, thereby improving the accuracy and reliability of the assessment.

[0134] Step S80: Match the fused standard image and the target ultrasound image to determine the scanning accuracy of the target ultrasound image.

[0135] Image matching algorithms are used to match the fused standard image with the target ultrasound image. Matching algorithms can be based on feature point matching, cross-correlation coefficients, or other similarity metrics.

[0136] Based on the matching results (the calculation results of the matching algorithm), it is determined whether the scanning accuracy of the target ultrasound image reaches a preset matching threshold. If it reaches the preset matching threshold, the current scanning operation is deemed qualified; if it is below the preset matching threshold, the current scanning operation is deemed unqualified.

[0137] In some implementations, when the scanning accuracy is lower than a preset matching threshold, such as Figure 5 As shown, after step S80, the method further includes:

[0138] Step A10: Obtain the set of continuous target ultrasound images corresponding to the continuous frames of the initial ultrasound image.

[0139] In the actual operation of ultrasound-guided regional anesthesia, the ultrasound equipment continuously acquires ultrasound images at a high frame rate. These continuously acquired image frames are continuous in time and can reflect the dynamic changes of anatomical structures. In this embodiment, the continuous frames of the initial ultrasound image refer to ultrasound image frames that are continuous in time with the initial ultrasound image. The continuous target ultrasound image set refers to the image set consisting of the initial ultrasound image and its continuous frames.

[0140] Step A20: Based on the type label distribution of each frame of the continuous target ultrasound image set, determine the key anatomical structures of each frame of the continuous target ultrasound image set.

[0141] For each frame in a continuous set of target ultrasound images, the distribution of its type labels is analyzed. Type labels include nerves, blood vessels, muscles, fascia, and bones. By analyzing the distribution of these type labels, the location and proportion of each key anatomical structure in each frame can be determined.

[0142] Step A30: Construct a virtual anatomical model based on the positional parameters of each of the key anatomical structures.

[0143] In this embodiment, the positional parameters include center coordinates, volume information, direction vector, and depth information. Based on the boundary information of each key anatomical structure, the center coordinates and volume information of each key anatomical structure are calculated; the direction vector of each key anatomical structure is determined by fitting its principal axis; and the depth information of each key anatomical structure is determined based on the original grayscale values ​​of the target ultrasound image.

[0144] In some implementations, such as Figure 6 As shown, step A30 includes:

[0145] Step A31: Integrate the center coordinates, volume information, orientation vector and depth information of each key anatomical structure in each frame of the continuous target ultrasound image set to generate a set of spatial parameters for each key anatomical structure.

[0146] Specifically, the center coordinates, volume information, orientation vectors, and depth information of each key anatomical structure in each frame of a continuous target ultrasound image set are integrated into a unified data structure, forming a set of spatial parameters for each key anatomical structure. The data structure can be a list, array, or dictionary, where each element corresponds to a key anatomical structure and contains all its positional parameters.

[0147] By pooling the location parameters scattered across different image frames, a more comprehensive and accurate analysis of each key anatomical structure can be performed.

[0148] Step A32: Based on the set of spatial parameters, each of the key anatomical structures is mapped into three-dimensional space to form a spatial distribution representation of each of the key anatomical structures.

[0149] Based on a set of spatial parameters, key anatomical structures are mapped to a unified three-dimensional or pseudo-three-dimensional spatial coordinate system through normalized coordinate transformation or affine transformation. In the three-dimensional or pseudo-three-dimensional space, each key anatomical structure is represented as a point or a region. These points or regions are used to construct a spatial distribution representation of the key anatomical structures, showing their position and shape in three-dimensional space.

[0150] Step A33: Render the spatial distribution representation of each of the key anatomical structures to generate a virtual anatomical model.

[0151] In this embodiment, the virtual anatomical model refers to a three-dimensional or pseudo-three-dimensional model constructed based on a set of spatial parameters of key anatomical structures, which can intuitively display the spatial distribution and interrelationships of anatomical structures. By using computer graphics technology (such as OpenGL or DirectX), the spatial distribution of each key anatomical structure is presented in a visual manner. Specific colors can be assigned to each key anatomical structure to facilitate the differentiation of different tissue types, thereby forming a virtual anatomical model.

[0152] In some implementations, the transparency and color contrast between key anatomical structures in the virtual anatomical model can be adjusted to form an optimized anatomical model, thereby more clearly showing the hierarchical relationships of key anatomical structures.

[0153] Step A40: Based on the virtual anatomical model and the fused standard image, determine the target scanning parameters of the ultrasound probe.

[0154] In this embodiment, the target scanning parameters refer to the parameters that the ultrasonic probe needs to achieve during the scanning process of the fused standard image, including the position and angle of the ultrasonic probe.

[0155] The virtual anatomical model is aligned and matched with a fused standard image to determine the cross-sectional image in the virtual anatomical model that matches the fused standard image. Based on the first position and first angle of the cross-sectional image in the virtual anatomical model, a coordinate transformation algorithm is used to convert the first position of the matched cross-sectional image into a second position of the ultrasound probe; and a direction vector transformation algorithm is used to convert the first angle of the matched cross-sectional image into a second angle of the ultrasound probe. The second position and second angle are the target scanning parameters that the ultrasound probe needs to achieve.

[0156] Step A50: Obtain the actual physical parameters of the ultrasound probe, and output the scanning guidance path based on the actual physical parameters and the target scanning parameters.

[0157] In this embodiment, the actual physical parameters refer to the physical parameters of the ultrasound probe during actual operation, including its current position, angle, and motion state. The scanning guidance path is the path generated from the actual physical parameters to the target scanning parameters, used to guide the doctor in adjusting the position and angle of the probe.

[0158] In some implementations, the actual physical parameters of the ultrasound probe, such as its position, angle, and motion, can be acquired in real time using sensors in the ultrasound equipment. Based on these actual physical parameters and the target scanning parameters, a scanning guidance path is generated to guide the scan from the actual physical parameters to the target scanning parameters. This scanning guidance path reduces the number of trial-and-error attempts required by the doctor during the scanning process, improving operational efficiency and accuracy.

[0159] In the ultrasound image processing method proposed in this application, firstly, by continuously acquiring high-frame-rate ultrasound image frames and forming a continuous target ultrasound image set, the accurate capture of dynamic changes in anatomical structures is ensured. Secondly, by analyzing the distribution of type labels (such as nerves, blood vessels, muscles, etc.) in each frame of the image, the position and proportion of key anatomical structures in each frame are determined. This not only improves the accuracy of localization but also provides reliable basic data for subsequent steps. Thirdly, based on the positional parameters of these key anatomical structures (including center coordinates, volume information, direction vectors, and depth information), a virtual anatomical model is constructed. This model integrates the positional parameters from multiple frames of images to form a more comprehensive and accurate set of spatial parameters, and these key anatomical structures are mapped onto three-dimensional space for rendering, intuitively displaying their spatial distribution and interrelationships. Fourthly, by aligning and matching the virtual anatomical model with the fused standard image, the target scanning parameters of the ultrasound probe (including position and angle) are determined, thereby guiding the doctor to adjust the probe to the optimal operating position. Fifthly, based on the actual physical parameters of the ultrasound probe and the target scanning parameters, a scanning guidance path is generated, reducing the number of trial and error attempts by the doctor and significantly improving the efficiency and accuracy of the scanning process.

[0160] In one embodiment, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the above method embodiments.

[0161] In one embodiment, an electronic device is also provided, including one or more processors; and a memory storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the steps in the above method embodiments.

[0162] In one embodiment, such as Figure 7 The diagram illustrates the structure of an electronic device used to implement an embodiment of this application. The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0163] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0164] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the instructions are executed by central processing unit (CPU) 701, the various method steps described in this application are performed.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0166] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, any of the embodiments claimed in this application can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. An ultrasound image processing method, characterized in that, The method includes: Acquire an initial ultrasound image and divide the initial ultrasound image into several sub-images; Extract global context information and local detail features from the several sub-image blocks; Based on the global context information and the local detailed features, a target feature map is generated; By performing a convolution operation on the target feature map, a type label is generated for each pixel in the initial ultrasound image. The type label includes one or more of the following: nerve, blood vessel, muscle, fascia, and bone. A segmentation mask for the initial ultrasound image is constructed based on the type label of each pixel, and the segmentation mask is superimposed on the initial ultrasound image to generate the target ultrasound image; Based on a preset regional anesthesia standard image set, a standard image subset corresponding to the target ultrasound image is determined. Each image in the standard image subset is fused to obtain a fused standard image. The fused standard image includes static anatomical structures, as well as the movement or deformation of nerves and blood vessels. The fused standard image and the target ultrasound image are matched to determine the scanning accuracy of the target ultrasound image. When the scanning accuracy is lower than a preset matching threshold, a continuous target ultrasound image set corresponding to consecutive frames of the initial ultrasound image is obtained. Based on the type label distribution of each frame in the continuous target ultrasound image set, the key anatomical structures of each frame in the continuous target ultrasound image set are determined. Based on the position parameters of each key anatomical structure, a virtual anatomical model is constructed. Based on the virtual anatomical model and the fused standard image, the target scanning parameters of the ultrasound probe are determined. The actual physical parameters of the ultrasound probe are obtained. Based on the actual physical parameters and the target scanning parameters, a scanning guidance path is output.

2. The ultrasound image processing method according to claim 1, characterized in that, The extraction of global context information and local detail features from the plurality of sub-image blocks includes: The aforementioned sub-image blocks are mapped to vector representations of a preset dimension; Add the positional encoding of the corresponding sub-image to the vector representation of each sub-image; Based on the positional encoding corresponding to each sub-image block, the global context information of the several sub-image blocks is calculated through a multi-head self-attention mechanism; Based on the global context information, a feedforward network is used to perform a nonlinear transformation on the vector representation of each sub-image to obtain the local detail features of the several sub-images.

3. The ultrasound image processing method according to claim 1, characterized in that, The step of generating a target feature map based on the global context information and the local detail features includes: The global context information and the local detail features are fused to generate a fused feature map; The fused feature map is input into the encoder network to generate a multi-level initial feature map; By using skip connections, the initial low-level feature maps are passed to the corresponding layers of the decoder network to obtain low-level features. Upsampling is performed on the initial high-level feature map to obtain high-level features; The low-level features and the high-level features are fused to generate a target feature map.

4. The ultrasound image processing method according to claim 1, characterized in that, The location information includes center coordinates, volume information, direction vector, and depth information. Before constructing the virtual anatomical model based on the location parameters of each of the key anatomical structures, the following is also included: Based on the boundary information of each of the key anatomical structures, calculate the center coordinates and volume information of each of the key anatomical structures; The orientation vector of each key anatomical structure is determined by fitting the principal axis of each key anatomical structure separately. The depth information of each of the key anatomical structures is determined based on the original grayscale values ​​of the target ultrasound image.

5. The ultrasound image processing method according to claim 4, characterized in that, The construction of the virtual anatomical model based on the positional parameters of each of the key anatomical structures further includes: The center coordinates, volume information, orientation vector and depth information of each key anatomical structure in each frame of the continuous target ultrasound image set are integrated to generate a set of spatial parameters for each key anatomical structure. Based on the set of spatial parameters, each of the key anatomical structures is mapped into three-dimensional space to form a spatial distribution representation of each of the key anatomical structures; The spatial distribution representation of each of the key anatomical structures is rendered to generate a virtual anatomical model.

6. The ultrasound image processing method according to any one of claims 1 to 5, characterized in that, After constructing a segmentation mask for the initial ultrasound image based on the type label of each pixel, and superimposing the segmentation mask on the initial ultrasound image to generate the target ultrasound image, the process further includes: The target anatomical structure is determined based on the type label distribution in the target ultrasound image; The target pressure range of the ultrasound probe is determined based on the tissue characteristics of the target anatomical structure. The real-time pressure of the ultrasound probe is acquired, and a correction prompt is output when the real-time pressure exceeds the target pressure range.

7. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the ultrasound image processing method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium stores executable instructions, which, when executed by a processor, cause the processor to perform the ultrasound image processing method as described in any one of claims 1 to 6.

Images