Two-dimensional ultrasonic bone contour end-to-end real-time segmentation method

Abstract

The invention provides a two-dimensional ultrasonic skeleton contour end-to-end real-time segmentation algorithm, which is characterized in that an end-to-end network (BoneRTNet) is provided and is used for segmenting a skeleton surface below in real time. According to the BoneRTNet, a standard U-net network is expanded, and a new feature extraction component, namely, a Squeeze-and-Excitation residual block, is added, so that robust feature learning is realized. Meanwhile, in order to relieve the challenge of foreground and background class height imbalance in a skeleton surface segmentation task in an ultrasonic image, a new loss function strategy is adopted in the training process. In addition, depth divisible convolution is introduced, so that network parameters and calculation cost are reduced, and real-time performance is guaranteed. According to the algorithm, the segmentation of the bone surface in the ultrasonic image can be realized by taking the original ultrasonic image as the input without any preprocessing.

Description

Technical Field

[0001] This invention belongs to the field of computer vision in computer-aided surgical navigation systems, and specifically relates to a deep learning-based end-to-end real-time segmentation method for two-dimensional ultrasonic skeletal contours. Background Technology

[0002] Fractures are a common surgical condition that seriously affects people's health and lives. With the increase in the types of outdoor sports, sports-related injuries are becoming more frequent, and the rising rate of traffic accidents is also leading to a gradual increase in fracture patients.

[0003] Orthopedic surgery is a crucial solution for treating fractures. High-precision intraoperative navigation technology is key to reducing postoperative complications in fracture patients. Computer-assisted orthopedic surgery (CAOS) systems are an effective aid for imaging and locating fracture sites, with imaging being the most important component. Currently, CAOS uses 2D / 3D fluorescence fluoroscopy for intraoperative imaging. However, 2D fluorescence fluoroscopy provides limited three-dimensional information about the fracture site, requiring surgeons to take multiple images from different angles during surgery to obtain structural and positional information for planning subsequent reduction procedures. Newer 3D fluorescence imaging can effectively address this issue, but its cost is twice that of 2D fluorescence imaging and it is not widely used. Furthermore, both of these fluoroscopic imaging methods involve ionizing radiation, which can harm the health of the medical team and the patient.

[0004] Ultrasound imaging can provide a real-time, radiation-free imaging method for CAOS (Cyclic Obstructive Surgery) to replace intraoperative fluoroscopy, and some studies have already attempted to incorporate ultrasound into CAOS systems. Currently, the mainstream method is to register intraoperative ultrasound images with high-precision 3D images from preoperative CT / MRI. However, the low signal-to-noise ratio, artifacts, severe speckle, low contrast, and difficulty in interpretation of ultrasound images make accurate bone segmentation from ultrasound images extremely difficult, severely limiting its application and development in CAOS systems. Therefore, how to accurately and in real-time extract bone segmentation from ultrasound images has become one of the key technologies (breakthrough points) for applying ultrasound imaging to CAOS systems.

[0005] Early research on bone edge segmentation methods under ultrasound images mainly relied on heuristic functions based on image morphology, intensity, and phase features. However, this method has poor robustness and long processing time. Subsequently, segmentation methods using machine learning (such as random forests) emerged. These methods have significantly improved segmentation accuracy compared to traditional heuristic processing methods. However, machine learning methods often require additional prior knowledge and good feature engineering to achieve good accuracy.

[0006] In recent years, methods based on convolutional neural networks (CNNs) have been proposed one after another. These methods have significantly improved accuracy, speed, and generalization ability compared to previous methods. However, existing CNN-based methods generally require different levels of preprocessing of ultrasound images, which takes an average of 1 to 2 seconds per image. After preprocessing, segmentation is performed, which makes it difficult for the algorithm's performance to meet the real-time requirements of the imaging system in CAOS.

[0007] To address the aforementioned issues, this invention proposes an end-to-end network (BoneRTNet) for real-time segmentation of the underlying bone surface. BoneRTNet extends the standard U-net network by adding a new feature extraction component—the Squeeze-and-Excitation residual block—to achieve robust feature learning. Simultaneously, to alleviate the challenge of high foreground-background class imbalance in ultrasound image bone surface segmentation, a novel loss function strategy is employed during training. Furthermore, depthwise separable convolution is introduced to reduce network parameters and computational cost, ensuring real-time performance. This algorithm enables bone surface segmentation from ultrasound images using the raw US image as input without any preprocessing. Summary of the Invention

[0008] (I) Purpose of the Invention: To achieve real-time segmentation of bone surfaces under ultrasound imaging, this invention conducts in-depth research on the characteristics of ultrasound imaging and image segmentation algorithms based on convolutional neural networks, proposing an end-to-end bone surface segmentation network under ultrasound images. To address the imbalance problem of bone surface samples under ultrasound images, we introduce a binary weighted cross-entropy of focus loss and class balance loss functions to reduce noise interference and obtain accurate and clean bone surface segmentation results. To address the potential annotation discrepancies in the "Ultrasound-based Bone Dataset," we employ instance normalization to mitigate the impact of the dataset.

[0009] (II) Technical Solution: The key part of this invention lies in the processing of ultrasound medical images, namely, how to accurately and in real time extract bone pixels from ultrasound images. The specific steps of each part are as follows.

[0010] Acquiring ultrasound images of the bone surface: Ultrasound images are formed by measuring the amplitude and propagation time of the reflected echoes reaching the ultrasound transducer. High-intensity pixel features in ultrasound images likely indicate the presence of boundaries, and bone has the highest acoustic impedance. Therefore, most ultrasound signals will be reflected back from the bone surface, forming high-intensity features in the ultrasound image. Thus, an ultrasound probe should first be used to acquire ultrasound images of the bone surface, and the acquired images should be denoised and cleaned.

[0011] Central cropping of images: To facilitate processing, all acquired ultrasound images are uniformly center-cropped to remove redundant display information. Subsequent images only contain the imaging area of ​​the ultrasound image, which facilitates image processing and greatly reduces computational load. The size of the cropped image is Height: 288 pixels, Width: 320 pixels.

[0012] Labeling: The segmentation network uses supervised learning, requiring the creation of a corresponding ground truth for each image. First, an ultrasound radiologist uses software to label the bone surfaces in the images: the label should use a curve with a width of 5 pixels, placing the bone surface pixels at the center of this curve. Converting the generated label file into a binary mask image completes the label creation.

[0013] Expanding the sample size: Image augmentation techniques can be used to process existing datasets in various ways, aiming to enrich the data content and expand the sample size. Through various augmentation operations, such as rotation, flipping, scaling, cropping, adjusting brightness and contrast, adding noise, and color transformation, a large number of diverse new images are generated. The augmented images not only retain the basic features of the original data but also introduce different visual variations, helping the model to better generalize and adapt to different scenarios. Furthermore, image augmentation techniques can help solve the problem of imbalanced data, making the sample size more even across categories, thereby improving the training effect and robustness of the model.

[0014] The dataset is randomly divided into 5 parts so that it can be alternated between training and validation sets in a 4:1 ratio.

[0015] Building the Network Model: The neural network model is shown in Figure 2, consisting of two main modules: an encoding module and a decoding module. The SE module (highlighting effective features) shown in Figure 3 is used in the encoding stage, and the bottom of the model uses... Figure 4 The inverse residual block network shown reduces the number of parameters. To mitigate the impact of annotation discrepancies on network performance during sample labeling, instance normalization is introduced. The specific algorithm of the SE module is as follows: A residual module based on Squeeze-and-Excitation (SE) is introduced into the feature extraction part of the network. This module was originally used in the ImageNet image classification task. It has strong feature representation capabilities by modeling the correlation between channels of the feature map after convolution. Therefore, it is hoped that by detecting coarse-grained context features in shallow layers and locating fine-grained features in deep layers, weaker features can be suppressed with minimal computational burden, resulting in better segmentation performance.

[0016] The expression for the SE convolutional residual block is shown in formula (1): Z=\left [ {{z}_{1},{z}_{2},...,{z}_{k},...,{z}_{k}} \right ] (1) S=\left [ {{s}_{1},{s}_{2},...,{s}_{k},...,{s}_{k}} \right ] {X}^{'}=\left [ {{x}_{1},{x}_{2},...,{x}_{k},...,{x}_{K}} \right ] in This indicates that the input... X After standard convolution operation The output obtained later U One of the channels. It's the Squeeze function—for Perform global average pooling. H , W These represent each channel in the feature map. Height and width. It is the excitation function, consisting of two fully connected layers (with weights of ). and ) is parameterized, where the activation function is and .

[0017] Validate network model Using the previously obtained five datasets, the network model was trained with 5-fold cross-validation. To enhance the model's generalization ability, the brightness, contrast, saturation, and chroma of the original ultrasound images were randomly changed during training. The target region pixels on the bone surface account for only 0.65% of the entire ultrasound image, and due to poor ultrasound imaging quality and severe speckle, the network struggles to accurately segment these target pixels. Therefore, directly using the binary cross-entropy loss function may lead to degraded network performance or even failure. To address this, during BoneRTNet training, a focus loss function was first used to quickly classify easily distinguishable samples, enabling the network to achieve initial convergence. Then, a weighted binary cross-entropy loss function was used to learn the class distribution, ultimately leading to network convergence. Through these steps, the model's training process and performance were further optimized.

[0018] The expression for focus loss is shown in equation (2): (2) in i It is the index value of the input image pixels. N It represents the total number of pixels. Representing pixels i The true category, It is pixels i The output obtained after passing through the network is a probability prediction value, which is between [0,1]. These are introduced hyperparameters. The goal is to balance the proportion of positive and negative samples in the total loss, in order to solve the problem of the loss function value being artificially low due to sample imbalance, which leads to network optimization failure. Its function is to adjust the loss function ratio for hard-to-classify samples, prompting the network to focus its attention on hard-to-classify samples.

[0019] The weighted binary cross-entropy loss function is expressed as shown in Equation 3: (3) in, Weights used to adjust the balance between positive and negative class samples. The meanings of the other parameters are the same as in... Same as above.

[0020] (III) Beneficial Effects Compared with the prior art, the present invention has the following advantages: 1. Compared with methods that segment skeletal contours using X-ray images, this invention utilizes two-dimensional ultrasound imaging to achieve radiation-free real-time segmentation, without causing radiation harm to patients and doctors; 2. Compared with similar deep learning-based algorithms, this method achieves accurate end-to-end segmentation results and is real-time, achieving both high-speed segmentation and high accuracy. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method involved in the present invention.

[0022] Figure 2 The semantic segmentation network structure diagram designed for this invention.

[0023] Figure 3 This is a schematic diagram of the squeeze-and-excitation block network involved in this invention.

[0024] Figure 4 This is a schematic diagram of the inverted residual block network designed for this invention. Detailed Implementation

[0025] This method is based on convolutional neural networks to segment and extract the skeletal surface in ultrasound images. The implementation method of this invention will be further described in detail below with reference to specific embodiments.

[0026] 1. Hardware configuration environment: (1) A computer with an image processing card (2080Ti), (2) Mindray DP10 portable ultrasound imager.

[0027] 2. Software configuration environment: Python, PyTorch, OpenCV.

[0028] like Figure 1 As shown, this invention proposes a bone surface segmentation method based on deep learning in ultrasound images, which specifically includes the following steps.

[0029] Step 1: Use an ultrasound probe to acquire ultrasound images of the bone surface and clean the acquired images (e.g., delete images with poor image quality).

[0030] Step 2: The original ultrasound image is cropped from the center to obtain an ultrasound imaging area of ​​288*320, and useless areas are removed.

[0031] Step 3: Label the bone ultrasound images to create masked annotation images, thereby creating a dataset.

[0032] Step 4: Use image enhancement techniques to enhance the dataset obtained in Step 3, enrich the data content, and expand the number of samples.

[0033] Step 4.1: Perform random distortion processing on the original ultrasound image with a probability of 0.3.

[0034] Step 4.2: With a probability of 0.5, perform a mirror flip process on the original ultrasound image and the masked image in the X-axis direction.

[0035] Step 4.3: Rotate the original ultrasound image and the masked image at the center with a probability of 0.3 by ±15 degrees.

[0036] Step 5: Divide the dataset into five equal parts randomly.

[0037] Step 6: Construct the neural network model as shown in Figure 2. It consists of two main modules: an encoding module and a decoding module. The encoding stage uses the module shown in Figure 3 (to highlight effective features), and the bottom of the model uses the inverted residual block shown in Figure 4 (to reduce the number of parameters).

[0038] Step 6.1: To mitigate the impact of annotation differences on network performance when annotating samples, instance normalization is introduced.

[0039] Step 7: Using the five datasets obtained in Step 5, perform five-fold cross-validation training on the network model.

[0040] Step 7.1: To promote network convergence, the input data is normalized.

[0041] Step 7.2: To enhance the model's generalization ability, the brightness, contrast, saturation, and chroma of the original ultrasound images are randomly changed during training.

[0042] Step 7.3: To alleviate the class imbalance problem in this task, a binary weighted cross-entropy of focus loss and class balance loss function can be used.

[0043] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention.

Claims

1. A two-dimensional ultrasound bone contour end-to-end real-time segmentation method, characterized in that, Includes the following steps: Step 1: Obtain ultrasound images of the bone surface; Step 2: Crop the image centrally so that the image only contains the imaging area of ​​the ultrasound image; Step 3: Create labels to mark the bone surfaces in the image; Step 4: Expand the number of samples by performing various enhancement operations, such as rotation, flipping, scaling, cropping, adjusting brightness and contrast, adding noise, and color transformation, to generate a large number of new images with diversity. Step 5: Randomly divide the dataset into 5 parts so that it can be divided into training and validation sets in a 4:1 ratio later; Step 6: Build the network model; Step 7: Validate the network model.

2. The two-dimensional ultrasound bone contour end-to-end real-time segmentation method according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Mark the curve with a width of 5 pixels; Step 3.2: Place the bone surface pixels at the center of the marker curve; Step 3.3: Convert the generated file into a binary mask image.

3. The two-dimensional ultrasound bone contour end-to-end real-time segmentation method according to claim 1, characterized in that, Step 6 specifically involves: Step 6.1: The network model consists of two main modules: an encoding module and a decoding module; Step 6.2: Use the Squeeze-and-Excitation module during the encoding phase; Step 6.3: Use an inverted residual block network at the bottom of the model; Step 6.4: To mitigate the impact of annotation differences on network performance when annotating samples, instance normalization is introduced.

4. The two-dimensional ultrasound bone contour end-to-end real-time segmentation method according to claim 1, characterized in that, Step 7 specifically involves: Step 7.1: Using the five datasets obtained previously, perform 5-fold cross-validation training on the network model; Step 7.2: To enhance the model's generalization ability, the brightness, contrast, saturation, and chroma of the original ultrasound images are randomly changed during training; Step 7.3: To alleviate the class imbalance problem in this task, a binary weighted cross-entropy of focus loss and class balance loss function can be used.

Images