A method for recognizing brachial plexus ultrasound images based on target detection

Abstract

The application discloses a kind of based on target detection to the recognition method of brachial plexus ultrasonic image, the application is based on the characteristics and shortcomings of two-stage target detector in brachial plexus ultrasonic image related combination natural image, an effective two-stage target detection network IDNet is proposed, to effectively assist anesthetist to identify target nerve.In IDNet, the original network is improved, two novel module networks are proposed, the features in the feature extraction network are efficiently fused, and the edge features are strengthened according to the features recognized by the human eye, the combination of single-point features and edge features improves the quality of the proposal box.A large number of experiments on the UBPD dataset verify the effectiveness of the proposed target detector IDNet, and compared with other advanced methods, the proposed method achieves good results in the accuracy of ultrasonic image detection.

Description

Technical Field

[0001] This invention relates to the field of medical ultrasound imaging technology, and in particular to a method for identifying brachial plexus ultrasound images based on target detection. Background Technology

[0002] Peripheral nerve block (PNB) has a wide range of applications in clinical medicine, commonly used for anesthesia during surgery, postoperative analgesia, and the treatment of acute pain such as labor pains, with significant analgesic effects. Furthermore, PNB is minimally invasive and can prevent residual limb pain and phantom limb pain before amputation surgery. PNB is not only rapid and effective but also minimizes complications and adverse reactions, shortening hospital stays and rehabilitation time. However, while PNB offers many advantages, it also carries certain risks. Improper nerve placement or overly severe nerve manipulation can not only prevent the nerve from functioning properly but may also lead to permanent nerve damage.

[0003] In recent years, deep learning algorithms have been widely used in image processing. Using computer vision to assist in the recognition of medical images is a research direction with great application value and promising prospects. Peripheral nerve recognition in ultrasound images, as a pre-processor for assisting peripheral nerve blocks, is of great significance in terms of accuracy and efficiency. Currently, there are two mainstream object detection schemes in the field of deep learning. One is a two-stage method, mainly including R-CNN, Fast R-CNN, Faster R-CNN, etc. The idea of ​​this type of method is to first generate a candidate region (Region Proposal) using selective search or convolutional neural networks, then extract features from each candidate region, and finally classify and regress the feature vectors extracted from the generated candidate regions. The other is a one-stage object detection method, mainly including SSD and YOLO. This type of method uses an end-to-end deep convolutional neural network, directly inputting the entire image, and directly predicting the location and category of the object to be recognized based on the selection of different task targets, resulting in significant improvements in recognition speed.

[0004] For greater safety, nerve block therapy has evolved from manual blind tapping, including parapsychiatric stimulation, vascular pulsation, and penetrating blood vessels, to a dual-guided method that combines nerve stimulation localization, ultrasound visualization, and both.

[0005] However, ultrasound examination also has its limitations. Because ultrasound imaging utilizes sound waves, its objectivity is relatively poor, and image quality is easily affected by gases, bones, etc. Its resolution is also lower than CT and MRI. It is particularly difficult to perform ultrasound on deep, relatively small, poorly defined, and complexly adjacent nerve tissues. Therefore, identifying small tissues such as nerves on ultrasound images is quite difficult and can affect anesthesia. Thus, the identification and analysis of brachial plexus ultrasound images places high demands on anesthesiologists, requiring not only a clear and in-depth understanding of the complex structure of the brachial plexus but also extensive clinical diagnostic experience. Even highly experienced anesthesiologists can sometimes fail to identify targets during ultrasound image interpretation, leading to missed diagnoses or misdiagnoses. Summary of the Invention

[0006] The purpose of this invention is to provide a method for identifying brachial plexus ultrasound images based on target detection. This invention aims to leverage the standards for ultrasound image recognition in computer vision technology, combined with the characteristics of clinical applications of brachial plexus ultrasound images. Starting from a target detection framework integrating clinical medical knowledge of the brachial plexus, the problem is expanded to multi-target detection, including medical structures such as nerves, arteries, veins, and muscle tissue. Accuracy is improved by comparing with peripheral tissues. Finally, a deep learning-based multi-target ultrasound image detector is established to automatically identify key tissue structures in patient ultrasound images during actual peripheral nerve block scenarios, assisting anesthesiologists in analysis. This significantly reduces the time anesthesiologists spend on ultrasound image recognition, and also improves recognition accuracy, clinical work efficiency, and anesthesia safety.

[0007] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0008] This invention adds a fully connected pyramid feature network (FCFPN) and a mean-iterative region generation network to form a new two-stage object detector, IDNet. The two-stage object detector IDNet consists of a feature extraction network (VGG-19), a fully connected feature pyramid network based on deconvolution, a mean-iterative region generation network, and a head structure. The input of the two-stage object detector IDNet is an ultrasound image of the brachial plexus. After processing by the two-stage object detector IDNet, the final output of the ultrasound image is the detection bounding box corresponding to the image, the corresponding classification label of the detection bounding box, and the probability of belonging to that category.

[0009] The feature extraction network VGG-19 extracts feature maps of different sizes and semantic intensities from the original brachial plexus ultrasound images. The feature extraction network VGG-19 has 5 stages, each stage contains convolution and pooling operations, and after the pooling operation of each stage, it will output a feature map of the corresponding size.

[0010] The deconvolution-based fully connected feature pyramid network fuses and further extracts the obtained feature maps. Through top-down, bottom-up, lateral connections, and operations similar to fully connected connections, it increases the expressive power of high-level features with smaller resolution and low-level features with larger resolution, outputting a series of new feature maps.

[0011] The mean-iterative region generation network obtains a series of efficient bounding box proposals: First, each feature point on different feature maps is used as an anchor point to generate default boxes with different aspect ratios. The predicted offsets are used to refine the default boxes to obtain modified boxes. Then, the edge features of the modified boxes are used to predict the offsets, and finally, the proposals are obtained.

[0012] The head network is responsible for detection and classification tasks, including classifiers and regressors. First, non-maximum suppression is used to filter out ROIs and separate positive and negative samples. Then, ROI Align is used to fix the corresponding feature maps to the same size. The aligned proposals are then classified and the bounding boxes are regressed through classification and regression branches.

[0013] The beneficial effects of this invention are:

[0014] This invention presents a method for identifying brachial plexus nerves in ultrasound images based on target detection. Compared with existing technologies, this invention proposes an effective two-stage target detection network, IDNet, based on knowledge related to brachial plexus nerve ultrasound images and the characteristics and shortcomings of two-stage target detectors in natural images. This network effectively assists anesthesiologists in identifying target nerves. IDNet improves upon existing networks by introducing two novel modular networks. It efficiently fuses features from the feature extraction network and enhances edge features based on human visual recognition characteristics. The combination of single-point and edge features improves the quality of bounding box proposals. Extensive experiments on the UBPD dataset validate the effectiveness of the proposed target detector, IDNet. Compared with other state-of-the-art methods, the proposed method achieves superior accuracy in ultrasound image detection. Attached Figure Description

[0015] Figure 1 This is a diagram of the IDNet detector structure of the present invention;

[0016] Figure 2 This is a structural diagram of the VGG-19 feature extraction network of the present invention;

[0017] Figure 3 This is a diagram of the fully connected feature pyramid network structure based on deconvolution of the present invention;

[0018] Figure 4 This is a diagram of the mean-iterative region generation network structure of the present invention;

[0019] Figure 5 This is a diagram of the head structure of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0021] like Figure 1 As shown, this invention is based on the classic two-stage object detector Mask R-CNN. Taking into account the characteristics of medical images, it designs two modules: a fully-connected feature pyramid network (FCFPN) and a mean iteration region proposal network (MIRPN). These modules are added to the network to form a new two-stage object detector, IDNet. IDNet mainly consists of four parts: a feature extraction network VGG-19, a fully-connected feature pyramid network (FCFPN) based on deconvolution, a mean iteration region proposal network (MIRPN), and a head architecture. The input to the network model is an ultrasound image of the brachial plexus. This ultrasound image undergoes a series of operations, ultimately outputting the corresponding detection bounding box (bbox), the corresponding class label, and the probability of belonging to that class.

[0022] like Figure 2 As shown: The feature extraction network VGG-19 extracts feature maps of different sizes and semantic intensities from the original brachial plexus ultrasound images. VGG-19 has five stages, each containing a series of convolutional and pooling operations, and each stage outputs a feature map of the corresponding size after the pooling operation. The five stages output C1 (320×320), C2 (160×160), C3 (80×80), C4 (40×40), and C5 (20×20).

[0023] like Figure 3As shown: Fully Connected Feature Pyramid Network: This part fuses and further extracts the obtained C1, C2, C3, C4, and C5 feature maps. Through top-down, bottom-up, lateral connections, and operations similar to fully connected networks, it increases the expressive power of high-level and low-level features, outputting a series of new feature maps. High-level feature maps with smaller resolution (larger stride) are beneficial for classification, while low-level feature maps with larger resolution (smaller stride) are beneficial for localization. Furthermore, high-resolution feature maps are suitable for detecting small objects, and high-resolution feature maps are suitable for detecting large objects. The sizes are P2 (160×160), P3 (80×80), P4 (40×40), P5 (20×20), and P6 (10×10), respectively.

[0024] like Figure 4 As shown: Mean Iterative Region Generation Network: This part obtains a series of efficient bounding box proposals. First, each feature point on different feature maps is used as an anchor point to generate default boxes with different aspect ratios. The predicted offsets are used to refine the default boxes to obtain modified boxes. Then, the edge features of the modified boxes are used to predict the offsets, ultimately resulting in 2000 proposals. The proposals obtained through iteration are more representative of the features, which is beneficial for subsequent operations. Six default box anchors with different aspect ratios are generated for each feature point. For each feature point on a high-level feature map, six default boxes with different aspect ratios but a size of 512 are generated; for each feature point on a low-level feature map, six default boxes with different aspect ratios but a size of 128 are generated. The aspect ratios are 0.5, 0.8, 1, 1.5, 2, and 2.5.

[0025] Offset prediction and default value refinement:

[0026] Each point on the feature map has generated 6 default boxes with different aspect ratios, all located on the original image. In the first stage of MIRPN, the traditional RPN network operation is used. First, a 3x3 convolution is applied to the input feature map Pi at a certain level, with 512 kernels, resulting in a shared feature map with 512 channels while maintaining the same size. This makes the extracted features more robust. After obtaining the shared feature map, consider the regression branch above. A 1x1 convolutional layer with 24 kernels is applied to predict the offset of each box. The reason for using 24 kernels is that each feature point has 6 default boxes, and each default box requires 4 offsets, so for a point there are 24 offsets, meaning the number of channels in the generated new feature map is 4x6 = 24. For the box scoring branch, a 1x1 convolution is also used with 12 kernels because each box has foreground and background confidence levels, resulting in a new feature map with 12 channels. Because 1x1 convolution does not change the size of the feature map, the size of the new feature map remains unchanged.

[0027] For the second stage of MIRPN, the offset obtained in the first stage is first applied to each corresponding default box to adjust it and obtain the modified box (the modified box is also on the original image). The feature map Pi (the feature layer level here corresponds to the feature layer level input in the first stage of MIRPN) is processed by 1x1 convolution to obtain the "box-sensitive" feature map. As the name suggests, it corresponds to the modified box and has 6C channels (the reason for 6C channels is that each feature point has 6 modified boxes, where C is the number of channels of the input feature map Pi. The influence of the value of C on the results will be discussed in the experimental section later). Next, for the four boundaries of each modified box on the original image, M points (M represents the number of points on the boundary, M=3 in this invention) are taken on average from each boundary except for the endpoints. The coordinates of these points are mapped from the original image to the "box-sensitive" feature map, and the feature values ​​of the mapped coordinates are averaged. For ease of description, this invention divides the "box-sensitive" feature map into 6 colors with each C channel, named L1, L2, L3, L4, L5, and L6 respectively. Figure 4As shown in the dashed box, for feature map L3 (green cube) with C channels in the "box-sensitive" feature map, there is a corresponding modification box. By averaging the feature values ​​of three points on the four boundaries of this modification box (top, bottom, left, and right), four feature maps with C channels are obtained. These four feature maps correspond to the four boundaries. It is worth noting that this invention uses a channel-based approach; the six modification boxes are calculated independently and operated identically within each of the C channels of the "box-sensitive" feature map. The output feature map of each box can be expressed as the following equations: Fleft=1 / M∑vb MK=1(x0,y0-hk / (M+1)) Fright=1 / M∑vb MK=1(x1,y0-hk / (M+1)) Ftop=1 / M∑vb MK=1(x0+wk / (M+1),y0) Fbottom=1 / M∑vb MK=1(x0+wk / (M+1),y1) (3-2) where Fleft is the feature value after averaging the M points of the left boundary, (x0,y0,x1,y1) are the coordinates of the top left and bottom right corners of the corresponding modified box, and w and h are the width and height of the modified box. To avoid quantization errors, the accurate value vb is calculated using bilinear interpolation of the nearest neighbor feature points. After obtaining the average boundary feature map of the modified box, this invention performs concatenation along the channel dimension. Because each average boundary feature map has C channels, each feature point generates 6 default anchors, and each anchor generates 4 average boundary points (top, bottom, left, and right), the number of channels in the concatenated feature map becomes 24C. The number of channels in the feature map becomes very large, but subsequent operations only involve 1x1 convolutions, without introducing additional computation. At this point, a 34x1 convolution is performed to predict the offset and foreground / background confidence again. It's important to note that this offset is for the modified bounding box, allowing for more precise positional adjustments. The remaining operations are the same as in RPN. The output of the MIRPN network is the same as the output of the traditional RPN network—proposal location coordinates. However, the proposals predicted iteratively by the MIRPN network are more representative.

[0028] like Figure 5 As shown: Head Network: This module is responsible for detection and classification tasks, including a classifier and a regressor. First, non-maximum suppression is used to filter out ROIs and distinguish between positive and negative samples. Then, ROI Align is used to fix the corresponding feature maps to the same 7x7 size. The aligned proposals are then classified (N-class classification) and the bounding boxes (bboxes) are detected and regressed through classification and regression branches.

[0029] Detailed operating procedures and comparative analysis:

[0030] Feature extraction network comparison experiment:

[0031] This paper compares the current mainstream feature extraction networks ResNet101, ResNet50, and VGG-19. To reduce parameters, during training, this invention reduces the number of feature channels in all convolutional and pooling layers to half that of the original network (by reducing the number of convolutional kernels). The detailed structures of the three feature extraction networks, ResNet-101, ResNet-50, and VGG-19, are shown in Table 3-1. It can be seen that all three feature extraction networks have five stages. Unlike VGG-19, the ResNet series feature extraction networks use 7x7 convolutional kernels in the first stage, and have a larger number of kernels. From the second stage onwards, the ResNet series feature extraction networks do not use pooling operations, but only use convolution to downsample the feature maps. After comparison, VGG was ultimately adopted.

[0032] Comparison experiment of fully connected feature pyramid upsampling methods:

[0033] Two common upsampling methods, bilinear interpolation and deconvolution, are compared. All other network settings remain unchanged, and the tests are performed on the newly proposed FCFPN+MIRPN module, i.e., IDNet.

[0034]

[0035] Deconvolution is used.

[0036] Comparison Experiment of Boundary Points of Mean Iteration Region Generation Network

[0037] M is a hyperparameter in the proposed MIRPN module, representing the number of points on a boundary of the modification box, mainly used to enhance boundary information. In this experiment, the proposed IDNet network was used as the baseline network, and different M values ​​were evaluated to obtain the optimal M value, which was set to 2, 3, 4 and 5 respectively.

[0038]

[0039] In all other experiments, the value of M was set to 3.

[0040] Experiment comparing the number of channels in the mean-iterative region generation network:

[0041] In feature extraction networks, the number of channels varies across feature maps. However, due to the weight-sharing mechanism of the Mean Iterative Region Generation (MIRPN) network, the number of channels in feature maps must be the same. Therefore, in the FCFPN network, the number of channels in all feature maps is set to be the same, denoted as C. The experiments in this section evaluate the impact of the number of channels in the Mean Iterative Region Generation (MIRPN) network on detection performance. The proposed network uses three different numbers of channels: 128, 256, and 512.

[0042]

[0043] A channel count of 256 yields the best results.

[0044] Compared with other advanced methods:

[0045] Since all experiments in this invention were conducted on the UBPD dataset, this invention trains other advanced methods on the UBPD dataset for comparison. This invention compares the detection performance experimental results of different commonly used classic one-stage and two-stage object detection methods, including YoLACT

[49] , Grid RCNN

[50] , Faster RCNN, Cascade RCNN

[51] , SSD, DSSD, ConerNet

[52] , and RetinaNet. There are also some methods for ultrasound image recognition of kidneys, thyroid glands, and bovine eggs, such as the improved YOLOv3

[53] model, DYOLO

[55] , and the cascade RCNN model with SE-Resnet as the backbone.

[0046]

[0047] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A method for recognizing brachial plexus nerve ultrasound images based on target detection, characterized in that: A new two-stage object detector, IDNet, is formed by adding a fully connected pyramid feature network (FCFPN) and a mean-iterative region generation network (MRI). IDNet consists of a feature extraction network (VGG-19), a fully connected feature pyramid network based on deconvolution, a mean-iterative region generation network, and a head structure. The input to IDNet is an ultrasound image of the brachial plexus. After processing by IDNet, the ultrasound image finally outputs the detection bounding box corresponding to the image, the corresponding classification label of the detection bounding box, and the probability of belonging to that category. The feature extraction network VGG-19 extracts feature maps of different sizes and semantic intensities from the original brachial plexus ultrasound images. The feature extraction network VGG-19 has 5 stages, each stage contains convolution and pooling operations, and after the pooling operation of each stage, it will output a feature map of the corresponding size. The deconvolution-based fully connected feature pyramid network fuses and further extracts the obtained feature maps. Through top-down, bottom-up, lateral connections, and operations similar to fully connected connections, it increases the expressive power of high-level features with smaller resolution and low-level features with larger resolution, outputting a series of new feature maps.

2. The method for recognizing brachial plexus ultrasound images based on target detection according to claim 1, characterized in that: The mean-iterative region generation network obtains a series of efficient bounding box proposals: First, each feature point on different feature maps is used as an anchor point to generate default boxes with different aspect ratios. The predicted offsets are used to refine the default boxes to obtain modified boxes. Then, the edge features of the modified boxes are used to predict the offsets, and finally, the proposals are obtained.

3. The method for recognizing brachial plexus ultrasound images based on target detection according to claim 2, characterized in that: The head network is responsible for detection and classification tasks, including classifiers and regressors. First, non-maximum suppression is used to filter out ROIs and separate positive and negative samples. Then, ROI Align is used to fix the corresponding feature maps to the same size. The aligned proposals are then classified and the bounding boxes are regressed through classification and regression branches.

Images