A pharyngeal image segmentation and localization method based on deeplabv3+ network

Abstract

This invention discloses a pharynx image segmentation and localization method based on the DeepLab V3+ network. The method comprises the following steps: using a depth camera to obtain a color image and a depth map of the pharynx; processing the color image using a segmentation model constructed based on the DeepLab V3+ network to obtain an image segmentation result; using a boundary detection operator to obtain the pixel coordinates of the pharynx in the image segmentation result; obtaining the depth value of the pharynx based on the pixel coordinates of the pharynx based on the depth map; and obtaining the position information of the pharynx in the world coordinate system based on the pixel coordinates and depth value of the pharynx according to the calibration parameters of the depth camera to complete the localization. This invention can improve the accuracy of image segmentation and localization.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning medical image segmentation, and specifically relates to a pharyngeal image segmentation and positioning method based on the deeplabv3+ network. Background Art

[0002] Currently, the most accurate and effective test for coronavirus is nucleic acid testing, which involves nasopharyngeal swabs, which are more common than nasal swabs. However, nucleic acid collection is mostly done manually, and the large-scale, high-intensity workload presents risks. First, there is the risk of cross-infection among medical staff; second, the high-intensity workload can affect their physical and mental health. To address these challenges, automating nucleic acid sampling is essential. This involves using robotic arms or robots to partially replace medical staff in sampling, and establishing a pharyngeal image segmentation model to assist in detecting the sampling site and implementing the sampling process.

[0003] Regarding research on pharyngeal image segmentation technology, from the perspective of datasets, there are currently few publicly available datasets for pharyngeal images, and pharyngeal image segmentation scenarios are often subject to interference such as light and cotton swab occlusion, so a standard dataset is needed. From the perspective of image segmentation, current research mainly uses two segmentation models, Mask R-CNN and UNET, for instance segmentation. However, these instance segmentation methods have some problems. For example, although Mask R-CNN has fast detection speed, it does not perform well on noisy images. In related studies using Mask R-CNN, the segmentation target is the tonsils (small in size), ignoring the interference of tonsillar disease on the segmentation effect. In related studies using UNET, detection speed is not considered as an indicator, and UNET is also not effective for extracting multi-scale information. Therefore, there is an urgent need for an effective segmentation model to address the segmentation and localization of pharyngeal images. This will not only rapidly promote the development of automated nucleic acid sampling, but also have great application prospects in conventional medical fields (such as nasopharyngeal carcinoma image segmentation and laryngopharyngeal endoscopy). Summary of the Invention

[0004] The purpose of the present invention is to provide a pharyngeal image segmentation and positioning method based on the deeplabv3+ network to improve the image segmentation and positioning accuracy.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A pharyngeal image segmentation and positioning method based on a deeplabv3+ network, comprising:

[0007] Step 1: Use a depth camera to obtain a color image and a depth image of the pharynx;

[0008] Step 2: Based on the color image, the segmentation model built based on the deeplabv3+ network is used to process the image to obtain the image segmentation result;

[0009] Step 3: Use the boundary detection operator to obtain the pixel coordinates of the pharynx in the image segmentation result;

[0010] Step 4: Based on the depth map, obtain the depth value of the pharynx according to the pixel coordinates of the pharynx;

[0011] Step 5: According to the calibration parameters of the depth camera, the position information of the pharynx in the world coordinate system is obtained based on the pixel coordinates and depth value of the pharynx to complete the positioning.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0013] Preferably, the color image is processed using a segmentation model built based on the deeplabv3+ network to obtain an image segmentation result, including:

[0014] The deeplabv3+ network is trained to obtain a segmentation model, wherein the deeplabv3+ network is mainly an encoder-decoder structure;

[0015] In the encoder part, depthwise separable convolution is used. The backbone network extracts features to obtain preliminary image features. Then, the parallel dilated convolution layer in the enhanced feature extraction ASPP module extracts different spatial information of the preliminary image features and performs feature fusion to obtain multi-scale features.

[0016] In the decoder part, the multi-scale features extracted by the enhanced feature extraction ASPP module are first upsampled by 4 times through bilinear interpolation, then spliced and fused with the preliminary image features extracted from the backbone network, and finally restored to the high-level feature map by 4 times bilinear interpolation upsampling to obtain the final image segmentation result.

[0017] Preferably, the backbone network is an improved MobilenetV2 network, which adopts the Bottleneck structure of the original MobilenetV2 network, adds an attention mechanism to the inverted residual structure with stride=1, and adds parallel branches with different expansion rates to the linear bottleneck structure with stride=2 to obtain multi-scale features.

[0018] Preferably, the adding of an attention mechanism to the inverted residual structure with stride=1 includes:

[0019] At the output end of the inverted residual structure with stride=1, two branches of the channel attention mechanism and the spatial attention mechanism are added in parallel. Two attention feature maps are generated for the feature map output by the inverted residual structure with stride=1 respectively. The attention feature maps of the two branches are fused, and then the final output attention feature map is obtained through the sigmoid activation function.

[0020] Preferably, the step of adding parallel branches with different expansion rates to the linear bottleneck structure with stride=2 to obtain multi-scale features includes:

[0021] In the linear bottleneck structure with stride = 2, the third convolution layer is first replaced with average pooling. Then, three 3×3 convolution parallel branches with expansion coefficients of 2, 4, and 6 are added to the output end of the linear bottleneck structure to perform multi-scale feature extraction. The obtained multi-scale feature maps are fused to obtain the final multi-scale feature map.

[0022] Preferably, the segmentation model is obtained by training the deeplabv3+ network, comprising:

[0023] Acquiring pharyngeal images to generate a training data set, wherein the pharyngeal images include normal pharyngeal images and pharyngeal infection images;

[0024] Use Adam optimizer to train the deeplabv3+ network. During the training process, when the loss value converges within a number of consecutive iterations, stop the model training and output it as the segmentation model.

[0025] The loss function Loss used in the loss value calculation is as follows:

[0026] Loss=λLoss FL +(1-λ)Loss DSC

[0027] Among them, Loss FL Represents the Focal Loss loss function, Loss DSC represents the DSC loss function, and λ is the weight factor.

[0028] Preferably, the collecting pharyngeal images to generate a training data set comprises:

[0029] De-reflection processing was performed on the pharyngeal images;

[0030] The de-reflected pharyngeal image is cropped to retain the segmented area;

[0031] The segmented areas are labeled, and data enhancement processing is performed after labeling to obtain a training data set.

[0032] Preferably, the de-reflection processing of the pharyngeal image comprises:

[0033] Performing reflection detection on the pharyngeal image to obtain the highlight area in the pharyngeal image;

[0034] The detected highlight area is filled with the pixel average value of the circular area formed by the circles with radii of 2 and 4 pixels around it to obtain the filled image;

[0035] Perform median filtering on the filled image to obtain a smooth non-reflective area color image, and compare each pixel in the original pharyngeal image with the corresponding pixel in the smooth non-reflective area color image to obtain the reflective area;

[0036] The reflective area is filled with the pixel average value of a ring area formed by circles with a radius of 2 and 4 pixels around the reflective area. Then, Gaussian blur processing is performed on the filled image to obtain a non-reflective strong smooth image, completing the de-reflection processing of the pharyngeal image.

[0037] The pharyngeal image segmentation and localization method based on the deeplabv3+ network provided by the present invention has the following beneficial effects compared with the existing technology:

[0038] 1) The data set collection takes into account factors such as pharyngeal infection, which enhances the algorithm's anti-interference ability.

[0039] 2) In the design of the loss function, the characteristics of Focal Loss and DSC Loss are integrated. By adjusting the weight factor, the imbalance problem can be solved and attention can be focused on samples with inaccurate predictions.

[0040] 3) The lightweight MobilenetV2 network is used to replace the computationally complex Xception network, which reduces hardware requirements, greatly reduces the amount of parameter calculation, and improves detection speed and real-time performance.

[0041] 4) Improved MobilenetV2 by adding branched dilated convolution to better extract multi-scale information and a parallel attention mechanism to improve the segmentation accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the pharyngeal image segmentation and positioning method based on the deeplabv3+ network of the present invention;

[0043] Figure 2 This is a structural diagram of the existing deeplabv3+ network;

[0044] Figure 3 This is a schematic diagram of the Bottlenecks structure in the existing MobilenetV2 network;

[0045] Figure 4 This is a schematic diagram of the improved linear bottleneck structure with stride=2 of the present invention;

[0046] Figure 5 Schematic diagram of the improved inverted residual structure with stride=1 of the present invention;

[0047] Figure 6 This is a processing flow chart of the Grab Cut algorithm adopted in the present invention;

[0048] Figure 7 Schematic diagram of depth camera calibration of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0051] To overcome the shortcomings of pharyngeal image segmentation in the prior art, this embodiment provides a pharyngeal image segmentation and localization method based on the DeepLabV3+ network. This method uses the DeepLabV3+ model, which is effective for extracting multi-scale information, and establishes a dataset with multiple noises to enhance the model's anti-interference capabilities. Based on the model used in this invention, while enhancing multi-scale information extraction, it is also necessary to increase detection speed. Therefore, this invention replaces the backbone network of the semantic segmentation model DeepLabV3+ with a lighter backbone network, and incorporates branched convolution and attention mechanisms to improve accuracy.

[0052] like Figure 1 As shown, the pharyngeal image segmentation and positioning method based on the deeplabv3+ network of this embodiment includes the following steps:

[0053] Step 1: Use a depth camera to obtain a color image and a depth image of the pharynx.

[0054] In this embodiment, a depth camera captures both a color image and a depth map. The color image is used as input to the segmentation model to generate the image segmentation results, and a boundary detection operator is used to obtain the pixel coordinates of the captured area. Furthermore, the depth camera is calibrated, and the position of the point in the world coordinate system is calculated based on the depth map and the pixel coordinates of the captured area, thereby completing positioning.

[0055] Step 2: Based on the color image, use the segmentation model built based on the deeplabv3+ network to process it and obtain the image segmentation result.

[0056] The segmentation model of this embodiment is improved based on the basic framework of deeplabv3+ network, and is specifically as follows:

[0057] 2-1. Build the basic framework of deeplabv3+ network

[0058] like Figure 2 As shown in the figure, the conventional DeepLabV3+ network consists of an encoder-decoder structure, which performs multi-scale information fusion. The encoder uses depthwise separable convolutions, and the backbone network (the backbone in the conventional DeepLabV3+ network is the Xception network) extracts features to obtain preliminary image features. Parallel dilated convolutional layers in the enhanced feature extraction (ASPP) module extract different spatial information from these preliminary image features, and 1x1 convolutions are used to fuse these features to obtain multi-scale features. The decoder first performs bilinear interpolation and 4x upsampling on the multi-scale features extracted by the enhanced feature extraction (ASPP) module. These features are then concatenated with the preliminary image features extracted by the backbone network after 1x1 convolutions (using 1x1 convolutions), and features are fused (using 3x3 convolutions). Finally, bilinear interpolation and 4x upsampling are used to restore the high-level feature map to obtain the final image segmentation result.

[0059] The enhanced feature extraction ASPP module of this embodiment is divided into two parts: encoder and decoder:

[0060] In the encoder of the enhanced feature extraction ASPP module, the preliminary effective feature layer of the backbone network compressed four times is used as the input of the dilated convolution spatial pyramid module at the end of the backbone network. Parallel dilated convolutions are used to extract features with different dilation rates, and then feature fusion is performed. Finally, 1x1 convolution is used to compress the features to obtain the final effective feature layer.

[0061] In the decoder of the enhanced feature extraction ASPP module, the initial effective feature layer compressed twice by the backbone network is adjusted to have a certain number of channels using 1x1 convolution, and then stacked with the up-sampled result of the effective feature layer after the dilated convolution. Finally, two depth-wise separable convolutions are performed to obtain the final effective feature layer.

[0062] As can be seen from the figure, the enhanced feature extraction ASPP module has five branches: a 1×1 normal convolution layer, a 3×3 dilated convolution with dilation coefficients of 6, 12, and 18, and a global pooling layer. The output feature maps of the five branches are concatenated through concat and further fused through 1×1 convolution. All five branches use a standardized batch normalization layer and a RELU activation function for regularization and activation. The global pooling layer first averages the input features to a 1×1 size, then adjusts the number of channels through a 1×1 convolution. Finally, bilinear interpolation is used to restore the height and width of the input feature layer, restoring the image to its original size.

[0063] 2-2. Improve the deeplabv3+ network structure

[0064] This embodiment uses the MobilenetV2 network with fewer parameters and lighter weight instead of the Xception network as the backbone network of the deeplabv3+ network encoder. MobilenetV2 is used to reduce the amount of model calculation, which can improve the detection speed and reduce hardware requirements.

[0065] The encoder backbone network in the model used in this embodiment is MobilenetV2. The network structure consists of standard convolutional layers, depthwise separable convolutional layers, batch normalization layers, average pooling layers, and fully connected layers. Because MobilenetV2 is only needed as the backbone network to extract deep and shallow features, this embodiment only uses the inverted residual structure in the existing MobilenetV2 network. The MobilenetV2 network structure of this embodiment is shown in Table 1.

[0066] Table 1 MobilenetV2 structure of this embodiment

[0067] Input Operator t c n s <![CDATA[512 2 ×3]]> Conv2d - 32 1 2 <![CDATA[256 2 ×32]]> Bottleneck 1 16 1 1 <![CDATA[256 2 ×16]]> Bottleneck 6 24 2 2 <![CDATA[128 2 ×24]]> Bottleneck 6 32 3 2 <![CDATA[64 2 ×32]]> Bottleneck 6 64 4 2 <![CDATA[32 2 ×64]]> Bottleneck 6 96 3 1 <![CDATA[32 2 ×96]]> Bottleneck 6 160 3 2 <![CDATA[16 2 ×160]]> Bottleneck 6 320 1 1

[0068] Among them, t represents the expansion factor, c represents the output feature matrix depth channel, n represents the number of repetitions of Bottleneck, and s represents the step size of the first layer in each block.

[0069] like Figure 3 As shown, the Bottleneck operation in MobilenetV2 includes a first Bottleneck operation with a stride of 1 (stride=1) and a second Bottleneck operation with a stride of 2 (stride=2).

[0070] The first and second Bottleneck operations both contain two normal convolutional layers (Conv) and one depthwise separable convolutional layer (Dwise). First, a 1×1 convolution is used to increase the dimensionality of the input data, and then a depthwise separable convolutional layer is used to extract features. These two convolution operations are followed by a Relu6 activation function, and finally a 1×1 dimensionality reduction is performed, followed by a Linear activation function to preserve linear features. The Relu6 activation function is defined as:

[0071] y=RELU6(x)=min(max(x,0),6)

[0072] That is, when the input value x is less than 0, the output result y is set to 0; when the input value x is greater than 0 and less than 6, no processing is performed; when the input value x is greater than 6, the output result y is set to 6.

[0073] In the Bottleneck operation of the MobilenetV2 network, when stride=1 and the input feature matrix and the output feature matrix have the same shape, the first Bottleneck operation, i.e., the inverted residual module, is taken. In each inverted residual module, multi-scale information can be fused through shortcut connections. The input feature of size h×w×k is output as h×w×(tk) after 1×1 convolution, where h and w are the length and width of the feature map, t is the expansion factor, and k is the number of channels; the output result is further used as the input of the next layer, and after 3×3 depth-separable convolution, the output is Where s represents the step size; finally, the last layer of 1×1 convolution is used to reduce the dimension and output is

[0074] In order to make the MobilenetV2 network more suitable for the segmentation task in the present invention, this embodiment modifies the original MobilenetV2 structure:

[0075] like Figure 4 As shown in the figure, based on the Bottleneck structure of the original MobilenetV2 network, the 1×1 convolution in the downsampling process (i.e. Figure 3 In the stride = 2, the convolution in Conv1*1, Linear is replaced with average pooling AvgPool (2x2, s=2) to reduce information loss during downsampling. Three 3×3 convolutional branches with expansion factors of 2, 4, and 6 are added to the output of the replaced linear bottleneck structure to extract multi-scale features. The resulting feature maps are fused using 1*1 convolution to obtain the final multi-scale feature map.

[0076] like Figure 5 As shown, based on the Bottleneck structure of the original MobilenetV2 network, an attention mechanism is added to the inverted residual structure with stride = 1, and attention feature maps are generated in parallel from the two aspects of spatial attention and channel attention. That is, this embodiment adds two branches of the channel attention mechanism and the spatial attention mechanism in parallel at the output end of the inverted residual structure with stride = 1, generates two attention feature maps for the feature map output by the inverted residual structure with stride = 1, fuses the attention feature maps of the two branches, and then obtains the final output attention feature map through the sigmoid activation function.

[0077] Among them, the channel attention mechanism first performs global average pooling on the input feature map to obtain a new feature map, then inputs the obtained feature map into the middle shared network hidden layer for processing, and finally performs batch normalization through the BN layer to obtain the channel attention feature map.

[0078] The calculation formula of the channel attention mechanism is as follows:

[0079] M c (F)=BN(MLP(AvgPool(F)))=BN(W1(W0AvgPool(F)+b0)+b1)

[0080] Where M c (F) is the channel attention feature map, F is the input feature map, W0∈R C / r×C , b0∈R C / r , W1∈R C ×C / r , b1∈R C , BN() is the BN layer, MLP() is the shared network hidden layer, and AvgPool() is the global average pooling layer.

[0081] Among them, the spatial attention mechanism first reduces the channel of the input feature map through a 1x1 convolution, and then passes it through two layers of 3x3 dilated convolution to aggregate contextual information with a larger receptive field, and then performs another 1x1 convolution to reduce the number of channels, and finally performs batch normalization through the BN layer to obtain the spatial attention feature map.

[0082] The calculation formula of the spatial attention mechanism is as follows:

[0083]

[0084] Where M s (F) is the spatial attention feature map, and is a 1x1 convolutional layer, and It is a 3x3 hole convolution layer.

[0085] Finally, the attention feature maps of the two branches are fused and the final attention feature map M(F) = σ(M c (F)+M s (F)).

[0086] After completing the feature extraction of MobilenetV2, two effective feature layers can be obtained. One effective feature layer is the result of compressing the height and width of the input image twice, that is, the preliminary effective feature layer of the backbone network compressed twice; the other is the result of compressing the height and width of the input image four times, that is, the preliminary effective feature layer of the backbone network compressed four times.

[0087] This embodiment is based on the constructed improved deeplabv3+ network for training, and the segmentation model is output for use after training. The specific training process is as follows:

[0088] 1) Acquire pharyngeal images, including normal pharyngeal images and pharyngeal infection images. In this embodiment, the pharyngeal images acquired for training include interference factors such as tonsil infection and tongue depressors to mitigate individual differences in sampled subjects and other environmental factors in actual testing.

[0089] 2) Preprocess the collected pharyngeal images to generate a training dataset.

[0090] 2-1) Perform de-reflection processing on the pharyngeal image.

[0091] During image capture, the palate, uvula, tonsils, and other parts of the mouth appear highlighted in the image due to specular reflections from the light source. These image features can affect the perceived image quality, so de-reflection processing is performed on the image before dataset enhancement. The specific steps for de-reflection processing in this embodiment are as follows:

[0092] 2-1-1) Perform reflection detection on the image to obtain the highlight area in the pharyngeal image.

[0093] Using the grayscale intensity as a reference, the green and blue channels cG and cB of the image are normalized, and then the grayscale intensity cE is calculated. The specific calculation formula is:

[0094] cE=0.2989·cR+0.5870·cG+0.1140·cB

[0095] Among them, cR is the red channel.

[0096] Then calculate the color balance ratio, the calculation formula is as follows:

[0097]

[0098] Where r GE is the color balance ratio of the green channel and grayscale intensity, r BE is the color balance ratio of the blue channel and grayscale intensity, P 95 () indicates the 95th percentile, i.e. the value above which 95% of the color intensity values exceed.

[0099] The conditions satisfied by the highlight pixel x0 in the image are as follows:

[0100] cG(x0)>r GE ·T∪cB(x0)>r BE T∪cB(x0)>T

[0101] Wherein, T=240 represents the grayscale threshold. The highlight area in the image is confirmed based on the determined highlight pixels.

[0102] 2-1-2) The detected highlight area is filled with the average pixel value of a circular area formed by a circle with a radius of 2 or 4 pixels around the highlight area to obtain a filled image. When forming the circular area with a radius of 2 or 4 pixels around the highlight area, the center of the circle (or a position close to the center) can be used as the center of the circle, and the radius can be expanded by 2 or 4 pixels based on the maximum distance from the highlight area to the center.

[0103] 2-1-3) Detect reflective areas: Perform median filtering on the filled image to obtain smooth non-reflective area color pixels. The image after median filtering is a smooth non-reflective area color image. Compare each pixel in the original image with the corresponding pixel in the smooth non-reflective area color image. Compare the pixel values of the two. If the pixel value in the original image is greater than the pixel value in the smooth non-reflective area color image, the corresponding pixel point is a reflective pixel, thereby obtaining the reflective area.

[0104] 2-1-4) Reflection repair: Use the filling method in the reflective area detection method to fill the reflective area, and then perform Gaussian blur (Gaussian kernel of 8) on the filled image to obtain a non-reflective strong smooth image, completing the de-reflection processing of the pharyngeal image.

[0105] 2-2) The de-reflected pharyngeal image is cropped to retain the segmented area with valuable information.

[0106] 2-3) Label the segmented areas and perform data augmentation after labeling to obtain a training data set.

[0107] In this embodiment, the segmented region is categorized (i.e., the pixel region is labeled as the pharynx). The segmented region is an M-shaped region of the pharynx that does not include the uvula. The dataset is then augmented using methods such as center cropping, random rotation, Gaussian blurring, and RGB translation. Furthermore, in data augmentation, the segmented region is pre-extracted using the Grab Cut algorithm. The coarse segmentation result obtained from the pre-extraction is then superimposed on the de-reflected image to enhance the signal strength at the segmented location and improve the model training effect.

[0108] like Figure 6 As shown in the figure, the Grab Cut algorithm is specifically as follows: the marked boundary is used as the initial information, the pixels outside the boundary are regarded as known background pixels, and the pixels within the marked area are marked as unknown pixels; the foreground and background are clustered using the K-means algorithm, and a new label for each pixel is obtained based on the color distribution after clustering; the new label and the corresponding pixel are used as input to enter the next iteration; after multiple iterations until the model converges, the final segmentation result is obtained.

[0109] In order to verify the effectiveness of the data preprocessing method (de-reflection + Grab Cut) proposed in this embodiment, this embodiment uses a public dataset for verification. The comparison experimental group is set to images without preprocessing and images with only de-reflection processing. The segmentation model of this embodiment is used for segmentation processing. The experimental verification results are shown in Table 2 below:

[0110] Table 2 Verification results of data preprocessing methods

[0111] Precision Recall DSC Unprocessed 0.869 0.851 0.843 Anti-glare 0.914 0.893 0.879 De-glare + Grab Cut 0.936 0.924 0.919

[0112] Where Precision is accuracy, Recall is recall rate, and DSC is Dice Coefficient. According to the results in Table 2, it can be seen that the image processed by de-reflection combined with Grab Cut algorithm in this embodiment can enable the segmentation model to obtain the optimal image segmentation result.

[0113] 3) Design loss function.

[0114] The loss function used in the network training of this embodiment is a hybrid loss function:

[0115] Loss=λLoss FL +(1-λ)Loss DSC

[0116] Among them, Loss FL Represents the Focal Loss loss function, Loss DSCDenotes the DSC loss function, and λ is a weight factor used to adjust the weights of the two loss functions. The Focal Loss loss function and the DSC loss function are conventional loss functions and will not be described in detail in this embodiment.

[0117] The present invention conducted an experimental comparison on the selection of the value of the weight factor λ, setting λ to 0, 0.2, 0.4, 0.6, 0.8, and 1 respectively. The improved deeplabv3+ network of this embodiment was trained based on different weight factors. The trained model was taken to obtain the weight with the best effect through comparative experiments. The results are shown in Table 3 below:

[0118] Table 3 Comparison experimental results of weight factors

[0119] λ Precision Recall DSC 0 0.838 0.841 0.835 0.2 0.859 0.867 0.851 0.4 0.884 0.892 0.879 0.6 0.913 0.926 0.911

[0120]

[0121] As can be seen from the above table, when the value of λ is 0.6, the network performance is optimal. Therefore, in the network training of this embodiment, the weight factor λ is set to 0.6.

[0122] 4) The Adam optimizer was used to train the network using the stochastic gradient descent (SGD) method. During the training process, the model training was stopped when the loss converged and did not change significantly within 120 consecutive iterations.

[0123] Step 3: Use the boundary detection operator to obtain the pixel coordinates of the pharynx in the image segmentation result.

[0124] In actual sampling, the collection part needs to be at the boundary of the segmentation result. This embodiment uses the conv2d operator, that is, the width direction convolution kernel is [-1,0,1]. When the pixel point is in the background or foreground, the convolution calculation result is 0; when the pixel point is at the boundary, the convolution result is not 0, so as to obtain the pixel coordinate value of the boundary.

[0125] Step 4: Based on the depth map, obtain the depth value of the pharynx according to the pixel coordinates of the pharynx.

[0126] Step 5: According to the calibration parameters of the depth camera, the position of the pharynx in the world coordinate system is obtained based on the pixel coordinates and depth value of the pharynx to complete the positioning.

[0127] The present invention uses Zhang Zhengyou calibration method to realize the self-calibration of the depth camera and calculate the camera internal parameters. When calibrating the depth camera in the pharyngeal image acquisition, the hand-eye calibration coordinate system is converted as follows Figure 7 In this step, two coordinate transformation relationships need to be obtained. The first is the coordinate system transformation from the end of the robotic arm to the camera, and the second is the position relationship of the cotton swab head relative to the end of the robotic arm.

[0128] Among them, T1 is the transformation matrix from the world coordinate system of the manipulator to the coordinate system of the end of the manipulator; T2 is the fixed matrix of substitution, which represents the transformation matrix from the coordinate system of the end of the manipulator to the camera coordinate system; T3 is the transformation matrix from the camera coordinate system to the coordinate system of the calibration plate; T4 is the transformation matrix from the coordinate system of the end of the manipulator to the coordinate system of the calibration plate; T5 is the transformation matrix from the world coordinate system of the manipulator to the coordinate system of the calibration plate.

[0129] Therefore, the conversion formula for capturing the calibration image is: T5 = T1T2T3. By changing the position of the calibration plate and capturing calibration images multiple times, we obtain the equation system and solve the matrix T2. By repeatedly touching the cotton swab tip to the corner points of the calibration plate's checkerboard grid and capturing images to obtain the corner coordinates, we can determine the position of the cotton swab tip relative to the end of the robotic arm using the following formula: T4 = T2T3.

[0130] Based on the above transformation matrix, the position information in the world coordinate system is obtained through the following steps:

[0131] First, the pixel coordinates of the captured part in the camera coordinate system are converted from two-dimensional to three-dimensional. The pixel coordinate projection formula in the image is as follows:

[0132]

[0133] Where [u, v] is the pixel coordinate of the pixel in the image, Z d Indicates the depth value, [c x ,c y ] represents the image center coordinates, f x is the focal length of the camera, [X camera ,Y camera ,Z camera ] is the three-dimensional pixel coordinate of the acquisition site.

[0134] The image pixel coordinate conversion formula in the world coordinate system is as follows:

[0135]

[0136] Among them, [X base ,Y base ,Z base ] is the position information of the collected part in the world coordinate system.

[0137] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A pharyngeal image segmentation and positioning method based on deeplabv3+ network, characterized in that: The pharyngeal image segmentation and positioning method based on the deeplabv3+ network includes: Step 1: Use a depth camera to obtain a color image and a depth image of the pharynx; Step 2: Based on the color image, use the segmentation model built based on the deeplabv3+ network to process it and obtain the image segmentation results, including: The deeplabv3+ network is trained to obtain a segmentation model, wherein the deeplabv3+ network is mainly an encoder-decoder structure; In the encoder part, depthwise separable convolution is used. The backbone network extracts features to obtain preliminary image features. Then, the parallel dilated convolution layer in the enhanced feature extraction ASPP module extracts different spatial information of the preliminary image features and performs feature fusion to obtain multi-scale features. In the decoder part, the multi-scale features extracted by the enhanced feature extraction ASPP module are first upsampled by 4 times using bilinear interpolation, then spliced and fused with the preliminary image features extracted from the backbone network. Finally, bilinear interpolation and 4 times upsampling are used to restore the high-level feature map to obtain the final image segmentation result. The backbone network is an improved MobilenetV2 network, which adopts the Bottleneck structure of the original MobilenetV2 network, adds an attention mechanism to the inverted residual structure with stride=1, and adds parallel branches with different expansion rates to the linear bottleneck structure with stride=2 to obtain multi-scale features. Step 3: Use the boundary detection operator to obtain the pixel coordinates of the pharynx in the image segmentation result; Step 4: Based on the depth map, obtain the depth value of the pharynx according to the pixel coordinates of the pharynx; Step 5: According to the calibration parameters of the depth camera, the position information of the pharynx in the world coordinate system is obtained based on the pixel coordinates and depth value of the pharynx to complete the positioning.

2. The pharyngeal image segmentation and positioning method based on the deeplabv3+ network according to claim 1, characterized in that: The attention mechanism is added to the inverted residual structure with stride=1, including: At the output end of the inverted residual structure with stride=1, two branches of the channel attention mechanism and the spatial attention mechanism are added in parallel. Two attention feature maps are generated for the feature map output by the inverted residual structure with stride=1 respectively. The attention feature maps of the two branches are fused, and then the final output attention feature map is obtained through the sigmoid activation function.

3. The pharyngeal image segmentation and positioning method based on the deeplabv3+ network according to claim 1, characterized in that: The parallel branches with different expansion rates are added to the linear bottleneck structure with stride=2 to obtain multi-scale features, including: In the linear bottleneck structure with stride = 2, the third convolution layer is first replaced with average pooling. Then, three 3×3 convolution parallel branches with expansion coefficients of 2, 4, and 6 are added to the output end of the linear bottleneck structure to perform multi-scale feature extraction. The obtained multi-scale feature maps are fused to obtain the final multi-scale feature map.

4. The pharyngeal image segmentation and positioning method based on the deeplabv3+ network according to claim 1, characterized in that: The segmentation model obtained by training the deeplabv3+ network includes: Acquiring pharyngeal images to generate a training data set, wherein the pharyngeal images include normal pharyngeal images and pharyngeal infection images; Use Adam optimizer to train the deeplabv3+ network. During the training process, when the loss value converges within a number of consecutive iterations, stop the model training and output it as the segmentation model. Among them, the loss function used in the loss value calculation is as follows: ； in, represents the Focal Loss loss function, represents the DSC loss function, is the weight factor.

5. The pharyngeal image segmentation and positioning method based on the deeplabv3+ network according to claim 4, characterized in that: The collecting of pharyngeal images to generate a training data set includes: Perform de-reflection processing on the pharyngeal image; The de-reflected pharyngeal image is cropped to retain the segmented area; The segmented areas are labeled, and data enhancement processing is performed after labeling to obtain a training data set.

6. The pharyngeal image segmentation and positioning method based on the deeplabv3+ network according to claim 5, characterized in that: The de-reflection processing of the pharyngeal image includes: Performing reflection detection on the pharyngeal image to obtain the highlight area in the pharyngeal image; The detected highlight area is filled with the pixel average value of the circular area formed by the circles with radii of 2 and 4 pixels around it to obtain the filled image; Perform median filtering on the filled image to obtain a smooth non-reflective area color image, and compare each pixel in the original pharyngeal image with the corresponding pixel in the smooth non-reflective area color image to obtain the reflective area; The reflective area is filled with the pixel average value of a ring area formed by circles with a radius of 2 and 4 pixels around the reflective area. Then, Gaussian blur processing is performed on the filled image to obtain a non-reflective strong smooth image, completing the de-reflection processing of the pharyngeal image.

Images