Laryngoscope vocal cord focus detection method based on adaptive multi-scale feature fusion and attention mechanism

Abstract

The invention provides a laryngoscope vocal cord focus detection method based on adaptive multi-scale feature fusion and an attention mechanism. An improved laryngoscope vocal cord focus detection network model based on YOLO11 is researched and designed. The method comprises the steps that an AMFF (adaptive Multi-scale Feature Fusion Module) feature extraction module is researched and designed for laryngoscope lesion morphology diversity and boundary fuzziness, the capturing capacity of the model for multi-scale information and irregular lesion features is enhanced, meanwhile, an adaptive Dynamic attention mechanism is added in front of each detection head, and the detection capacity of the model for complex and diversified lesions is enhanced. The laryngoscope vocal cord focus detection method based on self-adaptive multi-scale feature fusion and the attention mechanism is high in accuracy, the detection effect and robustness of the model are effectively improved, and powerful support is provided for the medical field of laryngoscope examination.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision (lesion detection), specifically to a method for detecting vocal cord lesions using laryngoscopy based on adaptive multi-scale feature fusion and attention mechanism. Background Technology

[0002] With the accelerating pace of modern life and the impact of environmental pollution and excessive voice use, the incidence of vocal cord lesions is increasing year by year. Common lesion types include vocal cord nodules, polyps, leukoplakia, and papillomas. If these lesions are not diagnosed and treated in a timely and accurate manner, they may lead to hoarseness, aphonia, or even cancer. Clinically, laryngoscopy is the main means of detecting and diagnosing vocal cord lesions. With the rapid development of deep learning and computer vision technologies, object detection methods based on convolutional neural networks (CNNs) have been widely used in the field of medical image analysis. In real-time automatic detection of laryngoscopy images, the YOLO method has good performance in general object detection tasks, but when directly applied to the detection of vocal cord lesions in laryngoscopy, there are still problems such as large differences in the scale of vocal cord lesions. Conventional detection networks have low detection accuracy when dealing with small-scale or blurred-boundary lesions. Existing models, due to the limitations of convolution, often limit feature extraction to local areas and find it difficult to establish long-distance dependencies. At the same time, for complex inputs, using convolution kernels with fixed parameters does not provide high detection accuracy for complex and variable laryngoscopy lesion images.

[0003] To address the problems of existing models, this patent proposes a laryngoscope vocal cord lesion detection method based on adaptive multi-scale feature fusion and attention mechanism. For some lesions that are irregular and scattered, the AMFF module in the backbone network enhances the model's ability to extract irregular lesions through deformable convolution and progressively increasing receptive fields. An adaptive dynamic attention mechanism is added before the detection head to enhance the model's adaptability to complex input image features before the output results, thereby enhancing the network's ability to detect complex and diverse lesions, effectively improving the network's accuracy and recall, and enhancing the network's robustness. Summary of the Invention

[0004] The purpose of this invention is to design a network for detecting lesion areas in laryngoscope images of the vocal cords, which can detect complex lesions in laryngoscope images and has high detection accuracy and speed.

[0005] This invention provides a laryngoscope vocal cord lesion detection network based on adaptive multi-scale feature fusion and attention mechanism. It mainly includes a backbone network based on AMFF, used to extract features from the input image for subsequent network training; a multi-scale feature fusion structure, used to fuse the output features of the backbone network's attention mechanism to obtain multi-scale features rich in semantic and positional information, which are then processed by an adaptive dynamic attention module based on the features output to the detection head; and a detection head, used to output the final detection result.

[0006] This invention mainly consists of the following steps:

[0007] 1. The detection dataset is constructed and labeled by professional doctors, and the dataset is divided into training set, validation set, and test set;

[0008] 2. Construct a vocal cord laryngoscopy lesion detection network based on an adaptive multi-scale fusion module and an attention mechanism;

[0009] 4. Train the completed target detection model on the created vocal cord laryngoscope image dataset and save the trained model weights;

[0010] 5. Convert the saved model weights to engine format files to accelerate inference using TensorRT. Deploy the detection model and build the relevant detection interface using C++. Input the vocal cord laryngoscope image into the detection model, and the detection interface displays the lesion area of ​​the laryngoscope image in real time.

[0011] In the above steps, the constructed vocal cord laryngoscope lesion target detection network mainly includes an AMFF-based backbone network, adaptive dynamic attention, a multi-scale feature fusion structure, and a detection head.

[0012] The backbone network is built based on the AMFF module, which has strong feature extraction capabilities. Output feature maps from different levels of the backbone network are selected for subsequent networks to continue extracting lesion features. The construction of the AMFF module includes the following steps:

[0013] Step 1: Divide the input features into four feature maps F1, F2, F3, and F4 on an average basis according to the channel dimension. At the same time, the input features are subjected to global average pooling and then MLP+Softmax to obtain scalar feature weights α1, α2, α3, and α4.

[0014] Step 2: For the four feature maps obtained in Step 1, F1 and F2 are processed by a 1×1 convolution and a 3×3 convolution, F3 is processed by a 1×1 convolution and a 5×5 Deformable convolution with a dilation of 2, and F4 is processed by a 1×1 convolution and a 7×7 Deformable convolution with a dilation of 3. The output of each feature map is used as the input of the next part, that is, the output of F1 is used as the input of F2, the output of F2 is used as the input of F3, and so on.

[0015] Step 3: Based on the four feature maps obtained in Step 2, multiply them by the scalar weights α1, α2, α3, and α4 obtained in Step 1 to obtain weighted feature maps. Then, concat the weighted feature maps, establish the connection between channels through a 1×1 convolution, and then pass them through a 3×3 Dynamic Conv to obtain the output feature map.

[0016] Step 4: Pass the feature map obtained in Step 3 through a 3*3 CA_Dynamic Conv, and then perform a residual connection with the original input feature map to obtain the final output feature map;

[0017] Step 5: For CA_Dynamic Conv in Step 4, its dynamic convolution kernel is obtained through the following steps: First, the input feature map is compressed into a scalar by global average pooling for each channel. Then, it is passed through a dimension reduction fully connected layer (the number of channels is compressed to 16), a ReLU activation function, and a dimension increase fully connected layer (restoring the original number of channels). Finally, the attention weights of each channel are generated by the Sigmoid function. Then, global average pooling and global max pooling are used in the HW dimension and the feature maps are added together. Then, it is passed through a 1×1 convolution (the number of channels is compressed to 16), a ReLU activation function, and a 1×1 convolution (restoring the number of channels). Finally, the channel dimension is reshaped to the HW dimension to generate the dynamic convolution kernel.

[0018] The adaptive dynamic attention mechanism is characterized by adjusting the sampling position of the fused features through a 3×3 deformable convolution, using the SE attention module to adaptively weight the channels, and dynamically generating attention weights using CA_Dynamic Conv. The construction of the adaptive attention module includes the following steps:

[0019] Step 1: The input features are first processed through a 3×3 deformable convolution;

[0020] Step 2: For the feature map obtained in Step 1, firstly, the spatial dimension of each channel is compressed into a scalar by global average pooling. Then, it is passed through a dimension reduction fully connected layer (the number of channels is compressed to 16), a ReLU activation function, and a dimension increase fully connected layer (the original number of channels is restored). Finally, the attention weights of each channel are generated by the Sigmoid function. The attention weights are then multiplied with the output of Step 1 to obtain the channel attention-weighted feature map.

[0021] Step 3: Based on the feature map obtained in Step 2, perform global average pooling and global max pooling on the channel dimension respectively. Concatenate the two feature maps and input them into a CA-Dynamic Conv module with a convolution kernel size of 7×7. Then, add the outputs of the two branches and generate spatial attention weights through the Sigmoid activation function. Multiply the weights element-wise with the original feature map to obtain the weighted feature representation. Then, perform residual connection (i.e., element-wise addition) with the original feature map to finally output the enhanced feature map.

[0022] Step 4: For CA-Dynamic Conv in Step 3, its dynamic convolution kernel is obtained through the following steps: First, the input feature map is compressed into a scalar by global average pooling for each channel. Then, it is passed through a dimension reduction fully connected layer (the number of channels is compressed to 16), a ReLU activation function, and a dimension increase fully connected layer (restoring the original number of channels). Finally, the attention weights of each channel are generated by the Sigmoid function. Then, global average pooling and global max pooling are used in the HW dimension and the feature maps are added together. Then, it is passed through a 1×1 convolution (the number of channels is compressed to 16), a ReLU activation function, and a 1×1 convolution (restoring the number of channels). Finally, the channel dimension is reshaped to the HW dimension to generate the dynamic convolution kernel.

[0023] The output feature map of the adaptive dynamic attention module is input into the detection head to obtain the final detection output;

[0024] The object detection module consists of a decoupled classification detection head and a bounding box regression detection head. The output feature map from the multi-scale feature fusion structure is first passed through a 1×1 convolution, and then further passed through two parallel 3×3 convolutions, a BatchNorm layer, and a Silu activation function to obtain feature maps for classification and regression, respectively. The classification branch is a 1×1 convolution with output channels equal to the number of classes, the regression branch is a 1×1 convolution with 4 output channels (i.e., the center point coordinates, width, and height of the bounding box), and the object confidence branch is a 1×1 convolution with 1 output channel (i.e., the confidence score). The loss function for the classification branch is the cross-entropy loss function.

[0025]

[0026] Where N Pos Let C represent the total number of positive samples, and y represent the number of categories. i,c This represents the true label of class c for the i-th sample. Let represent the predicted label of class c for the i-th sample.

[0027] The loss function for the regression branch is:

[0028]

[0029] Where N Pos B represents the total number of samples. gt =(x gt ,y gt ,w gt ,h gt ) represents the center coordinates and width and height of the true bounding box, B pred =(x pred y pred w pred h pred ) represents the center point coordinates and width and height of the prediction box.

[0030] The confidence loss function is:

[0031]

[0032] Where N represents the total number of samples, S represents the size of each image on the feature map, and C represents the number of classes. This indicates whether the target object corresponding to the j-th feature point in the i-th sample exists. If the target object exists... The value is 1 if it is 1, otherwise it is 0. This represents the probability that the target object corresponding to the j-th feature point in the i-th sample, as predicted by the model, exists.

[0033] The overall loss function can be expressed as:

[0034] L loss =L cls +ηL reg +L obj

[0035] Where η is a hyperparameter for adjusting the balance of various losses, and in this invention, η is set to 5.0.

[0036] By adopting the above technical solution, the present invention has the following advantages:

[0037] 1. This invention designs a laryngoscope vocal cord lesion detection network based on multi-scale deformable convolution and attention mechanism. In order to adapt to the irregularity and significant scale differences of vocal cord lesions in laryngoscope images, the AMFF module is used in the network, which enables the network to learn multi-scale features and better extract features of irregular lesions, thus overcoming the shortcomings of low recall rate for detecting lesions at multiple scales.

[0038] 2. To enhance the network's adaptability to complex and diverse image inputs, dynamic convolution is used. The convolution kernel is no longer a fixed parameter, but is adaptively adjusted according to the input features, thereby improving the model's adaptability. Attached Figure Description

[0039] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following drawings are provided for illustration:

[0040] Figure 1 This is a schematic diagram of the process of the vocal cord laryngoscope image detection method based on adaptive multi-scale fusion and attention mechanism of the present invention.

[0041] Figure 2 This is a schematic diagram of the AMFF module of the present invention.

[0042] Figure 3 This is a schematic diagram of the adaptive dynamic attention mechanism of the present invention.

[0043] Figure 4 This is a schematic diagram of the vocal cord laryngoscope image detection network based on the adaptive multi-scale fusion module and attention mechanism of the present invention. Detailed Implementation Plan

[0044] The present invention will be further described in detail below with reference to specific implementations. The description is for explanation and not limitation of the present invention.

[0045] This invention proposes a method for detecting vocal cord lesions via laryngoscopy based on multi-scale deformable convolution and attention mechanisms, such as... Figure 1 As shown, the specific steps include:

[0046] Step 1: A professional doctor constructs and labels the detection dataset, and divides the dataset into a training set, a validation set, and a test set;

[0047] Step 2: Construct a vocal cord laryngoscopy lesion detection network based on an adaptive multi-scale feature fusion module and an attention mechanism;

[0048] Step 3: Train the completed target detection model on the created vocal cord laryngoscope image dataset and save the trained model weights;

[0049] Step 4: Use the saved model weights to detect lesion areas in the laryngoscope image. Input the vocal cord laryngoscope image into the model and use the model to detect whether there are lesion areas.

[0050] Specific implementation methods:

[0051] The specific implementation method provides the steps for constructing a vocal cord laryngoscope lesion detection network based on multi-scale deformable convolution and attention mechanism, which includes: an AMFF-based backbone network, an adaptive dynamic attention mechanism module, a multi-scale feature fusion structure, and a target detection module.

[0052] Step 1: A professional doctor constructs and labels the detection dataset, and divides the dataset into a training set, a validation set, and a test set;

[0053] Step 2: Train the constructed vocal cord laryngoscopy lesion detection network using the created dataset. Use the AFFM module as the foundation of the backbone network to enhance the network's ability to extract irregular features. The steps for constructing the AFFM module are as follows:

[0054] Step 2-1: Divide the input features into four feature maps F1, F2, F3, and F4 on an average basis according to the channel dimension. At the same time, the input features are subjected to global average pooling and then MLP+Softmax to obtain scalar feature weights α1, α2, α3, and α4.

[0055] Step 2-2: For the four feature maps obtained in Step 2-1, F1 and F2 are processed by a 1×1 convolution and a 3×3 convolution, F3 is processed by a 1×1 convolution and a 5×5 Deformable convolution with a dilation of 2, and F4 is processed by a 1×1 convolution and a 7×7 Deformable convolution with a dilation of 3. The output of each feature map is used as the input of the next part, that is, the output of F1 is used as the input of F2, the output of F2 is used as the input of F3, and so on.

[0056] Step 2-3: Based on the four feature maps obtained in Step 2-2, multiply them by the scalar weights α1, α2, α3, and α4 obtained in Step 1 to obtain weighted feature maps. Then, concat the weighted feature maps, establish the connection between channels through a 1×1 convolution, and then pass them through a 3×3 Dynamic Conv to obtain the output feature map.

[0057] Step 2-4: Pass the feature map obtained in Step 2-3 through a 3×3 CA_Dynamic Conv, and then perform a residual connection with the original input feature map to obtain the final output feature map;

[0058] Steps 2-5: For CA_Dynamic Conv in Step 2-4, its dynamic convolution kernel is obtained through the following steps: First, the input feature map is compressed into a scalar by global average pooling for each channel. Then, it is passed through a dimension reduction fully connected layer (the number of channels is compressed to 16), a ReLU activation function, and a dimension increase fully connected layer (restoring the original number of channels). Finally, the attention weights of each channel are generated by the Sigmoid function. Then, global average pooling and global max pooling are used in the HW dimension and the feature maps are added together. Then, it is passed through a 1×1 convolution (the number of channels is compressed to 16), a ReLU activation function, and a 1×1 convolution (restoring the number of channels). Finally, the channel dimension is reshaped to the HW dimension to generate the dynamic convolution kernel.

[0059] Step 3: Perform feature fusion on the features extracted from the backbone network;

[0060] Step 4: The features fused and output to the detection head first pass through an adaptive dynamic attention module to enhance the model's adaptability to complex and diverse input images. The steps for constructing the adaptive dynamic attention module are as follows:

[0061] Step 4-1: The input features are first processed through a 5×5 deformable convolution with an inflation rate of 2;

[0062] Step 4-2: For the feature map obtained in Step 4-1, it goes through two branches. One branch first uses a 1×1 convolution, then a 3×3 dynamic convolution. Then the feature map is subjected to a global average pooling and a global max pooling and added together. The attention weights are obtained by passing the sigmoid activation function. Each of the above convolutions is followed by a BatchNorm layer and a Silu activation function.

[0063] Step 4-3: Based on the two branches in Step 4-2, the second branch uses the SE module. The SE module first performs a global average pooling on the feature map obtained in Step 1, and then passes the obtained feature map through two fully connected layers. The first fully connected layer reduces the channel dimension to 1 / 16 of the original, and the second fully connected layer restores the channel dimension. Finally, it passes through a sigmoid activation function to obtain the channel attention weights, which are then multiplied with the feature map obtained in Step 1.

[0064] Step 4-4: Multiply the attention weights obtained in Step 4-2 with the feature map after channel attention enhancement in Step 4-3 to obtain the final feature map;

[0065] Step 5: The object detection module consists of a decoupled classification detection head and a bounding box regression detection head. The output features of the multi-scale feature fusion structure are first passed through a 1×1 convolution, and then further passed through two parallel 3×3 convolutions, a BatchNorm layer, and a Silu activation function to obtain features for classification and regression, respectively. The classification branch is a 1×1 convolution with output channels equal to the number of classes, the regression branch is a 1×1 convolution with 4 output channels (i.e., the center point coordinates, width, and height of the bounding box), and the object confidence branch is a 1×1 convolution with 1 output channel (i.e., the confidence score). The loss function for the classification branch is the cross-entropy loss function.

[0066]

[0067] Where N Pos Let C represent the total number of positive samples, and y represent the number of categories. i，c This represents the true label of class c for the i-th sample. Let represent the predicted label of class c for the i-th sample.

[0068] The loss function for the regression branch is:

[0069]

[0070] Where N Pos B represents the total number of samples. gt =(x gt y gt w gt h gt ) represents the center coordinates and width and height of the true bounding box, B pred =(x pred y pred w pred h pred ) represents the center point coordinates and width and height of the prediction box.

[0071] The confidence loss function is:

[0072]

[0073] Where N represents the total number of samples, S represents the size of each image on the feature map, and C represents the number of classes. This indicates whether the target object corresponding to the j-th feature point in the i-th sample exists. If the target object exists... The value is 1 if it is 1, otherwise it is 0. This represents the probability that the target object corresponding to the j-th feature point in the i-th sample, as predicted by the model, exists.

[0074] The overall loss function can be expressed as:

[0075] Lloss =L cls +ηL reg +L obj

[0076] Where η is a hyperparameter for adjusting the balance of various losses, and in this invention, η is set to 5.0.

[0077] Step 6: Use the prepared training set data and corresponding object detection labels to train and construct a vocal cord laryngoscopy lesion detection network based on adaptive multi-scale fusion and attention mechanism.

[0078] Step 7: Test the trained network model using the prepared test set data and corresponding object detection labels.

Claims

1. A laryngoscope vocal cord lesion detection method based on adaptive multi-scale feature fusion and attention mechanism, characterized in that, First, the vocal cord lesion detection dataset labeled by professional doctors is obtained, and the YOLO11 network model is used as the basic network. According to the complex and diverse characteristics of the laryngoscope vocal cord lesion image, the network is optimized. The constructed vocal cord lesion detection network mainly includes: replacing PPHGNetV2 with YOLO11 backbone network, replacing HGBLOCK in PPHGNetV2 with AMFF module as the core feature extraction module of the backbone network. The constructed vocal cord laryngoscope lesion detection network includes the AMFF-based backbone network, the adaptive dynamic attention module, the multi-scale feature fusion structure and the target detection module. The data flow of the entire network is as follows: first, the output features of the AMFF module are passed through the multi-scale feature fusion structure to realize the multi-scale reuse and fusion of the lesion features. The feature map after the multi-scale feature fusion structure is used as the output of the network and is further enhanced by the adaptive dynamic attention module before generating the prediction. The target detection module is used for the detection task of the vocal cord lesion. The constructed detection model based on the multi-scale selection fusion attention mechanism is trained for several rounds on the corresponding dataset, the parameters are adjusted to obtain the optimal model, and the model is converted into an engine file for TensorRT acceleration inference. The model is deployed on a PC device using C++, and a related detection interface is constructed to display the detected lesions in real time.

2. The adaptive dynamic attention module of claim 1, wherein, The features after feature fusion are adjusted in sampling position by a 3x3 deformable convolution, the channels are adaptively weighted by an SE attention module, and the CA_DynamicConv dynamic attention weight is generated. The adaptive attention module is constructed by the following steps: Step 1: The input features are first passed through a 3x3 deformable convolution; Step 2: The feature map obtained in step 1 is first compressed to a scalar in spatial dimension by global average pooling, then sequentially passed through a dimension reduction fully connected layer (channel number compressed to 16), a ReLU activation function, a dimension increasing fully connected layer (original channel number restored), and finally the Sigmoid function is used to generate the attention weight of each channel. Multiply the attention weight by the output of step 1 to get the channel attention weighted feature map; Step 3: Based on the feature map obtained in step 2, global average pooling and global maximum pooling are performed on the channel dimension respectively. The concatenated features are input into a CA-Dynamic Conv module with a 7x7 convolution kernel. Then the outputs of the two branches are added and the spatial attention weight is generated by the Sigmoid activation function. The weight is multiplied by the original feature map element by element to get the weighted feature representation. Then the original feature map is connected in residual (i.e. element by element addition), and finally the enhanced feature map is output. Step 4: For the CA-Dynamic Conv of Step 3, the dynamic convolution kernel is obtained by the following steps: first, the input feature map is compressed to a scalar in each channel by global average pooling, then sequentially passes through a dimension reduction fully connected layer (channel number compressed to 16), a ReLU activation function, a dimension restoration fully connected layer (original channel number is restored), and finally generates attention weights for each channel through the Sigmoid function. Then, global average pooling and global maximum pooling are used in the HW dimension, and the feature maps are added. Then, a 1*1 convolution (channel number compressed to 16), a ReLU activation function, a 1*1 convolution (channel number restored), and a reshaping of the channel dimension to the HW dimension are sequentially performed to generate the dynamic convolution kernel.

3. The AMFF module of claim 1, wherein, The original feature map is evenly divided into four parts, and the output of each part is used as the input of the next part. The last two parts use deformable convolution with different kernel sizes and dilation rates. After concatenation, a 1*1 convolution is used to establish the connection between channels. Finally, a CA_DynamicConv is added to the input feature map to obtain the final output feature map. The AMFF module is constructed by the following steps: Step 1: The input feature is evenly divided into four feature maps F1, F2, F3, F4 in the channel dimension. The input feature is then globally averaged and passed through an MLP+Softmax to obtain scalar feature weights α1, α2, α3, α4. Step 2: For the four feature maps obtained in Step 1, F1 and F2 are passed through a 1*1 convolution and a 3*3 convolution, F3 is passed through a 1*1 convolution and a 5*5 deformable convolution with a dilation rate of 2, and F4 is passed through a 1*1 convolution and a 7*7 deformable convolution with a dilation rate of 3. The output of each feature map is used as the input of the next part, i.e., the output of F1 is used as the input of F2, the output of F2 is used as the input of F3, and so on. Step 3: According to the four feature maps obtained in Step 2, multiply them with the scalar weights α1, α2, α3, α4 obtained in Step 1 to obtain weighted feature maps. Then, concatenate the weighted feature maps, pass through a 1*1 convolution to establish the connection between channels, and then pass through a 3*3 Dynamic Conv to obtain the output feature map. Step 4: The feature map obtained in Step 3 is passed through a 3*3 CA_Dynamic Conv, and then a residual connection is made with the original input feature map to obtain the final output feature map. Step 5: For the CA_Dynamic Conv of step 4, the dynamic convolution kernel is obtained by the following steps: first, the input feature map is compressed to a scalar in each channel by global average pooling, then sequentially passes through a dimension reduction fully connected layer (channel number is compressed to 16), a ReLU activation function, a dimension increasing fully connected layer (restore the original channel number), and finally generates the attention weight of each channel through the Sigmoid function, then uses global average pooling and global maximum pooling in the HW dimension and adds the feature maps, and then sequentially passes through a 1x1 convolution (channel number is compressed to 16), a ReLU activation function, a 1x1 convolution (restore the channel number), and then reshapes the channel dimension to the HW dimension to generate the dynamic convolution kernel.

Images