A multi-view image classification system based on adaptive dual-domain weighted feature fusion

Abstract

This invention proposes a multi-view image classification system based on adaptive dual-domain weighted feature fusion, comprising a multi-view feature extraction module, an adaptive dual-domain weighted feature fusion module, and a feature classification module. The multi-view feature extraction module extracts image features from each viewpoint. The adaptive dual-domain weighted feature fusion module adaptively learns the weights of each viewpoint feature in both the channel and spatial domains based on a channel-space attention mechanism, and performs weighted fusion to generate a fused feature descriptor. Finally, the feature classification module outputs a preliminary classification result indicating whether the patient has a difficult or non-difficult airway. This invention solves the technical problems of insufficient information utilization and inadequate classification accuracy caused by the simple feature fusion strategy used in existing multi-view image classification systems, effectively improving the accuracy of multi-view image classification and the interpretability of system decisions.

Description

Technical Field

[0001] This invention relates to the field of medical image classification, specifically to a multi-view image classification system based on adaptive dual-domain weighted feature fusion. Background Technology

[0002] In the field of anesthesia-related safety, the identification of difficult airways is a crucial step in preventing adverse events and serious complications. Statistics show that approximately 25% of deaths in anesthetic accidents are related to failed management of difficult airways, while about 70% of difficult airway cases can be identified through preoperative assessment. Therefore, developing an efficient and accurate difficult airway identification system is of great value for improving anesthesia safety management. Traditionally, the assessment of difficult airways relies on experienced physicians manually observing and measuring the patient's physiological signs (such as maximum mouth opening, Malcolm Glasgow Coma Scale, and neck mobility). However, this method suffers from high subjectivity, poor consistency, low efficiency, and high resource consumption.

[0003] With the development of artificial intelligence technology, existing research mainly focuses on difficult airway identification systems based on machine learning and deep learning to overcome the limitations of manual assessment. These systems aim to improve the accuracy and efficiency of identification through automated analysis.

[0004] One type of existing system is built based on traditional machine learning methods. These systems typically collect or extract feature parameters related to difficult airways manually, such as gender, body mass index, neck length, and neck circumference, and use these features to train classification models, such as logistic regression, decision trees, or random forests. Although these systems have achieved some degree of automation in difficult airway identification, their feature extraction process still heavily relies on prior knowledge and experience, resulting in limited system generalization ability and inefficient feature engineering, making them unable to fully adapt to complex clinical scenarios.

[0005] Another type of existing system is based on deep learning methods, especially convolutional neural networks (CNNs), which can automatically learn features directly from input images, reducing reliance on manual features. These systems typically use single-view or multi-view images of the patient as input data. However, single-view image-based systems often suffer from insufficient recognition reliability due to limited input information and are susceptible to viewpoint bias. Multi-view image-based systems, on the other hand, provide more comprehensive feature representations by integrating information from multiple perspectives, thereby improving recognition performance. However, existing multi-view fusion methods often employ simple data integration techniques, such as feature concatenation or addition, failing to fully consider the correlation between different views and the differences in feature importance across views. For example, different views may contain redundant or complementary information, but existing systems lack adaptive mechanisms to weight and fuse this information, resulting in insufficient feature utilization and limiting further improvements in system classification accuracy.

[0006] In summary, existing airway identification systems for difficult airways suffer from the following technical problems: systems based on traditional machine learning rely on manual feature extraction, resulting in low efficiency and high subjectivity; in systems based on deep learning, single-view methods offer limited information and low reliability, while multi-view methods, due to their simple fusion strategies, fail to effectively capture the relationships between views and the importance of features, thus affecting overall performance. Therefore, it is necessary to develop a more intelligent and adaptive system to overcome these limitations. Summary of the Invention

[0007] To address the problems existing in current technologies and improve the quality and efficiency of difficult airway diagnosis, this invention proposes a multi-view image classification system based on adaptive dual-domain weighted feature fusion. By constructing a multi-view convolutional neural network module and designing a feature fusion operator based on channel-space attention mechanism within the module, the system adaptively learns the relative weights of each view in both channel and spatial domains. Feature fusion is then performed based on these weights, enhancing the description of patient features and improving classification performance. This allows for the automatic analysis of multi-view images of patients and the output of preliminary classification results indicating whether the patient has a difficult or non-difficult airway, assisting physicians in airway assessment.

[0008] The technical solution of this invention is as follows:

[0009] A multi-view image classification system based on adaptive dual-domain weighted feature fusion includes a processor and a memory; the memory is used to store program instructions executed by the processor.

[0010] When the program instructions are executed by the processor, the multi-view image classification system performs the following operations:

[0011] The multi-view feature extraction module extracts features from each viewpoint of the input multi-view airway image to obtain multi-view features.

[0012] The adaptive dual-domain weighted feature fusion module, based on the channel space attention mechanism, adaptively assigns weights to the multi-view features in the channel dimension and spatial dimension, and performs weighted fusion of the multi-view features according to the weights to generate a fused feature descriptor.

[0013] The fused feature descriptor is processed by the feature classification module to output the airway type discrimination result of the multi-view airway image.

[0014] In a further preferred embodiment, the multi-view feature extraction module adopts a convolutional neural network structure with shared weights.

[0015] In a further preferred embodiment, the multi-view feature extraction module uses the convolutional pooling layer of the first half of VGG16.

[0016] In a further preferred embodiment, the adaptive dual-domain weighted feature fusion module first calculates the global information of each view feature in the channel dimension to generate channel attention weights, then calculates the importance of each pixel position in the spatial dimension to generate spatial attention weights, and finally combines the channel attention weights and spatial attention weights to form the final feature weighting coefficients.

[0017] In a further preferred embodiment, the processing procedure of the adaptive dual-domain weighted feature fusion module is as follows:

[0018]

[0019]

[0020]

[0021] in and These refer to channel attention weights and spatial attention weights, respectively. This indicates element-wise multiplication. It is a view feature The output formed after channel attention weighting, yes The output formed after spatial attention weighting; yes One of the elements, It is a feature after fusion. One of the elements.

[0022] In a further preferred embodiment, the channel attention weights are applied to the input view features by the channel attention module. The result of the processing is:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] in yes The feature map of the c-th channel, yes One of the elements, and These represent the average pooling and max pooling processes, respectively. and These represent the average pooling outputs. and max pooling output The feature map of the c-th channel; and This represents two fully connected layers that together form an MLP network. The output size is ,in For the reduction ratio, The output size is ; Convert the two outputs of the MLP and The channel attention weights can be obtained by summing the elements and then activating them with the sigmoid function. .

[0029] In a further preferred embodiment, the spatial attention weights are applied to the input features by a spatial attention module. The result of the processing is:

[0030]

[0031]

[0032]

[0033]

[0034] in Input features All channels are at the same pixel position A collection of elements. yes One of the elements, and These represent the average pooling outputs. and max pooling output In position An element at a given position; the output of average pooling and max pooling. By splicing along the channel, we obtain Then on Perform convolution operations. This represents a convolution operation; the spatial attention weights can be obtained by activating the output of the convolution using the sigmoid function. .

[0035] A further optimized approach involves training a classification model comprised of a multi-view feature extraction module, a dual-domain adaptive weighted feature fusion module, and a feature classification module through the following process:

[0036] (1) The classification model is pre-trained using a public image dataset to obtain the pre-training weights of the classification model;

[0037] (2) Construct a multi-view difficult airway image dataset, divide the dataset into training set and test set according to the cross-validation data partitioning method, and then perform data augmentation, size adjustment and other preprocessing on the images in the dataset;

[0038] (3) The pre-trained classification model is fine-tuned on the pre-processed training set and then tested and verified on the pre-processed test set to obtain the classification results of difficult airways and non-difficult airways of the test set samples; after cross-validation, the statistical results of classification accuracy are obtained.

[0039] In a further preferred embodiment, the system employs Grad-CAM technology to visualize the hotspot regions of interest in the input image, thereby improving the interpretability of the classification results.

[0040] Beneficial effects

[0041] The multi-view image classification system based on adaptive dual-domain weighted feature fusion proposed in this invention has at least the following beneficial effects:

[0042] (1) This invention constructs a multi-view image classification model based on adaptive dual-domain weighted feature fusion. This model consists of three modules: multi-view feature extraction, dual-domain adaptive weighted feature fusion, and feature classification. It aims to extract, aggregate, and classify deep features in a data-driven manner. Compared to most existing systems that use single-view images to identify difficult airways, this application uses multi-view images of the patient as input and automatically learns more comprehensive and richer information from the multi-view images for difficult airway identification, resulting in higher accuracy and reliability of the identification results.

[0043] (2) This application adopts a feature fusion method based on dual-domain adaptive weighting. By introducing a channel space attention mechanism during the feature fusion process, the relationship between different views is explored. The relative weights of each view feature are adaptively learned in the dual-domain channel space, and then the feature is fused based on the weights, thereby enhancing the representational ability and discriminative power of the fused features. Compared with the existing simple fusion methods that use feature concatenation or addition, the proposed feature fusion method considers the importance of information at different views and different pixel positions in each view, focuses on information in key areas, and suppresses information in irrelevant areas. This is more in line with the visual diagnostic mechanism of human doctors and thus helps to improve classification performance.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0046] Figure 1 This diagram illustrates the model structure of a multi-view image classification system based on adaptive dual-domain weighted feature fusion in an exemplary embodiment of this application.

[0047] Figure 2 This diagram illustrates the VGG16 network structure in an exemplary embodiment of this application.

[0048] Figure 3 This diagram illustrates the structural composition of the channel space attention mechanism in an exemplary embodiment of this application.

[0049] Figure 4 This diagram illustrates the structural composition of the channel attention module in an exemplary embodiment of this application.

[0050] Figure 5 This diagram illustrates the structural composition of the spatial attention module in an exemplary embodiment of this application.

[0051] Figure 6 This diagram illustrates the dataset partitioning of the five-fold cross-validation method in an exemplary embodiment of this application.

[0052] Figure 7 This is a schematic diagram showing the Grad-CAM visualization results in an exemplary embodiment of this application. Detailed Implementation

[0053] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0054] This embodiment aims to meet the requirements of high accuracy and high reliability for automatic airway identification in difficult situations, assist doctors in airway assessment, and solve the problems of insufficient information contained in single-view images and limited feature representation capabilities due to simple fusion, as detailed below:

[0055] First, addressing the problem that most existing difficult airway identification systems use single-view images as input, which contain insufficient information and result in low accuracy and reliability in difficult airway identification, this invention integrates multi-view images to determine the patient's airway type and employs a multi-view image classification network to improve the accuracy and reliability of difficult airway identification.

[0056] Second, while some existing systems for identifying difficult airways use multi-view images as input, they employ simple feature fusion techniques such as feature concatenation or addition, failing to consider the inherent relationships between different views. This results in insufficient representational power of the fused features, limiting recognition performance. To address this issue, this invention designs a novel feature fusion method to uncover the interrelationships between different views and different pixel positions within each view, achieving more effective feature fusion and further improving the accuracy of difficult airway identification.

[0057] This embodiment proposes a multi-view image classification system based on adaptive dual-domain weighted feature fusion, which includes a processor and a memory; the memory is used to store program instructions executed by the processor.

[0058] like Figure 2 As shown, when the program instructions are executed by the processor, the multi-view image classification system performs the following operations using three modules: a multi-view feature extraction module, a feature fusion module based on dual-domain adaptive weighting, and a feature classification module:

[0059] Through multi-view feature extraction module The features of each viewpoint image in the input multi-view airway image are extracted to obtain multi-view features. In this embodiment, VGG16 convolutional pooling layers are used as the feature extraction structure to extract multi-view features. All viewpoint images are processed by the same feature extraction structure with the same parameters, that is, a convolutional neural network structure with shared weights.

[0060] Suppose the input dataset to be identified includes N patient samples, and each sample has V images from different viewpoints. Then this dataset can be denoted as... . Specifically, This refers to the first [item] in the dataset. One example, It refers to the first The first of the samples Individual views (perspective images) It refers to the first Example The corresponding label. In this embodiment, Each example has the following: Figure 1 Input images from the five perspectives shown. This indicates whether the airway is difficult or not.

[0061] Multi-view feature extraction module Responsible for Automatically extracting features from each view can be described as follows: ,in , It contains features from V views. In this embodiment, the convolutional pooling layer of the first half of VGG16 is used as the feature extraction structure. For example... Figure 2 As shown, VGG16 consists of 13 convolutional layers and 3 fully connected layers. After each convolutional layer, the feature map is fed into the ReLU function. After two or three convolutional layers, the feature map is max-pooled, which reduces the dimensionality of the features, decreases computation, and preserves the rotation invariance of the image. Figure 2 The convolutional pooling structure in the first half of the module is used to extract features from multi-view images. The output features are then fed into an adaptive dual-domain weighted feature fusion module for further processing. It's worth noting that the feature extraction structures for all views share the same parameters.

[0062] Through the adaptive dual-domain weighted feature fusion module The extracted V viewpoint features are fused into a concise and efficient feature descriptor, represented as follows: To enable the system to focus more on key views and key regions within each view during feature fusion, this embodiment introduces a channel spatial attention mechanism. This mechanism adaptively assigns weights to the multi-view features in both channel and spatial dimensions, and then performs weighted fusion of the multi-view features based on these weights to generate a fused feature descriptor, thereby enhancing the representational and discriminative power of the fused features.

[0063] like Figure 1 As shown, the features of each view All features undergo adaptive weighting via channel space attention mechanism, resulting in weighted features. Then, the attention is fused using a summation operator. The channel-space attention mechanism consists of two parts: a channel attention module and a spatial attention module, such as... Figure 3 As shown. The proposed feature fusion process can be expressed by the following formula:

[0064] (1)

[0065] (2)

[0066] (3)

[0067] Here and These refer to the relative weights generated by the channel attention module and the spatial attention module, respectively. This indicates element-wise multiplication. yes The output formed after channel attention weighting, yes The output formed after spatial attention weighting. yes One of the elements, It is a feature after fusion. One of the elements.

[0068] Figure 4 This describes the specific structure of the channel attention module. First, average pooling and max pooling layers are used to process the input feature maps, respectively. Their outputs are then fed into a shared multilayer perceptron (MLP) network. Element-wise summation is then used to merge the two outputs, and finally, the sigmoid function is used to activate the merged feature vector to obtain the channel attention weights. This process can be represented by the following formula:

[0069] (4)

[0070] (5)

[0071] (6)

[0072] (7)

[0073] (8)

[0074] in yes The feature map of the c-th channel, yes One of the elements, and These represent the average pooling and max pooling processes, respectively. and These represent the average pooling outputs. and max pooling output The feature map of the c-th channel. and This represents two fully connected layers that together form an MLP network. The output size is ,in For the reduction ratio, The output size is The two outputs of the MLP and The channel attention weights can be obtained by summing the elements and then activating them with the sigmoid function. .

[0075] Figure 5 This describes the specific structure of the spatial attention module. The feature vector after adjustment by the channel attention module. The input to the spatial attention module is first subjected to average pooling and max pooling operations along the channel axis. Then, the two resulting feature maps are concatenated. Next, a convolutional layer transforms the feature maps, and finally, the sigmoid function is used to activate the feature maps to obtain the spatial attention weights. This process can be represented by the following formula:

[0076] (9)

[0077] (10)

[0078] (11)

[0079] (12)

[0080] in Input features All channels are at the same pixel position A collection of elements. yes One of the elements, and These represent the average pooling outputs. and max pooling output In position An element at position 1. The output of average pooling and max pooling. By splicing along the channel, we obtain Then on Perform convolution operations. This represents a convolution operation with 2 input channels, 1 output channel, and a kernel size of 7×7. The spatial attention weights are obtained by activating the convolution output with a sigmoid function. .

[0081] Through feature classification module For the fusion feature descriptor The system performs classification processing and outputs airway type discrimination results for multi-view airway images. (Difficult airway / Non-difficult airway).

[0082] The feature classification module consists of fully connected layers of VGG16, such as... Figure 2 The latter half of the structure. Output of the feature fusion module. First, the data is fed into a fully connected layer, then activated by a ReLU function before being fed into a dropout layer to alleviate overfitting. This is followed by another combination of a fully connected layer, ReLU activation, and dropout layer, and finally another fully connected layer. To convert the classification module's output values ​​into a probability distribution within the range [0, 1] with a sum of 1, a softmax function is added after the last fully connected layer. The classification module outputs a vector containing G elements, i.e., P i = M cl ( O i ) = [ p i , 1 ,..., p i , g ,..., p i , G ] In this embodiment, G=2, indicating that there are two types of airways. Indicates the first Example Belongs to the The probability of the class. If If it is the largest value, then the sample is determined to belong to the first... Class. This process can be expressed by the formula:

[0083] (13)

[0084] here It is the model to the sample The generated predicted categories, in this embodiment It belongs to either a difficult airway or a non-difficult airway.

[0085] The classification model described above, consisting of a multi-view feature extraction module, a dual-domain adaptive weighted feature fusion module, and a feature classification module, needs to be trained before it can be used in practice. The specific training process is given below:

[0086] The training process includes:

[0087] (1) The classification model is pre-trained using a public image dataset to obtain the pre-training weights of the classification model.

[0088] In this embodiment, the publicly available image dataset used is the ImagNet dataset. The publicly available image dataset is input into the classification model, and the classification model is trained using the publicly available image dataset. The model weights after training are the pre-trained weights of the classification model.

[0089] (2) Construct a multi-view difficult airway image dataset, divide the dataset into training and test sets according to the five-fold cross-validation data partitioning method, and then perform data augmentation, size adjustment and other preprocessing on the images in the dataset; the specific process is as follows:

[0090] Multi-view patient images were collected to construct a multi-view difficult airway image dataset. Five facial and neck images from different perspectives were collected from each patient according to strict data acquisition standards, such as... Figure 1The input images are shown in the image. Clinically, doctors often use these perspectives to determine whether a patient has a difficult airway. The data collection criteria are as follows: The first image is a frontal view of the patient, with the patient instructed to open their mouth to the maximum extent, showing the degree of mouth opening (assessing the patient's mouth opening); the second image is a frontal view of the patient, in an upright sitting position with the head kept neutral, opening the mouth and forcefully extending the tongue without making a sound, observing the palatal arch, uvula, and soft palate (which can reflect the patient's Malcolm Glasgow Coma Scale and oropharyngeal condition); the third image is a side view of the patient, with the patient instructed to sit upright with the head kept neutral, capturing the complete neck shape (capturing the patient's natural appearance, reflecting whether the patient is obese or has anatomical deformities); the fourth image is a side view of the patient, with the patient instructed to tilt their neck back to the maximum extent; the fifth image is a side view of the patient, with the patient instructed to flex their neck forward to the maximum extent (the fourth and fifth images show the patient's head and neck mobility).

[0091] In this embodiment, 3290 images from 481 patients were collected. Three professional anesthesiologists labeled the airway type of each patient based on five images from each patient. Based on their combined labeling results, each sample in the dataset was labeled as either a difficult or non-difficult airway. The anesthesiologists concealed their airway assessment results from each patient during the labeling process. The final dataset of 481 samples contained 366 (76.09%) non-difficult airway samples and 115 (23.91%) difficult airway samples.

[0092] The dataset was divided into training and test sets using a five-fold cross-validation method. The training and test sets could not contain images of the same patient. Figure 6 As shown, the five-fold cross-validation method divides the dataset into five equal parts, using four parts (80%) as the training set and the remaining part (20%) as the test set, resulting in five different combinations of training and test sets, Set 1 to Set 5. In Set 1 to Set 5, the training set contains 385 samples (294 non-difficult airways and 91 difficult airways), and the test set contains 96 samples (72 non-difficult airways and 24 difficult airways). However, the patient samples included in these five test sets are completely different.

[0093] Because the number of samples in the difficult and non-difficult airway categories in each training set is imbalanced, with the number of non-difficult airway patients being approximately three times that of difficult airway patients, this can impact model training. To mitigate this issue, data augmentation was employed to increase the number of non-difficult airway patients by 5 times and the number of difficult airway patients by 16 times, increasing the data volume while ensuring a balance across all categories. Specific data augmentation methods included horizontal flipping, random rotation at a certain angle, scaling, adjusting brightness, and adjusting contrast. After data augmentation, the number of samples in the training sets (Sets 1-5) became 2926 (1470 non-difficult airway patients / 1456 difficult airway patients), while the number of samples in the test set remained unchanged at 96 (72 non-difficult airway patients / 24 difficult airway patients). The distribution of the original dataset after five-fold cross-validation and data augmentation is shown in Table 1.

[0094] Table 1 Dataset Distribution

[0095]

[0096] To accommodate the input size requirements of VGG16, the dimensions of all images in the dataset were adjusted to 224×224. Directly resizing images with unequal dimensions to squares would cause geometric distortion, affecting the representation of the original information in the image. Therefore, in this step, all images were proportionally adjusted. When the longer side was adjusted to 224, the shorter side was less than 224, so the insufficient image area was filled with a 0-pixel value. After this step, all images were adjusted to a 224×224 dimension without distortion.

[0097] (3) The pre-trained classification model is fine-tuned on the pre-processed training set, and then tested and validated on the pre-processed test set to obtain the classification results (difficult airway / non-difficult airway) of the test set samples; after five-fold cross-validation, the statistical results of the classification accuracy are obtained. The specific process is as follows:

[0098] First, a fine-tuned classification model is trained on a training set. This involves the following steps: ① loading the weights of the pre-trained classification model, ② extracting data from the training data... Randomly sample a batch of data {( )|𝑖=1,2,..., Send to the network, Refers to the sample The tag, Batch size, ③ The network begins forward propagation and calculates the cross-entropy loss value. The formula for calculating the cross-entropy loss is:

[0099] (14)

[0100] in ④ The number of training samples; ④ The parameters of the classification model are updated using the backpropagation algorithm. In the t-th iteration, the parameters The update formula is shown below:

[0101] (15)

[0102] in This represents the learning rate used to train the classification model. One training iteration is completed when all the training data has been sampled in batches. Since the classification model iterates a certain number of times... Therefore, it is executed repeatedly. Steps ②-④ are then taken to complete the training of the classification model.

[0103] Next, the trained classification model is tested and validated on a corresponding test set. This includes: ① setting the test set... Each sample All samples are fed into the trained classification model, which then outputs the model's correct understanding of the samples. Classification results (Difficult airway / Non-difficult airway). ②Predict the results. Real labels of the sample The comparison is performed, and those that match indicate correct classification. ③ The classification recognition rate for this test set is obtained by dividing the number of correctly classified samples by the total number of samples in the test set.

[0104] The five-fold cross-validation method is used to repeat the above training and testing steps on the five combinations of training and test sets to obtain the five classification recognition rates under the five test sets, such as... Figure 6 As shown. The average of these five classification accuracies is the final classification accuracy of the classification model.

[0105] Table 2 compares the proposed system with several representative deep learning-based systems for difficult airway identification in recent years. To ensure fairness, all systems were re-evaluated on the dataset constructed in this application. Aguilar et al. trained and tested MobileNetV2 on single-view oral images containing Mahalanobis grading information. Hayasaka et al. trained VGG16 using 16 single-view images per patient and determined the best-performing view as the optimal view. This application reproduces its method using single-view images in the dataset with angles close to this optimal view. For Wang et al.'s method, this application performs channel-level data merging on multi-view images and trains ResNet18 in a fully supervised manner. For Xia et al.'s method, this application uses ResNet-18 for multi-view feature extraction followed by logistic regression for difficult airway prediction. In addition to classification accuracy, Table 2 also presents sensitivity, specificity, and F1 score metrics. On the constructed multi-view difficult airway image dataset, the proposed system significantly outperforms other deep learning-based difficult airway assessment methods in all metrics. The proposed multi-view image classification system achieves a difficult airway identification accuracy of 93.54%, sensitivity of 83.33%, specificity of 96.94%, and F1 score of 86.53%.

[0106] Table 2 compares the performance metrics of the proposed system with existing deep learning-based difficult airway identification systems.

[0107]

[0108] Finally, this embodiment also employs Grad-CAM technology to visualize the hotspot regions that the classification model focuses on in the input image, improving the interpretability of the classification results. This step includes:

[0109] The multi-view images are input into the trained classification model for inference. The output of the last convolutional layer of the multi-view feature extraction module in the classification model is visualized using Grad-CAM to obtain the visualization result.

[0110] Figure 7 The Grad-CAM visualization results of the proposed system on the second to fifth input images are shown. It can be clearly seen that the hot spots in the input images are better concentrated in the posterior palate region containing Mahalanobis grading information and the neck region containing information on neck length, neck circumference, and head and neck mobility. The system proposed in this application can focus on information in key areas of the image and can learn well to extract key features that are helpful for identifying difficult airways from patient images.

[0111] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A multi-view image classification system based on adaptive dual-domain weighted feature fusion, characterized in that: It includes a processor and a memory; the memory is used to store program instructions executed by the processor; When the program instructions are executed by the processor, the multi-view image classification system performs the following operations: The multi-view feature extraction module extracts features from each viewpoint of the input multi-view airway image to obtain multi-view features. The adaptive dual-domain weighted feature fusion module, based on the channel space attention mechanism, adaptively assigns weights to the multi-view features in the channel dimension and spatial dimension, and performs weighted fusion of the multi-view features according to the weights to generate a fused feature descriptor. The adaptive dual-domain weighted feature fusion module first calculates the global information of each view feature in the channel dimension to generate channel attention weights, then calculates the importance of each pixel position in the spatial dimension to generate spatial attention weights, and finally combines the channel attention weights and spatial attention weights to form the final feature weighting coefficients. The processing procedure of the adaptive dual-domain weighted feature fusion module is as follows: in and These refer to channel attention weights and spatial attention weights, respectively. This indicates element-wise multiplication. It is a view feature The output formed after channel attention weighting, yes The output formed after spatial attention weighting; yes One of the elements, It is a feature after fusion. One of the elements; The fused feature descriptor is processed by the feature classification module to output the airway type discrimination result of the multi-view airway image.

2. The multi-view image classification system based on adaptive dual-domain weighted feature fusion according to claim 1, characterized in that: The multi-view feature extraction module adopts a convolutional neural network structure with shared weights.

3. The multi-view image classification system based on adaptive dual-domain weighted feature fusion according to claim 2, characterized in that: The multi-view feature extraction module uses the first half of the VGG16 convolutional pooling layer.

4. The multi-view image classification system based on adaptive dual-domain weighted feature fusion according to claim 1, characterized in that: The channel attention weights are applied to the input view features by the channel attention module. The result of the processing is: in yes The feature map of the c-th channel, yes One of the elements, and These represent the average pooling and max pooling processes, respectively. and These represent the average pooling outputs. and max pooling output The feature map of the c-th channel; and This represents two fully connected layers that together form an MLP network. The output size is ,in For the reduction ratio, The output size is ; The two outputs of the MLP and The channel attention weights can be obtained by summing the elements and then activating them with the sigmoid function. .

5. The multi-view image classification system based on adaptive dual-domain weighted feature fusion according to claim 4, characterized in that: The spatial attention weights are applied to the input features by the spatial attention module. The result of the processing is: in Input features All channels are at the same pixel position A collection of elements. yes One of the elements, and These represent the average pooling outputs. and max pooling output In position An element at a given position; the output of average pooling and max pooling. By splicing along the channel, we obtain Then on Perform convolution operations. This represents a convolution operation; the spatial attention weights can be obtained by activating the output of the convolution using the sigmoid function. .

6. The multi-view image classification system based on adaptive dual-domain weighted feature fusion according to claim 1, characterized in that: The classification model, consisting of a multi-view feature extraction module, a feature fusion module based on dual-domain adaptive weighting, and a feature classification module, is trained through the following process: (1) The classification model is pre-trained using a public image dataset to obtain the pre-training weights of the classification model; (2) Construct a multi-view difficult airway image dataset, divide the dataset into training set and test set according to the cross-validation data partitioning method, and then perform data augmentation, size adjustment and other preprocessing on the images in the dataset; (3) The pre-trained classification model is fine-tuned on the pre-processed training set and then tested and verified on the pre-processed test set to obtain the classification results of difficult airways and non-difficult airways of the test set samples; after cross-validation, the statistical results of classification accuracy are obtained.

7. The multi-view image classification system based on adaptive dual-domain weighted feature fusion according to claim 1, characterized in that: The system uses Grad-CAM technology to visualize the hotspot regions of interest in the input image, thereby improving the interpretability of the classification results.

Images