Medical image and electronic text information multi-modal deep learning classification method and system based on XG-YOLO11 model
By employing a multimodal deep learning approach using the XG-YOLO11 model, the problems of low accuracy and feature redundancy in medical image classification were solved, achieving efficient image classification and improving system performance.
Patent Information
- Application Number
- CN202510005930.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-03
- Publication Date
- 2026-01-16
AI Technical Summary
Existing image classification methods have low accuracy and redundant features in the field of medical imaging. Traditional machine learning is highly dependent on it, and deep learning models have redundant features in multimodal data processing, resulting in low system efficiency.
A multimodal deep learning method based on the XG-YOLO11 model is adopted. Feature selection is carried out through image preprocessing, feature extraction and fusion, Pearson correlation analysis and maximum correlation minimum redundancy algorithm. Combined with gradient boosting tree for classification, redundant features are reduced and classification accuracy is improved.
It improves the accuracy of medical image classification, reduces the waste of computing resources, avoids overfitting problems, and improves system efficiency.
Smart Images
Figure CN121353718A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of image processing and computer vision, and particularly relates to an image classification method and system combining a convolutional neural network YOLO11 and a gradient boosting tree for image and text information. BACKGROUND
[0002] The overall promotion of artificial intelligence (AI) theory modeling, technological innovation, software and hardware upgrading, etc. is triggering a chain breakthrough, accelerating the intelligent development of various disciplines. The medical field is an important application field and a frontier of exploration of artificial intelligence. Existing image classification methods include traditional machine learning methods and deep learning methods, but traditional machine learning methods have problems such as low classification accuracy and strong dependence on feature extraction. With the advancement of technology and the auxiliary improvement of data, computer-aided diagnosis systems are gradually used to evaluate medical images to improve clinical diagnosis and provide more medical assistance information for doctors. Deep learning thus emerged as the times require. Deep learning is a branch of artificial intelligence, which has advantages over humans in processing large data, complex multi-dimensional data, and deeply mining potential features of data. It can automatically identify, learn and master the internal relationship of data, and then predict new data by applying the trained model. In visual recognition tasks, for the diagnosis and analysis of medical images, convolutional neural networks (CNN) have shown high performance, which can accurately extract features of the region of interest and use the features as input for other algorithms. Compared with traditional manually designed features, visual features extracted based on convolutional neural networks have better description ability. However, there are few high-accuracy image classification methods, and the classic network is used more habitually. The YOLO model is a commonly used model in the field of target detection, which aims to expand its application to the field of medical image classification. At present, the data types of existing classification systems are single, and a multi-modal classification system and method based on deep learning is needed to improve classification efficiency. At the same time, the features extracted by the deep learning network and the feature number of the text information are relatively large, and there is redundancy. An improved algorithm for reducing feature redundancy can improve system efficiency. SUMMARY
[0003] The present application provides an image classification system and method based on a multi-modal deep learning model to solve the above-mentioned defects.
[0004] The basic scheme provided by the present application is an image classification method based on multi-modal deep learning, which comprises the following steps:
[0005] S1: image preprocessing, preprocessing the input image and text image, preprocessing the input image and text information to reduce noise and information redundancy, One-Hot encoding the text data, and effectively enhancing and expanding the image data;
[0006] S2: Feature extraction, a pre-trained YOLO11 model is used to extract and abstract features from the pre-processed images, and a multi-layer perceptron (MLP) is used to extract features from the pre-processed text information, and then the two types of features are reasonably spliced and fused; the model is:
[0007] The method of transfer learning uses the structure, weights and bias parameters of the already trained YOLO11 model, and compresses the second-to-last layer of the model into a one-dimensional vector. In this way, the pre-trained model is loaded to quickly start the training process.
[0008] In order to avoid the influence of redundant features on the performance of the model, an improved algorithm of Pearson correlation analysis, maximum correlation minimum redundancy (mRMR) and weighted results is used to gradually select the most important features for the prediction task. Ensure that the feature set after feature selection can support the final classification task, while avoiding the waste of computing resources and overfitting problems caused by high-dimensional features.
[0009] S3: Image classification, using gradient boosting trees to map feature representations to class space to distinguish different classes, and further optimize the classification effect through model tuning, cross-validation and hyperparameter adjustment;
[0010] S4: Image storage and result output, store the results output by the classifier module in the designated storage medium, and output the feature importance analysis and prediction result evaluation.
[0011] Further, S1 includes:
[0012] S1-1: Image scaling: adjust the image to a uniform size of 640*640;
[0013] S1-2: Image color adjustment: set a random range to adjust the brightness and image contrast of the image, and expand the data by 2 times;
[0014] S1-3: Denoising: remove noise and impurities in the image;
[0015] S1-4: Normalization: limit the pixel value of the image within a certain range;
[0016] S1-5: Image enhancement: rotate, zoom, crop and other operations are performed on the image;
[0017] S1-6: Text information encoding: One-Hot encoding is performed on the case information.
[0018] Further, S2 includes:
[0019] S2-1: Convolutional Neural Network: Using the newly released YOLO11 deep learning model that has been trained on the ImageNet dataset, feature extraction and abstraction are performed on the preprocessed image;
[0020] S2-2: Multilayer Perceptron: Using a multilayer perceptron (MLP) model to extract and abstract features from preprocessed tabular data;
[0021] S2-3: Feature Representation: The outputs of the YOLO11 model and the MLP model are fused into a comprehensive feature vector, which serves as the feature representation of the image. The output of this feature layer is extracted to form a feature vector.
[0022] S2-3: Freeze the parameters of YOLOv11 to ensure that they are not updated during training.
[0023] Design steps to avoid the impact of redundant features on model performance; for each pair of features x i and x j Calculate the Pearson correlation coefficient: in, This indicates that the k-th sample in feature x i The value on, It is feature x i The mean, γ ij It is feature x i and x j The Pearson correlation coefficient between them. Set a threshold T, if |γ| ij If |>Τ, then the two features are considered highly correlated, and one of the features is removed; then the maximum correlation minimum redundancy algorithm is used to further refine the features, first calculating the x of each feature. i The correlation with the target y is measured using mutual information (MI): Correlation is measured by redundancy between computational features, and redundancy is also calculated using mutual information. Finally, the objective function that can select features that are highly correlated with the target value and have lower redundancy than other features is calculated: Where S is a selected feature set, x i These are candidate features, resulting in the current feature set I. Finally, the feature sets I' of all deleted features are counted. Let δ be the weight, and use the weights assigned to δ to obtain the most important feature set, Im = {{x} i ∈{I-I'}}*(1-δ)AND{x i ∈I'}*δ};
[0024] Furthermore, S3 includes:
[0025] S3-1: Selecting the classifier model: XGBOOST model is used as the classifier. In the training process of gradient boosting trees, the number of leaves and the depth of the tree are the main parameters to control the complexity of the tree model, which are initialized respectively;
[0026] S3-2: Training the classifier: training the determined classifier model using the training data to obtain the optimal training model;
[0027] S3-3: Classification: input the feature representation into the classifier for classification.
[0028] The medical image and electronic text information multi-modal deep learning classification system based on the XG-YOLO11 model includes a preprocessing module, a feature extraction module, a classifier module, and a storage module;
[0029] The preprocessing module is responsible for preprocessing the input image and text information to reduce noise, information redundancy, and data enhancement;
[0030] The feature extraction module uses a deep learning model to extract and abstract features from the preprocessed image and text information, and reasonably splices and fuses the two types of features;
[0031] The classifier module uses a gradient boosting tree classifier to classify the image features obtained from the feature extraction module;
[0032] The storage module is responsible for storing the output results of the classifier module in the designated storage medium, and outputting the feature importance analysis and prediction result evaluation. BRIEF DESCRIPTION OF DRAWINGS
[0033] Figure 1 The flowchart of the embodiment of the present application;
[0034] Figure 2 The YOLO11 model basic framework diagram in S2 of the embodiment of the present application;
[0035] Figure 3 The system block diagram of the embodiment of the present application;
[0036] Figure 4 The file storage form diagram of the data set format required by the model of the embodiment of the present application. DETAILED DESCRIPTION
[0037] The present application will be further described below in conjunction with the drawings and the specific embodiments of the present application. The specific embodiments described herein are only to explain the present application, but not to limit the present application. In addition, in order to facilitate the description, only part of the embodiments of the present application are shown in the drawings, but not all the embodiments.
[0038] The present application will be further described below in conjunction with the drawings and the specific embodiments of the present application. The specific embodiments described herein are only to explain the present application, but not to limit the present application. In addition, in order to facilitate the description, only part of the embodiments of the present application are shown in the drawings, but not all the embodiments.
[0039] The application provides a kind of based on XG-YOLO11 model medical image and electronic text information multi-modal deep learning classification method and system, with knee joint image osteoporosis classification as example, input image is divided into normal, osteopenia, osteoporosis.As shown in Figure 1 CT or X-ray scanning data obtained is imported into the system, first, image sample and case information are preprocessed and expanded, the original input image processed is input into YOLO11 neural network model for training, obtains trained YOLO11 model, after model calculation, the number of features that each input image can extract, the image features obtained and text information mixed features are combined with the corresponding image classification label as the input data of gradient boosting tree to train gradient boosting tree, obtains trained gradient boosting tree model, finally, input image is identified and classified according to gradient boosting tree classifier.In order to better illustrate the embodiment of the application, the framework diagram of the algorithm implementation of the application is shown, as Figure 1 , specific method includes the following steps:
[0040] Step 1, first, determine whether the data set conforms to the specified format of YOLO model, its data set storage form is shown in Figure 4 ; wherein image and label file need to be named, image is stored with “.jpeg”, “.png”, “.jpg” etc. representing image suffix extension, and label is stored with “.txt” representing pure text suffix extension. Then, input image is preprocessed and sample is expanded, the image sample expansion at least contains one of the following methods:
[0041] Random cropping: according to the size of input image, select appropriate cropping size, the basic input is still 640*640, randomly crop the original image, and expand the data by 2 times.
[0042] Image flipping: up and down, left and right, diagonal line flipping is carried out on the image, and the data is expanded by 3 times.
[0043] Image color adjustment: set random range to adjust the brightness and image contrast of image, and expand the data by 2 times.
[0044] Batch normalization method is used for image data preprocessing;
[0045] One-Hot encoding is used for text data normalization.
[0046] Step 2, train convolutional neural network model YOLO11:
[0047] According to the pre-training data set of the model and the number of data sets to be trained, an all-connection output layer is added to the improved model, and an all-connection layer composed of several neurons is added according to the number of classes to be classified, so that 3 neurons are needed; the cross-entropy loss function is used for training the data set to be trained, and the weights of each layer are updated by backward propagation according to the size of the loss value of forward propagation, until the loss value of the model tends to converge, the training model is stopped, the added output layer is removed, and the deep learning model is obtained, such as Figure 3 , wherein the CBS module represents the Conv+BN+SiLU of pytorch, Conv is a convolutional layer, BN is a batch normalization layer, SiLU is an activation function, DWConv represents a depth separable convolution, and the key training settings include batch size, learning rate, momentum and weight decay. In addition, the selection of the optimizer, the loss function and the training data set combination also affects the training process. Epoch is set to 100, batch is set to 16, the random seed number seed of the training is set to 0, freeze is set to none, and the last layer of the model is frozen.
[0048] Step 3, image feature extraction and text feature extraction and fusion:
[0049] First, the YOLO11 model is used to extract image features, and the image data after image sample expansion and data preprocessing is input into the trained YOLO11 model, so that it can learn the features of all categories and correctly identify the category of the image in the new image. The network structure diagram of YOLO11 model is shown in Figure 2 , the neural network comprises:
[0050] A. input layer
[0051] The input layer is that the input image data is processed by the sample expansion and image data preprocessing method proposed in step 1 and used as the input of the model;
[0052] B. hidden layer
[0053] The hidden layer is composed of multiple convolutional layers and pooling layers, and also incorporates a transformer or attention module to improve the globality of feature expression. The convolutional layer is used to extract image features, and different channel numbers are used to control the number of output images. The pooling layer performs down-sampling on the extracted features, and also adds an up-sampling operation of deconvolution to restore the spatial resolution, which is convenient for fusing multi-scale features. A 2*2 size convolution kernel and a size of 2 are used to control the number of output image features;
[0054] C. output layer
[0055] The output layer is a fully connected layer composed of 1024 neurons;
[0056] Then the case text information (18 columns) is extracted using a multi-layer perceptron (MLP). The preprocessed data is input into the MLP model, features are extracted, and a fixed-dimensional feature vector is generated. The network includes:
[0057] A. Input layer
[0058] The input layer is the input text data processed by the data preprocessing method in step 1 as the input of the model;
[0059] B. Hidden layer
[0060] A 3-layer network is used, with 128 neurons decreasing to 32;
[0061] C. The output layer is a fully connected layer composed of 32 neurons;
[0062] The obtained feature vectors of the two modalities are spliced, a part of the feature set is preliminarily screened using the method of calculating the Pearson correlation coefficient, then the features are further selected using the maximum correlation minimum redundancy algorithm, and finally the final most compact feature vector Im is obtained by weighting;
[0063] Step 4, training gradient boosting tree:
[0064] During the training process of the gradient boosting tree, the number of leaves and the tree depth are the main parameters to control the complexity of the tree model, which are set to 63 and 6 respectively. The extracted image features and labels are input into the gradient boosting tree for training to obtain a basic classifier.
[0065] Step 5, obtain classification results:
[0066] The original input image is used as input data, and the most compact feature vector Im in step 3 is used as input features. The first model calculates the input image using the trained YOLO11 model for feature extraction, the second model calculates the extracted features using the gradient boosting tree to obtain the corresponding classification results, the merged results are the classification results of each image, and the image classification results are obtained as the final classification results. The final classification results and feature analysis results are presented to the user through the output module of the system, and the system framework is shown in Figure 3 .
[0067] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the technical solutions of the present application are not limited to the various applications mentioned in the patent embodiments. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application can be made to achieve the various gain effects mentioned herein. Various changes made within the knowledge of those skilled in the art without departing from the purpose of the present application shall be included in the protection scope of the present application.
Claims
1. A method and system for classifying medical images and electronic text information based on XG-YOLO11 model multi-modal deep learning, characterized in that, The method comprises the following steps: S1: image preprocessing, preprocessing the input image and text information to reduce noise and information redundancy, One-Hot encoding of text data, and effective enhancement and expansion of image data; S2: feature extraction, using a pre-trained YOLO11 model to extract and abstract features from the preprocessed image, and using a multi-layer perceptron (MLP) to extract features from the preprocessed text information, and then reasonably splicing and fusing the two kinds of features; the model is: Using the method of transfer learning, the structure, weight and bias parameters of the trained YOLO11 model are used to compress the second-to-last layer of the model into a one-dimensional vector. In this way, the pre-trained model is loaded to quickly start the training process. In order to avoid the influence of redundant features on the performance of the model, the improved algorithm of Pearson correlation analysis and maximum relevance minimum redundancy (mRMR) based on the result is used to gradually select the most important features for the prediction task. Ensure that the feature set after feature selection can support the final classification task, while avoiding the waste of computing resources and overfitting problems caused by high-dimensional features. S3: image classification, using gradient boosting trees to map feature representations to class space to distinguish different classes, and further optimizing the classification effect through model tuning, cross-validation and hyperparameter adjustment; S4: image storage and result output, storing the results output by the classifier module in the specified storage medium, and outputting the feature importance analysis and prediction result evaluation.
2. The XG-YOLO11 model-based medical image and electronic text information multi-modal deep learning classification method and system according to claim 1, characterized in that: The format of the image data in step S1 should meet the official data set input format required by YOLO11, that is, the data is divided into training set and test set (validation set can be selected whether to use), the file should be named and labeled, that is, the data set name / --train / --category 1 / …… / category 2……; the label file has ".txt" as the file extension, and the single label information is <file location><category information>, wherein the category information of the target has been normalized.
3. The XG-YOLO11 model-based medical image and electronic text information multi-modal deep learning classification method and system according to claim 1, characterized in that: S1 includes the following steps: S1-1: image scaling: adjusting the image to a uniform size of 640*640; S1-2: image color adjustment: setting a random range to adjust the brightness and image contrast of the image, and expanding the data by 2 times; S1-3: denoising: removing noise and impurities in the image; S1-4: normalization: limiting the pixel value of the image within a certain range; S1-5: image enhancement: rotating, scaling, cropping and other operations are performed on the image; S1-6: text information encoding: One-Hot encoding of case information.
4. The XG-YOLO11 model-based medical image and electronic text information multi-modal deep learning classification method and system according to claim 2, characterized in that: S2 includes the following steps: S2-1: convolutional neural network: using the newly released YOLO11 deep learning model trained on the ImageNet data set to extract and abstract features from the preprocessed image; S2-2: multi-layer perceptron: using a multi-layer perceptron model (MLP) to extract and abstract features from the preprocessed table data; S2-3: Feature representation: fuse the output of the YOLO11 model and the output of the MLP model into a comprehensive feature vector as the feature representation of the image, extract the output of this feature map layer to form a feature vector; S2-3: Freeze the parameters of YOLOv11 to ensure that they are not updated during the training process.
5. The XG-YOLO11 model-based medical image and electronic text information multi-modal deep learning classification method and system according to claim 2, characterized in that: S2 in order to avoid the design steps of redundant features on the performance of the model; for each pair of features x i and x j , calculate the Pearson correlation coefficient: where, is the value of the kth sample on the feature x i , is the mean of the feature x i , γ ij is the Pearson correlation coefficient between the features x i and x j . Set a threshold T, if |γ ij |>T, consider that the two features are highly correlated, and remove one of the features; then use the maximum correlation minimum redundancy algorithm based on the results again to select features, first find the correlation between each feature x i and the target y, measured by mutual information (Mutual Information, MI): In calculating the correlation between the features, the redundancy is also calculated by mutual information: Finally, calculate the objective function that can select the features that are highly correlated with the target value and have lower redundancy than other features: where S is a selected feature set, x i is a candidate feature, and the feature set I is obtained at this time. Finally, count all the deleted feature set I', and let δ be the most important feature set, Im = {{x i ∈{I-I'}}*(1-δ) AND{x i ∈I'}*δ}.
6. The XG-YOLO11 model-based medical image and electronic text information multi-modal deep learning classification method and system according to claim 3, characterized in that: S3 includes the following steps: S3-1: Select the classifier model: use the XGBOOST model as the classifier, and initialize the number of leaves and the depth of the tree in the training process of the gradient boosting tree, which are the main parameters to control the complexity of the tree model; S3-2: Train the classifier: train the determined classifier model using the training data to obtain the optimal training model; S3-3: Classification: input the feature representation into the classifier for classification.
7. The image storage and result output according to claim 4, characterized in that: S4 mainly includes the following contents: S4-1: Evaluate the classification results using prediction accuracy, recall rate, and F1 coefficient; S4-2: Output the feature importance heat map of the final prediction result.
8. The method and system for classification of medical images and electronic text information based on multi-modal deep learning of XG-YOLO11 model, characterized in that: It includes a preprocessing module, a feature extraction module, a classifier module, and a storage module; The preprocessing module is responsible for preprocessing the input image and text information to reduce noise and information redundancy, and effectively enhancing the data; The feature extraction module uses a deep learning model to extract and abstract features from the preprocessed image and text information, and reasonably splices and fuses the two types of features; The classification module uses the machine learning model XGBOOST to classify the image features obtained from the feature extraction module; The result storage module is responsible for storing the output results of the classifier module in the designated storage medium, and outputting the feature importance analysis and prediction result evaluation.
9. The medical image and electronic text information multi-modal deep learning classification method and system based on the XG-YOLO11 model according to any one of claims 1-8, characterized in that, The image recognition method is applied to medical image classification.