Human medical image classification method based on ultrasound images and CT images

Abstract

The present application relates to the field of human medical image classification, and particularly relates to a human medical image classification method based on ultrasound images and CT images.The technical scheme comprises the following steps: collecting ultrasound images and CT images of the same subject, the images being two-dimensional slices or selected layers of three-dimensional volume data, and converting the images into two-dimensional tensors as input;preprocessing the collected data, extracting ultrasound image features through an ultrasound branch module and CT image features through a CT branch module after preprocessing, then aligning the features output by the ultrasound branch module and the CT branch module in the spatial dimension, performing feature fusion, and finally restoring the fused features to the input resolution by convolution and bilinear interpolation up-sampling, and outputting a segmentation mask of the set region;performing global pooling and full connection mapping on the fused features, and outputting a data classification result.The present application is suitable for classifying human medical images.

Description

Technical Field

[0001] This invention relates to the field of human medical image classification, specifically to a method for classifying human medical images based on ultrasound and CT images. Background Technology

[0002] Most existing AI-assisted diagnostic methods are based on single-modal images or employ simple multimodal feature stitching and weighting. For example, a disease ultrasound image diagnostic prediction method disclosed in CN118521537A extracts global information feature vectors from a thyroid ultrasound image after data preprocessing, using a first model to extract pathological tissue information feature vectors from thyroid nodules, and a third model to extract normal healthy tissue information feature vectors. The feature vectors of the three types of information are then stitched together and input into a fully connected layer. The fully connected layer dynamically selects the weights of different features based on learning, and finally calculates the benign or malignant probability of the thyroid ultrasound image.

[0003] The above scheme uses the entire image for prediction, no longer relying on manual annotation of regions of interest, and can be directly used for clinical diagnosis. It has better integration and achieves indiscriminate prediction of ultrasound images from different sources, showing good practical applicability, similar to the proposed strategy.

[0004] However, the above scheme uses a single modal image, which has incomplete and limited information and cannot provide accurate information for subsequent model classification. At the same time, speckle noise in ultrasound images is not effectively processed, resulting in unstable feature extraction. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a human medical image classification method based on ultrasound images and CT images, thereby improving the accuracy of human medical image classification.

[0006] The present invention achieves the above objectives by adopting the following technical solution: The present invention provides a method for classifying human medical images based on ultrasound images and CT images, comprising:

[0007] S1. Collect data;

[0008] Ultrasound and CT images of the same subject are collected. The images are either two-dimensional slices or selected layers of three-dimensional volume data. They are uniformly converted into two-dimensional tensors as input.

[0009] S2, Data Preprocessing;

[0010] S3. Feature extraction from ultrasound and CT images;

[0011] S4, Feature Fusion;

[0012] S5, Data Classification;

[0013] By using convolution and bilinear interpolation upsampling, the fused features are restored to the input resolution, and the output is a segmentation mask for the set region;

[0014] Global pooling and fully connected mapping are applied to the fused features to output the data classification results.

[0015] Furthermore, step S2 specifically includes:

[0016] Denoising and Enhancement: The BM3D algorithm is used to denoise the ultrasound images, suppress speckle noise, and enhance texture and edge features;

[0017] Normalization processing: Pixel values ​​of ultrasound images and CT images are normalized separately to eliminate modal differences;

[0018] Geometric alignment: Aligning ultrasound and CT images in terms of resolution and spatial scale using interpolation or registration algorithms;

[0019] Data augmentation: Randomly rotate, flip, translate, and adjust the brightness of ultrasound and CT images.

[0020] Furthermore, step S3 specifically includes:

[0021] Ultrasound image features are extracted through an ultrasound branch module, which consists of shallow convolution, MV2 module and lightweight MobileViTBlock. Local convolution + Transformer is used to model small block sequences and extract local texture and boundary features of ultrasound images.

[0022] The CT branch module extracts CT image features. The CT branch module consists of a convolutional patch-embed and several layers of Transformer encoders. It models the global context relationship through a multi-head self-attention mechanism, extracts the global structure and spatial dependency features of the CT image, and the output features are upsampled by interpolation and aligned with the 1×1 convolutional channel to be unified to the same scale as the ultrasound branch features.

[0023] Furthermore, step S4 specifically includes:

[0024] After aligning the features output by the ultrasound branch module and the CT branch module in the spatial dimension, they are concatenated in the channel dimension. The concatenated features are then reweighted through convolutional layers and channel attention mechanisms to generate a unified multimodal representation that combines local texture and global structure.

[0025] The beneficial effects of this invention are as follows:

[0026] This invention employs multimodal fusion, which fuses extracted ultrasound and CT image features. By using convolution and attention mechanisms, it generates a multimodal representation that takes into account both global and local features, thereby improving classification accuracy.

[0027] This invention uses the BM3D algorithm to denoise ultrasound images, suppress speckle noise, and enhance texture and edge features.

[0028] The present invention, MedFusionNet, improves the multimodal fusion effect by combining the advantages of global Transformer and local MobileViT through a cut-and-paste design. Attached Figure Description

[0029] Figure 1 This is a flowchart of a human medical image classification method based on ultrasound images and CT images provided by the present invention;

[0030] Figure 2 This is a structural block diagram of the ultrasonic branch module provided by the present invention;

[0031] Figure 3 This is a block diagram of the MobileViTBlock structure provided by the present invention;

[0032] Figure 4 This is a block diagram of the MobileNetV2 architecture provided by the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0034] This invention provides a method for classifying human medical images based on ultrasound and CT images, such as... Figure 1 As shown, it includes:

[0035] S1. Acquire ultrasound and CT image data;

[0036] Ultrasound and CT images of the same subject are collected. The images are either two-dimensional slices or selected layers of three-dimensional volume data. They are uniformly converted into two-dimensional tensors as input.

[0037] S2, Data Preprocessing;

[0038] Denoising and Enhancement: The BM3D algorithm is used to denoise the ultrasound images, suppress speckle noise, and enhance texture and edge features.

[0039] Normalization: Pixel values ​​of ultrasound and CT images are normalized (e.g., 0–1 or z-score normalization) to eliminate modal differences.

[0040] Geometric alignment: Aligning ultrasound and CT images in terms of resolution and spatial scale (e.g., unifying them to 256×256) using interpolation or registration algorithms.

[0041] Data augmentation includes random rotation, flipping, translation, and brightness adjustment to improve the model's generalization performance.

[0042] S3. Feature extraction from ultrasound and CT images;

[0043] Ultrasound image features are extracted using the ultrasound branching module, such as... Figure 2 As shown, the ultrasound image is sequentially processed through a 3×3 convolution kernel, MobileNetV2, another 3×3 convolution kernel, MobileNetV2, MobileViTBlock, a 3×3 convolution kernel, and finally MobileViTBlock before being output. Small block sequences are modeled jointly using local convolution and Transformer. First, shallow 3×3 convolutions are used to extract local spatial features. Then, a multi-head self-attention mechanism is used to model long-range dependencies, achieving adaptive enhancement of local texture and boundary details, thereby extracting fine-grained texture and boundary features from the ultrasound image. The output resolution is approximately 1 / 8 of the input image; the MobileViTBlock structure is as follows. Figure 3 As shown, the output sequentially passes through a 3×3 convolution kernel, a 1×1 convolution kernel, image segmentation, a Transformer encoder, image stitching, and then back to the previous 3×3 and 1×1 convolution kernels before being output. The MobileNetV2 structure is as follows: Figure 4 As shown, the input is sequentially passed through a 1×1 convolution kernel, a depthwise 3×3 convolution kernel, and a 1×1 convolution kernel before being added to the input and output.

[0044] CT image features are extracted using a CT branch module, which consists of a convolutional patch-embedded module and several Transformer encoder layers. The patch-embedded module divides the CT image into local block embedding vectors and then establishes global dependencies between different spatial locations of the feature maps using a multi-head self-attention mechanism. Combined with residual connections and layer normalization modules, this enhances the spatial correlation modeling capability of the global structure of the CT image, thereby extracting global structure and spatial dependency features. The output features are interpolated and upsampled, aligned with 1×1 convolutional channels, and unified to the same scale as the ultrasound branch features.

[0045] After cutting, delete the MobileViTBlock and InvertedResidual of subsequent stages, keeping only the first two layers of MobileViTBlock, and delete the original deep structure and classification / segmentation header.

[0046] S4, Feature Fusion;

[0047] After aligning the features output from the ultrasound and CT branches in the spatial dimension, they are concatenated in the channel dimension. The concatenated features are then reweighted through convolutional layers and a channel attention mechanism to generate a unified multimodal representation that combines local texture and global structure.

[0048] S5, Data Classification;

[0049] Segmentation subnetwork: Multi-layer convolution and bilinear interpolation upsampling are used to gradually restore the fused features to the input resolution. During the upsampling process, skip connections and feature splicing strategies are combined to preserve high-resolution detail information. Finally, a segmentation mask for lesions or regions of interest is output through 1×1 convolution.

[0050] Classification subnetwork: Performs global pooling and fully connected mapping on the fused features to output the lesion classification result (e.g., benign / malignant).

[0051] S6. Output the results;

[0052] The system outputs segmentation results of lesions or regions of interest, as well as classification results of lesion characteristics. It can further generate structured diagnostic reports, including lesion location, size, probability values, and visual overlays, to assist doctors in clinical decision-making.

[0053] This invention employs BM3D denoising, effectively suppressing noise in medical images and improving the signal-to-noise ratio through similar block matching and 3D transformation. The MobileViT pruning branch focuses on local texture and boundary features, while the Medical Transformer pruning branch focuses on global structure and contextual dependencies. After channel concatenation, the outputs of the two branches are fused through convolution and attention mechanisms to generate a multimodal representation that considers both global and local aspects. By segmenting the subnetwork to locate lesion regions and classifying the subnetwork to output properties or categories, integrated classification is achieved.

[0054] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for classifying human medical images based on ultrasound and CT images, characterized in that, include: S1. Acquire ultrasound and CT image data; Ultrasound and CT images of the same subject are collected. The images are either two-dimensional slices or selected layers of three-dimensional volume data. They are uniformly converted into two-dimensional tensors as input. S2, Data Preprocessing; Denoising and Enhancement: The BM3D algorithm is used to denoise the ultrasound images, suppress speckle noise, and enhance texture and edge features; Normalization processing: Pixel values ​​of ultrasound images and CT images are normalized separately to eliminate modal differences; Geometric alignment: Aligning ultrasound and CT images in terms of resolution and spatial scale using interpolation or registration algorithms; Data augmentation: Random rotation, flipping, translation, and brightness adjustment of ultrasound and CT images; S3. Feature extraction from ultrasound and CT images; Ultrasound image features are extracted through an ultrasound branch module, which consists of shallow convolution, MV2 module and lightweight MobileViTBlock. Local convolution and Transformer are used to model small block sequences to extract local texture and boundary features of ultrasound images. CT image features are extracted through the CT branch module, which consists of a convolutional patch-embed and several layers of Transformer encoders. The global context relationship is modeled through a multi-head self-attention mechanism to extract the global structure and spatial dependency features of the CT image. The output features are upsampled by interpolation and aligned with the 1×1 convolutional channel to be unified to the same scale as the ultrasound branch features. S4, Feature Fusion; After aligning the features output by the ultrasound branch module and the CT branch module in the spatial dimension, they are stitched together in the channel dimension. The stitched features are then reweighted through a convolutional layer and a channel attention mechanism to generate a unified multimodal representation that combines local texture and global structure. S5, Data Classification; By using convolution and bilinear interpolation upsampling, the fused features are restored to the input resolution, and the output is a segmentation mask for the set region; Global pooling and fully connected mapping are applied to the fused features to output the data classification results.

Images