Image classification method and system based on multimodal 3D convolutional neural network

Abstract

This application provides an image classification method and system based on a multimodal 3D convolutional neural network. The method, after acquiring first and second modal images of different modalities, extracts a substance mask using an image segmentation model. Simultaneously, it establishes a positional correspondence between the first and second modal images based on a scaling factor, thereby extracting a first substance region in the first modal image and a second substance region in the second modal image. Then, an image classification model performs feature extraction, modality fusion, and feature classification on the first and second substance regions to obtain a classification result. The method cascades and combines an image segmentation model and an image classification model, employing a multimodal input 3D convolutional neural network to classify the image. The input consists of image data from two different modalities; after feature extraction using the neural network, the two features are cascaded and fused to obtain a classification prediction result, thus improving classification accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image classification method and system based on a multimodal three-dimensional convolutional neural network. Background Technology

[0002] Image classification refers to categorizing images into several classes based on the different features reflected in the image information. Image classification can be applied in various fields, such as medical imaging, artificial intelligence, deep learning, medical image analysis, and lung nodule diagnosis. For example, in the field of early clinical imaging diagnosis of lung nodules, lung nodule screening mainly relies on computed tomography (CT) or positron emission tomography (PET) to image the user's lungs. The user then visually observes the lung images to identify lung nodules and classify them as benign or malignant. Due to subjective factors, time constraints, and differences in interpretation, the analysis of these images has limitations, leading to the risk of misdiagnosis. Therefore, artificial intelligence methods can be used to classify lung images, thereby assisting in the identification of lung nodule categories and improving screening efficiency and accuracy.

[0003] Deep learning technology can be used to automate image classification. Deep learning can automatically learn and extract feature information from different images to perform image classification prediction. While image classification can be achieved using feature information from a single image, in some fields, the feature information from one image is insufficient for accurate classification. For example, in the field of early clinical imaging diagnosis of pulmonary nodules, users refer to the state of surrounding tissues or organs to detect and classify pulmonary nodules. Summary of the Invention

[0004] This application provides an image classification method and system based on a multimodal three-dimensional convolutional neural network to solve the problem of low classification accuracy in image classification.

[0005] In a first aspect, this application provides an image classification method based on a multimodal three-dimensional convolutional neural network, including:

[0006] Acquire a first modal image and a second modal image, wherein the first modal image is an image obtained by imaging the region under test based on computed tomography (CT) technology, and the second modal image is an image obtained by imaging the region under test based on positron emission tomography (PET) technology.

[0007] The first modality image is input into the image segmentation model to obtain the first substance mask output by the image segmentation model, wherein the image segmentation model is a neural network model trained based on the first sample image dataset;

[0008] The first physical mask is scaled based on a scaling factor to obtain a second physical mask, wherein the scaling factor is the ratio of the actual imaging of the region to be tested between the first modal image and the second modal image;

[0009] Extract the first real region from the first modal image according to the first real mask;

[0010] Extract the second real region from the second modal image according to the second real mask;

[0011] The first and second physical regions are input into an image classification model to obtain the classification result output by the image classification model. The classification result includes a first result and a second result. The image classification model is a neural network model trained based on the first sample physical region dataset and the second sample physical region dataset.

[0012] In an optional implementation, the method further includes:

[0013] Obtain a first sample image dataset, which includes multiple first sample images labeled with different substance masks;

[0014] Construct the first neural network model;

[0015] The first neural network model is trained based on the first sample image dataset to obtain an image segmentation model.

[0016] In an optional implementation, after the step of obtaining the first sample image set, the method further includes:

[0017] The first sample image is resampled according to the preset image specifications;

[0018] Adjust the window width and window level of the first sample image according to the preset window width and window level values.

[0019] In an optional implementation, the image segmentation model includes multiple encoders and multiple decoders. The encoder includes convolutional layers and pooling layers, and the decoder includes deconvolutional layers and skip connection layers. The skip connection layers are used to fuse feature maps in the encoder with positional correspondences in the decoder. The feature maps in the encoder are obtained by performing convolution and pooling operations on the first modality image, and the feature maps in the decoder are obtained by performing deconvolution and skip connection operations on the feature maps in the encoder. The positional correspondence is used to characterize the relationship that the feature maps in the encoder and the feature maps in the decoder have the same size.

[0020] In an optional implementation, the step of scaling the second modal image and the first substantial region based on a scaling factor further includes:

[0021] Construct a coordinate system;

[0022] The first modal image and the second modal image are registered according to the coordinate system to obtain the actual imaging correspondence between the first modal image and the second modal image of the test area under the same coordinate system;

[0023] The scaling factor is calculated based on the correspondence.

[0024] In an optional implementation, the method further includes:

[0025] Obtain the sample essence dataset, which includes multiple first sample essence regions and second sample essence regions labeled with different classification result labels;

[0026] Construct a second neural network model;

[0027] The second neural network model is trained based on the actual dataset of the samples to obtain an image classification model.

[0028] In an optional implementation, the image classification model includes a feature extraction module, a feature fusion module, and a feature classification module, and the method further includes:

[0029] The feature extraction module extracts a first substantial feature map in the first substantial region and a second substantial feature map in the second substantial region.

[0030] The first substantial feature map and the second substantial feature map are fused by the feature fusion module to obtain fused features;

[0031] The fused features are mapped to classification results through the feature classification module.

[0032] In an optional implementation, the image classification model includes seven network layers, with the first to fifth network layers each including convolutional layers and pooling layers, the sixth network layer including a connection function, and the seventh network layer including multiple fully connected layers.

[0033] In an optional implementation, the first neural network model is a three-dimensional semantic image segmentation model, and the second neural network model is a multimodal three-dimensional convolutional neural network.

[0034] Secondly, this application provides an image classification system based on a multimodal three-dimensional convolutional neural network, comprising:

[0035] The image acquisition module is used to acquire a first modal image and a second modal image, wherein the first modal image is an image obtained based on a computed tomography scan of the area to be tested, and the second modal image is an image obtained based on a positron emission tomography scan of the area to be tested;

[0036] A segmentation module is configured to input the first modality image into an image segmentation model to obtain a first substance mask output by the image segmentation model, wherein the image segmentation model is a neural network model trained based on a first sample image; and to scale the first substance mask based on a scaling factor to obtain a second substance mask, wherein the scaling factor is the image ratio between the second modality image and the first modality image; and to extract a first substance region in the first modality image according to the first substance mask; and to extract a second substance region in the second modality image according to the second substance mask.

[0037] The classification module is used to input the first physical region and the first physical region into a trained image classification model to obtain the classification result output by the image classification model. The classification result includes a first result and a second result. The image classification model is a neural network model trained based on the first sample physical region and the first sample physical region.

[0038] As can be seen from the above technical solutions, this application provides an image classification method and system based on a multimodal three-dimensional convolutional neural network. The method, after acquiring a first modality image and a second modality image, extracts a substantial mask using an image segmentation model. Simultaneously, it establishes a positional correspondence between the first and second modality images based on a scaling factor, thereby extracting a first substantial region from the first modality image and a second substantial region from the second modality image. The first and second modality images are obtained from the regions to be measured using computed tomography (CT) and positron emission tomography (PET), respectively. Then, an image classification model is used to extract features, fuse modalities, and classify features from the first and second substantial regions to obtain a classification result. The method cascades and combines an image segmentation model and an image classification model, and uses a multimodal input three-dimensional convolutional neural network to classify the image. The input consists of image data from two different modalities. After feature extraction using the neural network, the two features are cascaded and fused to obtain a classification prediction result, thereby improving classification accuracy. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is an architectural diagram of the image classification method provided in the embodiments of this application;

[0041] Figure 2 A schematic flowchart illustrating the image classification method provided in this application embodiment;

[0042] Figure 3 This is a schematic diagram of the structure of the image segmentation model provided in the embodiments of this application;

[0043] Figure 4 A schematic diagram of the first physical mask provided in an embodiment of this application;

[0044] Figure 5 A schematic diagram of the first substantial region provided in the embodiments of this application;

[0045] Figure 6 This is a schematic diagram of the structure of the image classification model provided in the embodiments of this application. Detailed Implementation

[0046] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0047] Image classification refers to categorizing images into several classes based on the different features reflected in the image information. Image classification can be applied in various fields, such as medical imaging, artificial intelligence, deep learning, medical image analysis, and lung nodule diagnosis. For example, in the field of lung nodule diagnosis, lung nodule screening mainly relies on computed tomography (CT) or positron emission tomography (PET) to image the user's lungs, and then visually observing the lung images to identify lung nodules and classify them as benign or malignant. However, due to subjective factors, time constraints, and differences in interpretation, the analysis of these images has limitations, leading to a risk of misdiagnosis. Therefore, image classification methods can be used to detect and classify lung images, thereby assisting in the identification of lung nodule categories and improving screening efficiency and accuracy.

[0048] In some embodiments, deep learning techniques can be used for image classification processing. Deep learning can automatically learn and extract feature information from different images to perform image classification prediction. For example, feature information can be extracted from medical images to assist in disease identification. Deep learning methods include at least convolutional neural networks, recurrent neural networks, and generative adversarial networks. Neural networks, due to their unique convolution and pooling structures, can extract high-dimensional features that are difficult to obtain in feature extraction, improving subsequent detection speed and robustness.

[0049] In some embodiments, to improve the accuracy of image classification, the substantive content in the image can be extracted for analysis. This substantive content segmentation can be achieved using a semantic segmentation algorithm (U-Net). Semantic segmentation is a segmentation method based on a convolutional neural network, whose structure includes an encoder and a decoder, enabling fast and accurate segmentation of the substantive content of an image. For example, in CT images of the lungs obtained from computed tomography (CT), the lung parenchyma region can be extracted for subsequent analysis. CT imaging data is essentially a three-dimensional scan slice of the human anatomy. By inputting a set of slices as a whole into the semantic segmentation neural network, the neural network can capture richer spatial information and guide the accurate segmentation of the lung parenchyma, thereby segmenting the lung parenchyma region in the CT image.

[0050] In some embodiments, to improve classification accuracy, image classification can also be performed based on a feature-level fusion strategy. Input images of different sizes and modalities are processed using a neural network to extract their corresponding features, which are then fused according to the fusion strategy to obtain the image classification result. The methods described above all utilize feature information from a single image for classification, failing to incorporate the state of the actual content within the image for classification prediction, thus reducing classification accuracy. For example, in the field of lung nodule diagnosis, screening for benign or malignant lung nodules relies solely on feature information from a single CT scan, ignoring the state of surrounding tissues or organs, thereby reducing detection and classification accuracy.

[0051] To improve classification accuracy, some embodiments of this application provide an image classification method based on a multimodal three-dimensional convolutional neural network. This method can perform three-dimensional imaging of the region to be tested to achieve detection and classification. For example, in the field of early clinical imaging diagnosis of lung nodules, three-dimensional imaging of lung nodules can be used to classify lung nodules. By extracting features from three-dimensional images of different modalities and performing image classification processing based on feature-level fusion strategies, input images of different sizes and modalities are processed by using neural networks to extract their corresponding features. The features are then fused according to the fusion strategy and classified according to the features to obtain the image classification result.

[0052] like Figure 1 The diagram shown illustrates the architecture of the image classification method provided in this application. The classification method includes two stages: an image segmentation stage and an image classification stage. First, a substantial region is extracted from the image using an image segmentation model. Then, the extracted substantial region is used as input to an image classification model to obtain the classification result. By cascading the image segmentation model and the image classification model and employing a multimodal input 3D convolutional neural network to achieve image classification, the classification accuracy is improved. For example... Figure 2 The diagram shown is a flowchart of an image classification method provided in an embodiment of this application. The method includes the following steps:

[0053] S100: Acquire the first modality image and the second modality image.

[0054] The first modality image is an image obtained by imaging the region under test using computed tomography (CT) technology, and the second modality image is an image obtained by imaging the region under test using positron emission tomography (PET) technology. For example, in the field of lung nodule diagnosis, the lung under test can be imaged using either computed tomography or positron emission tomography to obtain CT imaging and PET imaging, i.e., the first modality image and the second modality image.

[0055] S200: Input the first modality image into the image segmentation model to obtain the first physical mask output by the image segmentation model.

[0056] After acquiring the first modality image and the second modality image, image segmentation can be performed on the first modality image and the second modality image to extract the substantial region, that is, to remove the interference regions in the image, such as air, chest wall tissue, and instruments and equipment. In the process of image segmentation, the first modality image can be input into a pre-trained image segmentation model to obtain a first substantial mask, wherein the image segmentation model is a neural network model trained based on the first sample image dataset.

[0057] In some embodiments, when training an image segmentation model, a first sample image dataset can be obtained, wherein the first sample image dataset includes multiple first sample images labeled with different substance masks, and a first neural network model is constructed and trained based on the first sample image dataset to obtain an image segmentation model.

[0058] To reduce the impact of noise and artifacts in images on model training and prediction, after obtaining the first sample image set, data preprocessing can be performed on the first sample images. Data preprocessing includes image resampling and window width and level adjustment. Since different imaging devices differ in spatial resolution, pixel size, slice thickness, etc., image resampling can be performed on the first sample images according to preset image specifications, and the window width and level of the first sample images can be adjusted according to preset window width and level values ​​to eliminate differences between images. For example, for ease of observation, the preset window width and level values ​​can be [-400, 1500].

[0059] In some embodiments, the first neural network model can be a three-dimensional semantic image segmentation model (3D U-Net). The 3D semantic image segmentation model is trained using a first set of preprocessed sample images to obtain an image segmentation model. The image segmentation model can perform decoding and encoding operations on the preprocessed first modality image to obtain a first substantial mask, thereby segmenting substantial regions according to the first substantial mask. The image segmentation model includes multiple encoders and multiple decoders. The encoders include convolutional layers and pooling layers, and the decoders include deconvolutional layers and skip connection layers. The skip connection layers are used to fuse feature maps in the encoder with positional correspondences in the decoder. The feature maps are obtained by performing convolution and pooling operations on the first modality image, and the feature maps in the decoder are obtained by performing deconvolution and skip connection operations on the feature maps in the encoder. The positional correspondence represents the relationship that the feature maps in the encoder and the feature maps in the decoder are of the same size.

[0060] For example, such as Figure 3 The diagram shows the structure of an image segmentation model. This model includes four encoders and four decoders. Each encoder consists of a 3×3×3 convolutional structure and a 2×2×2 max-pooling structure. The network has a central 3×3×3 convolutional structure. Each decoder contains a deconvolutional layer and a skip connection layer. The skip connection layer fuses the corresponding feature maps from the encoder and decoder, preventing information loss. During model training, the preprocessed first sample image data is fed into the encoder, progressively downsampling to obtain feature maps of different sizes. Simultaneously, the convolutional layers in the encoder reduce the dimensionality of the feature maps, extracting higher-level features, and outputting them to the next encoder via pooling layers. During this process, the data size is halved. Figure 3 Each feature map is labeled with its size; for example, "16ch.10×256×256" indicates 16 channels with dimensions of 10×256×256. The decoder uses deconvolution layers to perform deconvolution on the feature maps, restoring them to their original data size. At each layer of the decoder, skip connection layers fuse the corresponding feature maps from the encoder and decoder, thereby training the network model to obtain the image segmentation model. Taking a CT image of the lungs as an example, inputting the first modality image into the image segmentation model yields the first solid mask, as shown below. Figure 4 The first modality image shown is a solid mask of the lungs, which includes information on the shape and location of the lungs.

[0061] S300: Scaling the first physical mask based on the scaling factor to obtain the second physical mask.

[0062] The scaling factor represents the ratio of the actual imaging of the test region between the first modal image and the second modal image. Since the first and second modal images may have different sizes, the scaling factor can be used to establish a positional correspondence between them, thus obtaining the second substantial mask. When scaling the first substantial mask, a coordinate system can be constructed, and the first and second modal images can be registered according to this system. This means aligning the data of the first and second modal images within the same coordinate system to obtain the correspondence between the actual imaging of the test region in the first and second modal images under the same coordinate system. The scaling factor is then calculated based on this correspondence. For example, if the first modal image is 1.33 times the size of the second modal image, the first substantial mask can be scaled down by a factor of 1.33 to obtain the second substantial mask.

[0063] S400: Extract a first real region from a first real region image according to a first real region mask, and extract a second real region from a second real region image according to a second real region mask.

[0064] After obtaining the first and second substance masks, the first modal image can be segmented according to the first substance mask to obtain a first substance region containing only image substance. Similarly, the second modal image can be segmented according to the second substance mask to obtain a second substance region containing only image substance. For example, using... Figure 4 Taking the first substantial mask as an example, the first modal image is segmented according to the first substantial mask to obtain the following: Figure 5 The diagram shows only the parenchymal area of ​​the lungs.

[0065] It is understandable that a first substantial region containing only the image substance is extracted from the first modal image using a first substantial mask, and a second substantial region containing only the image substance is extracted from the second modal image using a second substantial mask. The first substantial region and the second substantial region are image data corresponding to the substantial parts in the first modal image and the second modal image, respectively. By combining image data from different modalities, the features of the substance in the image can be identified more accurately, thereby improving classification accuracy.

[0066] S600: Input the first and second physical regions into the image classification model to obtain the classification result output by the image classification model.

[0067] The image classification model is a neural network model trained using a first sample substantial region dataset and a second sample substantial region dataset. After obtaining the first and second substantial regions, they can be input into the image classification model to perform feature extraction, modality fusion, and feature classification on the first and second substantial regions, thereby obtaining the classification result.

[0068] The classification result is the classification corresponding to the first modality image and the second modality image. The classification result can include a first result and a second result. For example, in the field of lung nodule diagnosis, the first modality image and the second modality image are CT images and PET images obtained by scanning the lung to be tested, respectively. The corresponding first classification result can indicate that the lung nodule is benign, and the second classification result can indicate that the lung nodule is malignant.

[0069] In some embodiments, when training an image classification model, a sample substance dataset can be obtained, wherein the sample substance dataset includes multiple first sample substance regions and second sample substance regions labeled with different classification result labels. A second neural network model is then constructed and trained based on the sample substance dataset to obtain the image classification model.

[0070] In some embodiments, the second neural network model can be a multimodal 3D convolutional neural network (MM-3D-CNN). The MM-3D-CNN is trained using a sample dataset to obtain an image classification model. This model performs feature extraction, modality fusion, and feature classification on the first and second substantial regions to obtain a classification result. The image classification model includes a feature extraction module, a feature fusion module, and a feature classification module. The feature extraction module extracts a first substantial feature map from the first substantial region and a second substantial feature map from the second substantial region. The feature fusion module fuses the first and second substantial feature maps to obtain fused features. The feature classification module maps the fused features to the classification result.

[0071] For example, such as Figure 6 The diagram shows the structure of an image classification model. This model consists of seven network layers. Layers one through five each include convolutional and pooling layers, layer six includes a connection function, and layer seven includes multiple fully connected layers. Figure 6 As shown, the feature extraction module can receive input from two modalities: a first substantial region and a second substantial region. For each modal data, it performs multiple convolutional and pooling operations to extract deep abstract features, namely the first substantial feature map and the second substantial feature map. The feature extractors can share weights. This feature extraction module includes five network layers. The first network layer consists of a 1×3×3 convolutional structure with a stride of 1 and a 2×2×2 max-pooling structure with a stride of 1. Each of the second to fifth network layers consists of a 3×3×3 convolutional structure with a stride of 2 and a 2×2×2 max-pooling structure with a stride of 2. The feature extraction module obtains a feature map with 128 channels and dimensions of 1×32×32.

[0072] The feature fusion module can fuse features from different modalities using a cascaded approach after feature extraction, thereby enhancing the representational power of the features while reducing computational complexity. The feature fusion module includes a sixth network layer, which is composed of a Concat function. The input to this module is the output of the feature extraction module, i.e. Figure 6 The 128 channels of the two parallel paths shown have feature maps with dimensions of 1×32×32, namely the first feature map and the second feature map. A flattening operation is performed on the first and second feature maps to convert the multidimensional data into one dimension, and the output is a one-dimensional feature vector with a length of 524288, namely the fused feature.

[0073] The feature classification module receives the multimodal features obtained by the feature fusion module after feature fusion is completed; these are the fused features. The feature classification module includes a seventh network layer, which consists of four fully connected layers (FC). Each FC layer has 262144, 65536, 1024, and 2 neurons respectively. The final FC layer uses an activation function (e.g., the Sottmax function) to map the fused feature representation to the desired classification result.

[0074] In this embodiment, deep learning and multimodal fusion techniques are used to achieve image segmentation and image classification tasks, thereby constructing a model based on Computer-Aided Diagnosis (CAD) technology. First, the image segmentation model extracts the image substance, obtaining a substance mask. Simultaneously, considering the size differences of the images, a scaling factor is used to achieve the positional correspondence between the first and second modal images, thus realizing the substance extraction of the first and second modal images. Then, the image classification model performs multimodal fusion analysis on the first and second substance regions to obtain classification results, improving classification accuracy.

[0075] Based on the above image classification method, some embodiments of this application also provide an image classification system based on a multimodal three-dimensional convolutional neural network, including:

[0076] The image acquisition module is used to acquire a first modal image and a second modal image. The first modal image is an image obtained based on computed tomography of the region to be tested, and the second modal image is an image obtained based on positron emission tomography of the region to be tested.

[0077] The segmentation module is configured to input the first modality image into an image segmentation model to obtain a first substance mask output by the image segmentation model, wherein the image segmentation model is a neural network model trained based on a first sample image; and to scale the first substance mask based on a scaling factor to obtain a second substance mask, wherein the scaling factor is the image ratio between the second modality image and the first modality image; and to extract a first substance region in the first modality image according to the first substance mask; and to extract a second substance region in the second modality image according to the second substance mask.

[0078] The classification module is used to input the first physical region and the first physical region into a trained image classification model to obtain the classification result output by the image classification model. The classification result includes a first result and a second result. The image classification model is a neural network model trained based on the first sample physical region and the first sample physical region.

[0079] As can be seen from the above technical solutions, this application provides an image classification method and system based on a multimodal three-dimensional convolutional neural network. The method, after acquiring a first modality image and a second modality image, extracts a substantial mask using an image segmentation model. Simultaneously, it establishes a positional correspondence between the first and second modality images based on a scaling factor, thereby extracting a first substantial region from the first modality image and a second substantial region from the second modality image. The first and second modality images are obtained from the regions to be measured using computed tomography (CT) and positron emission tomography (PET), respectively. Then, an image classification model is used to extract features, fuse modalities, and classify the first and second substantial regions to obtain a classification result. This method cascades and combines an image segmentation model and an image classification model, and uses a multimodal input three-dimensional convolutional neural network to classify the image. The input consists of image data from two different modalities. After feature extraction using the neural network, the two features are cascaded and fused to obtain a classification prediction result, thus improving classification accuracy.

[0080] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. An image classification method based on a multimodal three-dimensional convolutional neural network, characterized in that, include: Acquire a first modal image and a second modal image, wherein the first modal image is an image obtained by imaging the region under test based on computed tomography (CT) technology, and the second modal image is an image obtained by imaging the region under test based on positron emission tomography (PET) technology. The first modality image is input into the image segmentation model to obtain the first substance mask output by the image segmentation model, wherein the image segmentation model is a neural network model trained based on the first sample image dataset; The first physical mask is scaled based on a scaling factor to obtain a second physical mask, wherein the scaling factor is the ratio of the actual imaging of the region to be tested between the first modal image and the second modal image; Extract the first real region from the first modal image according to the first real mask; Extract the second real region from the second modal image according to the second real mask; The first and second physical regions are input into an image classification model to obtain the classification result output by the image classification model. The classification result includes a first result and a second result. The image classification model is a neural network model trained based on the first sample physical region dataset and the second sample physical region dataset.

2. The classification method according to claim 1, characterized in that, The method further includes: Obtain a first sample image dataset, which includes multiple first sample images labeled with different substance masks; Construct the first neural network model; The first neural network model is trained based on the first sample image dataset to obtain an image segmentation model.

3. The classification method according to claim 2, characterized in that, After obtaining the first sample image set, the process also includes: The first sample image is resampled according to the preset image specifications; Adjust the window width and window level of the first sample image according to the preset window width and window level values.

4. The classification method according to claim 2, characterized in that, The image segmentation model includes multiple encoders and multiple decoders. The encoders include convolutional layers and pooling layers, and the decoders include deconvolutional layers and skip connection layers. The skip connection layers are used to fuse feature maps in the encoders with positional correspondences in the decoders. The feature maps in the encoders are obtained by performing convolution and pooling operations on the first modality image, and the feature maps in the decoders are obtained by performing deconvolution and skip connection operations on the feature maps in the encoders. The positional correspondence is used to characterize the relationship that the feature maps in the encoders and the feature maps in the decoders are of the same size.

5. The classification method according to claim 1, characterized in that, The step of scaling the second modality image and the first physical region based on a scaling factor further includes: Construct a coordinate system; The first modal image and the second modal image are registered according to the coordinate system to obtain the actual imaging correspondence between the first modal image and the second modal image of the test area under the same coordinate system; The scaling factor is calculated based on the correspondence.

6. The classification method according to claim 2, characterized in that, The method further includes: Obtain the sample essence dataset, which includes multiple first sample essence regions and second sample essence regions labeled with different classification result labels; Construct a second neural network model; The second neural network model is trained based on the actual dataset of the samples to obtain an image classification model.

7. The classification method according to claim 1, characterized in that, The image classification model includes a feature extraction module, a feature fusion module, and a feature classification module; the method further includes: The feature extraction module extracts a first substantial feature map in the first substantial region and a second substantial feature map in the second substantial region. The first substantial feature map and the second substantial feature map are fused by the feature fusion module to obtain fused features; The fused features are mapped to classification results through the feature classification module.

8. The classification method according to claim 1, characterized in that, The image classification model comprises seven network layers. The first to fifth network layers each include convolutional layers and pooling layers, the sixth network layer includes a connection function, and the seventh network layer includes multiple fully connected layers.

9. The classification method according to claim 6, characterized in that, The first neural network model is a three-dimensional semantic image segmentation model, and the second neural network model is a multimodal three-dimensional convolutional neural network.

10. An image classification system based on a multimodal three-dimensional convolutional neural network, characterized in that, The system includes: The image acquisition module is used to acquire a first modal image and a second modal image. The first modal image is an image obtained by imaging the region under test based on computed tomography (CT) technology, and the second modal image is an image obtained by imaging the region under test based on positron emission tomography (PET) technology. The segmentation module is configured to input the first modality image into an image segmentation model to obtain a first substance mask output by the image segmentation model, wherein the image segmentation model is a neural network model trained based on a first sample image dataset; and to scale the first substance mask based on a scaling factor to obtain a second substance mask, wherein the scaling factor is the ratio of the substance image of the region to be tested between the second modality image and the first modality image; and to extract a first substance region in the first modality image according to the first substance mask; and to extract a second substance region in the second modality image according to the second substance mask. The classification module is used to input the first physical region and the first physical region into a trained image classification model to obtain the classification result output by the image classification model. The classification result includes a first result and a second result. The image classification model is a neural network model trained based on the first sample physical region dataset and the first sample physical region dataset.

Images