Image omics feature extraction optimization method and system based on deep learning

Abstract

The invention relates to the field of computer vision, and discloses an image omics feature extraction optimization method and system based on deep learning, and the method comprises the steps: constructing a cumulative distribution function of a multi-modal image, carrying out the matching of the multi-modal image, and obtaining a matched multi-modal image; establishing a multi-modal segmentation model matched with the multi-modal image, and performing ROI segmentation on the matched multi-modal image to obtain a target ROI image; slicing the target ROI image to obtain a sliced ROI image, extracting an image feature map of the sliced ROI image by using a convolutional layer corresponding to the integrated 3D-CNN model, calculating an attention weight of the image feature map by using an attention mechanism in the integrated 3D-CNN model, and extracting image features of the image feature map by using a full connection layer corresponding to the integrated 3D-CNN model; and carrying out feature fusion on the image features to obtain target image features. According to the invention, the feature extraction effect of radiomics can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and system for optimizing image omics feature extraction based on deep learning. Background Art

[0002] Radiomics refers to the use of computer algorithms to automatically extract a large number of quantitative features from medical images (such as CT, MRI, PET, etc.). These features can be simple statistics (such as mean, standard deviation) or complex texture features (such as entropy, correlation) and shape features (such as volume, surface area). Radiomics has broad application prospects in clinical diagnosis and treatment.

[0003] Currently, radiomics feature extraction is mainly achieved by acquiring medical imaging data and extracting features through deep learning models. The deep neural network of this method model usually contains a large number of layers and parameters. Each layer performs nonlinear transformation on the input data. These transformations are combined to form a very complex decision boundary, making it difficult to explain the feature representation within the model in a simple and intuitive way, resulting in poor radiomics feature extraction results. Summary of the invention

[0004] The present invention provides a method and system for optimizing feature extraction of radiomics based on deep learning, the main purpose of which is to improve the feature extraction effect of radiomics.

[0005] To achieve the above objectives, the present invention provides a method for optimizing image omics feature extraction based on deep learning, comprising:

[0006] Acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image;

[0007] Establishing a multimodal segmentation model for the matching multimodal image, and performing ROI segmentation on the matching multimodal image by using the multimodal segmentation model to obtain a target ROI image;

[0008] Constructing a 3D-CNN model of the target ROI image, determining an attention mechanism of the 3D-CNN model, and integrating the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model;

[0009] Slicing the target ROI image to obtain a sliced ​​ROI image, extracting an image feature map of the sliced ​​ROI image using a convolutional layer corresponding to the integrated 3D-CNN model, calculating an attention weight of the image feature map using an attention mechanism in the integrated 3D-CNN model, and extracting image features of the image feature map using a fully connected layer corresponding to the integrated 3D-CNN model based on the attention weight;

[0010] The image features are subjected to feature fusion to obtain target image features.

[0011] Optionally, constructing the cumulative distribution function of the multimodal image includes:

[0012] Preprocessing the multimodal image to obtain a preprocessed multimodal image;

[0013] Calculating a modality histogram of the preprocessed multimodal image;

[0014] trimming the extreme values ​​in the modal histogram to obtain a target modal histogram;

[0015] The cumulative distribution function of the target modal histogram is constructed using the following formula;

[0016] Among them, the cumulative distribution function is:

[0017] A(i)=∑D(j), where j

[0018] Among them, A(i) represents the probability that the pixel value is less than or equal to i, i represents a specific value of the target modality histogram corresponding to the pixel value, D(j) represents the target modality histogram, and j represents the index of the pixel value corresponding to the accumulated target modality histogram when calculating the cumulative distribution function.

[0019] Optionally, matching the multimodal images based on the cumulative distribution function to obtain a matched multimodal image includes:

[0020] Identifying an image state of the multimodal image corresponding to the preprocessed multimodal image;

[0021] Based on the image state, dividing the preprocessed multimodal image into a reference image and a target image;

[0022] Based on the cumulative distribution function, calculating the KL divergence of the reference image and the target image;

[0023] Constructing a matching strategy between the reference image and the target image through the KL divergence;

[0024] Based on the matching strategy, the reference image and the target image are matched to obtain the matched multimodal image.​

[0025] Optionally, the calculating the KL divergence of the reference image and the target image based on the cumulative distribution function includes:

[0026] Based on the cumulative distribution function, marking a reference image distribution function and a target image distribution function of the reference image and the target image, respectively;

[0027] The KL divergence of the reference image distribution function and the target image distribution function is calculated using the following formula:

[0028] S(F||K)=∑F(i)*log(F(i) / K(i))

[0029] Wherein, S(F||K) represents the KL divergence of the reference image distribution function and the target image distribution function, F(i) represents the target image distribution function of the target image, K(i) represents the reference image distribution function, and log(F(i) / K(i) represents the natural logarithm function.

[0030] Optionally, the establishing of the multimodal segmentation model for matching the multimodal image includes:

[0031] Obtaining the matching multimodal image corresponding model training image;

[0032] Performing ROI marking on the model training image to obtain a ROI marked training image;

[0033] Performing modality clustering on the ROI-marked training image to obtain a modality clustered image;

[0034] Constructing an initial modality segmentation model of the modality type corresponding to the modality clustering image;

[0035] defining a segmentation task of the modality type;

[0036] Based on the segmentation task, the initial modality segmentation model is collaboratively trained using the modality clustering images to obtain the multimodal segmentation model.

[0037] Optionally, based on the segmentation task, the initial modality segmentation model is collaboratively trained using the modality clustering image to obtain the multimodal segmentation model, including:

[0038] Identifying image modality features of the modality clustering images;

[0039] Analyzing the modality correlation coefficient of the modality clustering image based on the image modality characteristics;

[0040] Based on the modal association coefficient, construct a multi-task loss function of the initial modal segmentation model;

[0041] Co-training the initial modality segmentation model based on the multi-task loss function to obtain a training modality segmentation model;

[0042] Calculating the Dice coefficient of the training modality segmentation model;

[0043] When the Dice coefficient meets a preset Dice coefficient threshold, the training modality segmentation model is used as the multimodal segmentation model.

[0044] Optionally, extracting the image feature map of the slice ROI image by using the convolution layer corresponding to the integrated 3D-CNN model includes:

[0045] Standardizing the slice ROI image to obtain a standard ROI image;

[0046] Initializing the convolution kernel of the convolution layer to obtain an initialized convolution layer;

[0047] Calculating the dot product of the initialized convolutional layer and the standard ROI image;

[0048] Constructing an initial feature map of the standard ROI image through the dot product;

[0049] A preset activation function is applied to the initial feature map to obtain the image feature map.

[0050] Optionally, the attention mechanism in the integrated 3D-CNN model calculates the attention weight of the image feature map, including:

[0051] Input the image feature map into the image storage module of the attention mechanism, and calculate the channel attention and spatial attention of the image feature through the attention calculation module of the attention mechanism;

[0052] Based on the channel attention and the spatial attention, the attention weight of the image feature map is calculated using the following formula:

[0053]

[0054] Among them, τ represents the attention weight of the image feature map, G r represents the channel attention matrix, G c represents the spatial attention matrix, Softmax(G r ) represents channel attention, Softmax(G c ) represents spatial attention, Softmax represents a function that normalizes the vector to a probability distribution, represents the attention weighted operation.

[0055] Optionally, extracting image features of the image feature map based on the attention weight using a corresponding fully connected layer of the integrated 3D-CNN model includes:

[0056] Multiplying the attention weight and the image feature map element by element to obtain a weighted feature map;

[0057] Flattening the weighted feature map into a one-dimensional vector using the fully connected layer;

[0058] Multiplying the one-dimensional vector by a preset weight matrix to obtain a vector multiplication result;

[0059] Based on the vector multiplication result and a preset bias vector, the image features of the image feature map are output.

[0060] In order to solve the above problems, the present invention also provides an imaging omics feature extraction optimization system based on deep learning, the system comprising:

[0061] A multimodal image matching module, used to acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image;

[0062] An image ROI extraction module is used to establish a multimodal segmentation model for the matching multimodal image, and perform ROI segmentation on the matching multimodal image through the multimodal segmentation model to obtain a target ROI image;

[0063] A 3D-CNN model construction module, used to construct a 3D-CNN model of the target ROI image, determine an attention mechanism of the 3D-CNN model, and integrate the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model;

[0064] An image feature extraction module is used to slice the target ROI image to obtain a sliced ​​ROI image, extract an image feature map of the sliced ​​ROI image using the convolutional layer corresponding to the integrated 3D-CNN model, calculate the attention weight of the image feature map using the attention mechanism in the integrated 3D-CNN model, and extract image features of the image feature map based on the attention weight using the fully connected layer corresponding to the integrated 3D-CNN model;

[0065] The image feature fusion module is used to perform feature fusion on the image features to obtain target image features.

[0066] The present invention matches the multimodal images based on the cumulative distribution function to obtain matched multimodal images to achieve consistency in brightness distribution of the multimodal images; optionally, the present invention can utilize complementary information between different modalities by establishing a multimodal segmentation model for the matched multimodal images to improve the accuracy and reliability of multimodal image segmentation; the present invention integrates the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model that can simultaneously utilize the advantages of 3D convolution operations and attention mechanisms, thereby improving the effect of feature extraction; further, determining the attention mechanism of the 3D-CNN model can be used to guide the model to automatically focus on the most distinguishable image area of ​​the ROI image, It can help the model to more effectively capture the key features related to the task and improve the interpretability of the features. The present invention can improve the reliability of image analysis by slicing the target ROI image to obtain a sliced ​​ROI image. The present invention uses the attention mechanism in the integrated 3D-CNN model to calculate the attention weight of the image feature map to reflect the key areas in the feature map, thereby improving the feature extraction effect. Finally, based on the attention weight, the present invention uses the corresponding fully connected layer of the integrated 3D-CNN model to extract the image features of the image feature map to effectively extract the image features of the image feature map and obtain the target image features by feature fusion of the image features, thereby improving the richness of the features and thus improving the feature quality. Therefore, the imaging genomics feature extraction optimization method and system based on deep learning proposed in the present invention can improve the feature extraction effect of imaging genomics. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of a flow chart of a method for optimizing image omics feature extraction based on deep learning provided by an embodiment of the present invention;

[0068] Figure 2 A functional module diagram of an imaging omics feature extraction optimization system based on deep learning provided by an embodiment of the present invention;

[0069] Figure 3 A schematic diagram of the structure of an electronic device of an imaging omics feature extraction optimization system based on deep learning provided by an embodiment of the present invention;

[0070] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0071] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0072] The embodiment of the present application provides an imaging genomics feature extraction optimization method based on deep learning. The execution subject of the imaging genomics feature extraction optimization method based on deep learning includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided in the embodiment of the present application. In other words, the imaging genomics feature extraction optimization method based on deep learning can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0073] Reference Figure 1 FIG. 1 is a flow chart of an image omics feature extraction optimization method based on deep learning provided by an embodiment of the present invention. In this embodiment, the image omics feature extraction optimization method based on deep learning includes:

[0074] S1. Acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image.

[0075] It should be explained that the multimodal image refers to image data obtained from different imaging technologies. For example, a multimodal image may include a cross-sectional image of a CT scan, a T1-weighted image of an MRI, and a 3D volume data of a PET. The cumulative distribution function refers to an extension of a histogram, which represents the probability that a pixel value in an image is less than or equal to a certain value.

[0076] The present invention realizes consistency processing of multimodal image data by constructing a cumulative distribution function of the multimodal image.

[0077] In detail, constructing the cumulative distribution function of the multimodal image includes:

[0078] Preprocessing the multimodal image to obtain a preprocessed multimodal image;

[0079] Calculating a modality histogram of the preprocessed multimodal image;

[0080] trimming the extreme values ​​in the modal histogram to obtain a target modal histogram;

[0081] The cumulative distribution function of the target modal histogram is constructed using the following formula;

[0082] Among them, the cumulative distribution function is:

[0083] A(i)=∑D(j), where j

[0084] Among them, A(i) represents the probability that the pixel value is less than or equal to i, i represents a specific value of the target modality histogram corresponding to the pixel value, D(j) represents the target modality histogram, and j represents the index of the pixel value corresponding to the accumulated target modality histogram when calculating the cumulative distribution function.

[0085] Among them, the pre-processed multimodal image refers to the image after the multimodal image is subjected to steps such as adjusting the pixel value range, removing noise, and image enhancement. The modal histogram refers to a statistical method for describing the pixel distribution of the pre-processed multimodal image. The target modal histogram refers to the histogram after removing the extreme pixel values ​​that affect the pixel distribution in the target modal histogram.

[0086] Furthermore, the calculation of the modal histogram of the preprocessed multimodal image can be implemented by tools such as OpenCV, SimpleITK, and ITK.

[0087] The present invention matches the multimodal images based on the cumulative distribution function to obtain a matched multimodal image to achieve consistency in brightness distribution of the multimodal image. The matched multimodal image refers to an image after the multimodal image is adjusted according to an ideal image state.

[0088] In detail, matching the multimodal images based on the cumulative distribution function to obtain a matched multimodal image includes:

[0089] Identifying an image state of the multimodal image corresponding to the preprocessed multimodal image;

[0090] Based on the image state, dividing the preprocessed multimodal image into a reference image and a target image;

[0091] Based on the cumulative distribution function, calculating the KL divergence of the reference image and the target image;

[0092] Constructing a matching strategy between the reference image and the target image through the KL divergence;

[0093] Based on the matching strategy, the reference image and the target image are matched to obtain the matched multimodal image.

[0094] ​Among them, the image state refers to the specific condition or state of the image data, such as brightness, contrast, noise level, etc., the reference image refers to an image in an ideal state, the target image refers to an image that needs to be adjusted to match the reference image state, the KL divergence refers to the difference between the reference image and the target image, and the matching strategy may include adjusting the brightness, contrast, color balance, etc. of the target image to reduce the KL divergence.

[0095] Further, the calculating the KL divergence of the reference image and the target image based on the cumulative distribution function includes:

[0096] Based on the cumulative distribution function, marking a reference image distribution function and a target image distribution function of the reference image and the target image, respectively;

[0097] The KL divergence of the reference image distribution function and the target image distribution function is calculated using the following formula:

[0098] S(F||K)=∑F(i)*log(F(i) / K(i))

[0099] Wherein, S(F||K) represents the KL divergence of the reference image distribution function and the target image distribution function, F(i) represents the target image distribution function of the target image, K(i) represents the reference image distribution function,

[0100] log(F(i) / K(i)) represents the natural logarithm function.

[0101] The reference image distribution function refers to the probability distribution of pixel values ​​in the reference image, and the target image distribution function refers to the probability distribution of pixel values ​​in the target image.

[0102] S2. Establish a multimodal segmentation model for the matching multimodal image, and perform ROI segmentation on the matching multimodal image using the multimodal segmentation model to obtain a target ROI image.

[0103] The present invention can utilize complementary information between different modalities to improve the accuracy and reliability of multimodal image segmentation by establishing the multimodal segmentation model that matches the multimodal images. The multimodal segmentation model refers to a model that can simultaneously process and analyze medical image data from multiple different imaging modalities.

[0104] In detail, the establishing of the multimodal segmentation model for matching the multimodal images includes:

[0105] Obtaining the matching multimodal image corresponding model training image;

[0106] Performing ROI marking on the model training image to obtain a ROI marked training image;

[0107] Performing modality clustering on the ROI-marked training image to obtain a modality clustered image;

[0108] Constructing an initial modality segmentation model of the modality type corresponding to the modality clustering image;

[0109] defining a segmentation task of the modality type;

[0110] Based on the segmentation task, the initial modality segmentation model is collaboratively trained using the modality clustering images to obtain the multimodal segmentation model.

[0111] Among them, the model training image refers to an image dataset used to train a multimodal segmentation model, and these images should contain images of multiple modalities. The ROI marked training image refers to an image on which a region of interest (ROI) is marked on the model training image. The modality clustering image refers to an image obtained by performing modality clustering analysis on the ROI marked training image. Modality clustering can help group similar modality data to facilitate subsequent segmentation model training. The initial modality segmentation model refers to a basic segmentation model of the corresponding modality type constructed according to the modality clustering results. This model can be a fully connected network or a convolutional neural network. The segmentation task refers to a segmentation task defined for each modality. For example, for the CT modality, the segmentation task may be to separate the tumor area from other tissue areas; for the MRI modality, the segmentation task may be to separate the brain area from the non-brain area. The segmentation task defines the features and decision boundaries that the model needs to learn.

[0112] Furthermore, based on the segmentation task, the initial modality segmentation model is collaboratively trained using the modality clustering images to obtain the multimodal segmentation model, including:

[0113] Identifying image modality features of the modality clustering images;

[0114] Analyzing the modality correlation coefficient of the modality clustering image based on the image modality characteristics;

[0115] Based on the modal association coefficient, construct a multi-task loss function of the initial modal segmentation model;

[0116] Co-training the initial modality segmentation model based on the multi-task loss function to obtain a training modality segmentation model;

[0117] Calculating the Dice coefficient of the training modality segmentation model;

[0118] When the Dice coefficient meets a preset Dice coefficient threshold, the training modality segmentation model is used as the multimodal segmentation model.

[0119] Among them, the image modality feature refers to the feature extracted from the clustered image of each modality, which can be features such as morphology and spatial relationship. The modality association coefficient refers to a quantitative value used to evaluate the correlation between different modalities. The multi-task loss function refers to a loss function used to simultaneously optimize the segmentation tasks of multiple modalities. The training modality segmentation model refers to a model obtained after collaborative training. The Dice coefficient refers to an indicator for evaluating the performance of a segmentation task, which measures the similarity between the segmentation results predicted by the model and the actual segmentation results. The higher the Dice coefficient, the higher the accuracy of the model segmentation. The Dice coefficient threshold refers to a preset Dice coefficient threshold, which is used to determine whether the performance of the training modality segmentation model meets the requirements. When the Dice coefficient exceeds the preset threshold, it can be considered that the model segmentation performance is good and can be used as a multimodal segmentation model.

[0120] Furthermore, the analysis of the modal correlation coefficient of the modal clustering image uses visualization technology (such as heat map, scatter plot, etc.) to show the correlation between modalities.

[0121] S3. Construct a 3D-CNN model of the target ROI image, determine the attention mechanism of the 3D-CNN model, integrate the attention mechanism into the 3D-CNN model, and obtain an integrated 3D-CNN model.

[0122] It should be explained that the 3D-CNN model refers to a model for extracting stereoscopic features of an image in a target ROI image. In the present invention, the 3D-CNN model is trained by a 3D U-Net architecture, which can process three-dimensional volume data, such as CT, MRI or PET scans. The attention mechanism refers to a mechanism for guiding the model to automatically focus on the most distinguishing image area of ​​the ROI image, which can help the model more effectively capture key features related to the task and improve the interpretability of the features. Such as the squeezing and excitation mechanism in SENet, the channel attention and spatial attention mechanism in CBAM, or the self-attention mechanism in Transformer.

[0123] The present invention integrates the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model that can simultaneously utilize the advantages of 3D convolution operation and attention mechanism, thereby improving the effect of feature extraction. The integrated 3D-CNN model refers to combining the attention mechanism with the 3D-CNN model to form a new model architecture.

[0124] In detail, the attention mechanism can be integrated after the convolution layer and before the pooling layer of the 3D-CNN model.

[0125] S4. Slice the target ROI image to obtain a sliced ​​ROI image, use the corresponding convolutional layer of the integrated 3D-CNN model to extract an image feature map of the sliced ​​ROI image, use the attention mechanism in the integrated 3D-CNN model to calculate the attention weight of the image feature map, and based on the attention weight, use the corresponding fully connected layer of the integrated 3D-CNN model to extract image features of the image feature map.

[0126] The present invention can improve the reliability of image analysis by slicing the target ROI image to obtain a sliced ​​ROI image. The sliced ​​ROI image refers to an image obtained by dividing the target ROI image into multiple slices. In detail, slicing the target ROI image can be implemented by ITK, SimpleITK, or 3D Slicer.

[0127] Optionally, the present invention uses the image feature map of the slice ROI image extracted by the corresponding convolution layer of the integrated 3D-CNN model to capture the key features in the image, which is helpful for subsequent analysis and processing. The image feature map refers to a map that reflects the features of the input image at different scales and different positions.

[0128] In detail, the extracting the image feature map of the slice ROI image by using the corresponding convolutional layer of the integrated 3D-CNN model includes:

[0129] Standardizing the slice ROI image to obtain a standard ROI image;

[0130] Initializing the convolution kernel of the convolution layer to obtain an initialized convolution layer;

[0131] Calculating the dot product of the initialized convolutional layer and the standard ROI image;

[0132] Constructing an initial feature map of the standard ROI image through the dot product;

[0133] A preset activation function is applied to the initial feature map to obtain the image feature map.

[0134] Among them, the convolution layer refers to a layer that uses a 3D convolution kernel to slide on the input data to perform a convolution operation, the standard ROI image refers to the ROI image after standardization, and the standardization process usually includes scaling the pixel values ​​of the image to a specific range, such as between 0 and 1, to ensure that all images are on the same scale. The initialized convolution layer refers to setting the initial convolution kernel weights for the convolution layer, and these weights are usually initialized randomly. The dot product refers to the product of each convolution kernel and the local area in the image. The initial feature map refers to a feature map constructed based on the result of the dot product calculation. The activation function is used for the initial feature map to increase the nonlinear expression ability of the network. The activation function includes ReLU, Leaky ReLU, Sigmoid, Tanh and other functions.

[0135] Furthermore, the calculating the dot product of the initialized convolution layer and the standard ROI image means that each convolution kernel is calculated by performing a dot product with each pixel in the standard ROI image.

[0136] The present invention uses the attention mechanism in the integrated 3D-CNN model to calculate the attention weight of the image feature map to reflect the key areas in the feature map, thereby improving the feature extraction effect. The attention weight refers to a value that reflects the importance of each feature map at different positions or channels.

[0137] In detail, the attention mechanism in the integrated 3D-CNN model calculates the attention weight of the image feature map, including:

[0138] Input the image feature map into the image storage module of the attention mechanism, and calculate the channel attention and spatial attention of the image feature through the attention calculation module of the attention mechanism;

[0139] Based on the channel attention and the spatial attention, the attention weight of the image feature map is calculated using the following formula:

[0140]

[0141] Among them, τ represents the attention weight of the image feature map, G r represents the channel attention matrix, G c represents the spatial attention matrix, Softmax(G r ) represents channel attention, Softmax(G c ) represents spatial attention, Softmax represents a function that normalizes the vector to a probability distribution, represents the attention weighted operation.

[0142] Among them, the image storage module refers to the module in the attention mechanism, which is responsible for receiving the feature map from the convolutional layer, the attention calculation module refers to the module responsible for calculating the channel attention and spatial attention of the feature map, the channel attention refers to the importance of different channels, the spatial attention refers to the importance of different positions, the channel attention matrix refers to the attention weight of each channel in the feature map, and the spatial attention matrix refers to the attention weight of each position in the feature map.

[0143] Optionally, the present invention effectively extracts the image features of the image feature map by extracting the image features of the image feature map using the corresponding fully connected layer of the integrated 3D-CNN model based on the attention weight. The fully connected layer refers to a layer for extracting image features, and the image features refer to representative abstract features extracted from the image that can be used for further analysis and classification. These features can be features such as pixel intensity, texture, shape, spatial relationship, etc.

[0144] In detail, based on the attention weight, extracting the image features of the image feature map using the corresponding fully connected layer of the integrated 3D-CNN model includes:

[0145] Multiplying the attention weight and the image feature map element by element to obtain a weighted feature map;

[0146] Flattening the weighted feature map into a one-dimensional vector using the fully connected layer;

[0147] Multiplying the one-dimensional vector by a preset weight matrix to obtain a vector multiplication result;

[0148] Based on the vector multiplication result and a preset bias vector, the image features of the image feature map are output.

[0149] Among them, the weighted feature map refers to the feature map obtained by element-by-element multiplication of the attention weight and the original image feature map, the one-dimensional vector refers to all pixel values ​​in the weighted feature map flattened into a single one-dimensional vector, the weight matrix refers to the weight matrix used to map the one-dimensional vector to a higher dimension, the vector multiplication result refers to the result obtained by matrix multiplication of the one-dimensional vector and the weight matrix, and the vector multiplication result is a two-dimensional matrix, which contains the features extracted from the original feature map, and the bias vector refers to the vector used in conjunction with the weight matrix, and the size of the bias vector is the same as the output dimension of the weight matrix.

[0150] Furthermore, the flattening of the weighted feature map into a one-dimensional vector using the fully connected layer can expand all pixel values ​​in the feature map along a certain dimension (usually height and width dimensions). The image features of the output image feature map can be obtained by adding the vector multiplication result and a preset bias vector.

[0151] S5. Perform feature fusion on the image features to obtain target image features.

[0152] Finally, the present invention can improve the richness of features by fusing the image features to obtain target image features, thereby improving the feature quality. The target image feature refers to the feature fusion of the image features to generate a richer and more comprehensive feature. In detail, the combination method of feature fusion may include feature splicing, feature weighting, feature fusion module, etc. It should be noted that according to the scene requirements, the fused features are adjusted to generate the target image features, and the adjustment methods include the scale, direction, spatial position, etc. of the features.

[0153] The present invention matches the multimodal images based on the cumulative distribution function to obtain matched multimodal images to achieve consistency in brightness distribution of the multimodal images; optionally, the present invention can utilize complementary information between different modalities by establishing a multimodal segmentation model for the matched multimodal images to improve the accuracy and reliability of multimodal image segmentation; the present invention integrates the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model that can simultaneously utilize the advantages of 3D convolution operations and attention mechanisms, thereby improving the effect of feature extraction; further, determining the attention mechanism of the 3D-CNN model can be used to guide the model to automatically focus on the most distinguishable image area of ​​the ROI image, It can help the model to more effectively capture the key features related to the task and improve the interpretability of the features. The present invention can improve the reliability of image analysis by slicing the target ROI image to obtain a sliced ​​ROI image. The present invention uses the attention mechanism in the integrated 3D-CNN model to calculate the attention weight of the image feature map to reflect the key areas in the feature map, thereby improving the feature extraction effect. Finally, based on the attention weight, the present invention uses the corresponding fully connected layer of the integrated 3D-CNN model to extract the image features of the image feature map to effectively extract the image features of the image feature map and obtain the target image features by feature fusion of the image features, thereby improving the richness of the features and thus improving the feature quality. Therefore, the imaging genomics feature extraction optimization method based on deep learning proposed in the present invention can improve the feature extraction effect of imaging genomics.

[0154] like Figure 2, is a functional module diagram of an imaging omics feature extraction optimization system based on deep learning provided by an embodiment of the present invention.

[0155] The imaging genomics feature extraction and optimization system 200 based on deep learning of the present invention can be installed in an electronic device. According to the functions implemented, the imaging genomics feature extraction and optimization system 200 based on deep learning can include a multimodal image matching module 201, an image ROI extraction module 202, a 3D-CNN model construction module 203, an image feature extraction module 204 and an image feature fusion module 205. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0156] In this embodiment, the functions of each module / unit are as follows:

[0157] The multimodal image matching module 201 is used to acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image;

[0158] The image ROI extraction module 202 is used to establish a multimodal segmentation model for the matching multimodal image, and perform ROI segmentation on the matching multimodal image through the multimodal segmentation model to obtain a target ROI image;

[0159] The 3D-CNN model construction module 203 is used to construct a 3D-CNN model of the target ROI image, determine the attention mechanism of the 3D-CNN model, and integrate the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model;

[0160] The image feature extraction module 204 is used to slice the target ROI image to obtain a sliced ​​ROI image, extract an image feature map of the sliced ​​ROI image using the convolutional layer corresponding to the integrated 3D-CNN model, calculate the attention weight of the image feature map using the attention mechanism in the integrated 3D-CNN model, and extract image features of the image feature map based on the attention weight using the fully connected layer corresponding to the integrated 3D-CNN model;

[0161] The image feature fusion module 205 is used to perform feature fusion on the image features to obtain target image features.

[0162] In detail, each module described in the deep learning-based imaging genomics feature extraction optimization system 200 described in the embodiment of the present invention adopts the same technical means as the deep learning-based imaging genomics feature extraction optimization method described in the accompanying drawings when used, and can produce the same technical effects, which will not be repeated here.

[0163] An embodiment of the present invention provides an electronic device for implementing an imaging omics feature extraction optimization method based on deep learning.

[0164] See also Figure 3 As shown, the electronic device may include a processor 30, a memory 31, a communication bus 32 and a communication interface 33, and may also include a computer program stored in the memory 31 and executable on the processor 30, such as an imaging genomics feature extraction optimization method program based on deep learning.

[0165] In some embodiments, the processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes programs or modules stored in the memory (for example, executing an imaging genomics feature extraction optimization program based on deep learning, etc.), and calls the data stored in the memory to execute various functions of the electronic device and process data.

[0166] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory may also include both an internal storage unit of the electronic device and an external storage device. The memory can be used not only to store application software and various types of data installed in the electronic device, such as the code of the imaging genomics feature extraction optimization program based on deep learning, but also to temporarily store data that has been output or is to be output.

[0167] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory and at least one processor, etc.

[0168] The communication interface is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0169] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor through a power management system, so that the power management system can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0170] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0171] The imaging omics feature extraction optimization program based on deep learning stored in the memory of the electronic device is a combination of multiple instructions. When running in the processor, it can achieve:

[0172] Acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image;

[0173] Establishing a multimodal segmentation model for the matching multimodal image, and performing ROI segmentation on the matching multimodal image by using the multimodal segmentation model to obtain a target ROI image;

[0174] Constructing a 3D-CNN model of the target ROI image, determining an attention mechanism of the 3D-CNN model, and integrating the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model;

[0175] Slicing the target ROI image to obtain a sliced ​​ROI image, extracting an image feature map of the sliced ​​ROI image using a convolutional layer corresponding to the integrated 3D-CNN model, calculating an attention weight of the image feature map using an attention mechanism in the integrated 3D-CNN model, and extracting image features of the image feature map using a fully connected layer corresponding to the integrated 3D-CNN model based on the attention weight;

[0176] The image features are subjected to feature fusion to obtain target image features.

[0177] Specifically, the specific implementation method of the processor for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0178] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0179] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0180] Acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image;

[0181] Establishing a multimodal segmentation model for the matching multimodal image, and performing ROI segmentation on the matching multimodal image by using the multimodal segmentation model to obtain a target ROI image;

[0182] Constructing a 3D-CNN model of the target ROI image, determining an attention mechanism of the 3D-CNN model, and integrating the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model;

[0183] Slicing the target ROI image to obtain a sliced ​​ROI image, extracting an image feature map of the sliced ​​ROI image using a convolutional layer corresponding to the integrated 3D-CNN model, calculating an attention weight of the image feature map using an attention mechanism in the integrated 3D-CNN model, and extracting image features of the image feature map using a fully connected layer corresponding to the integrated 3D-CNN model based on the attention weight;

[0184] The image features are subjected to feature fusion to obtain target image features.

[0185] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0186] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0188] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0189] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0190] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A radiomics feature extraction optimization method based on deep learning, characterized in that: The method comprises: Acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image; Establishing a multimodal segmentation model for the matching multimodal image, and performing ROI segmentation on the matching multimodal image by using the multimodal segmentation model to obtain a target ROI image; Constructing a 3D-CNN model of the target ROI image, determining an attention mechanism of the 3D-CNN model, and integrating the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model; Slicing the target ROI image to obtain a sliced ​​ROI image, extracting an image feature map of the sliced ​​ROI image using a convolutional layer corresponding to the integrated 3D-CNN model, calculating an attention weight of the image feature map using an attention mechanism in the integrated 3D-CNN model, and extracting image features of the image feature map using a fully connected layer corresponding to the integrated 3D-CNN model based on the attention weight; The image features are subjected to feature fusion to obtain target image features.

2. The method for optimizing radiomics feature extraction based on deep learning according to claim 1, characterized in that: The constructing the cumulative distribution function of the multimodal image includes: Preprocessing the multimodal image to obtain a preprocessed multimodal image; Calculating a modality histogram of the preprocessed multimodal image; trimming the extreme values ​​in the modal histogram to obtain a target modal histogram; The cumulative distribution function of the target modal histogram is constructed using the following formula; Among them, the cumulative distribution function is: A(i)=∑D(j), where j is the probability that the pixel value is less than or equal to i, i represents a specific value of the pixel value corresponding to the target modal histogram, D(j) represents the target modal histogram, and j represents the index of the pixel value corresponding to the accumulated target modal histogram when calculating the cumulative distribution function. The step of matching the multimodal images based on the cumulative distribution function to obtain a matched multimodal image includes:

3. The method for optimizing radiomics feature extraction based on deep learning according to claim 2, characterized in that: Identifying an image state of the multimodal image corresponding to the preprocessed multimodal image; Based on the image state, dividing the preprocessed multimodal image into a reference image and a target image; Based on the cumulative distribution function, calculating the KL divergence of the reference image and the target image; Constructing a matching strategy between the reference image and the target image through the KL divergence; Based on the matching strategy, the reference image and the target image are matched to obtain the matched multimodal image. The calculating the KL divergence of the reference image and the target image based on the cumulative distribution function includes:

4. The method for optimizing radiomics feature extraction based on deep learning according to claim 3, characterized in that: Based on the cumulative distribution function, marking a reference image distribution function and a target image distribution function of the reference image and the target image, respectively; The KL divergence of the reference image distribution function and the target image distribution function is calculated using the following formula: ​ Wherein, S(F||K) represents the KL divergence of the reference image distribution function and the target image distribution function, F(i) represents the target image distribution function of the target image, K(i) represents the reference image distribution function, and log(F(i) / K(i) represents the natural logarithm function.

5. The method for optimizing radiomics feature extraction based on deep learning according to claim 4, characterized in that: The step of establishing the multimodal segmentation model for matching the multimodal images comprises: Obtaining the matching multimodal image corresponding model training image; Performing ROI marking on the model training image to obtain a ROI marked training image; Performing modality clustering on the ROI-marked training image to obtain a modality clustered image; Constructing an initial modality segmentation model of the modality type corresponding to the modality clustering image; defining a segmentation task of the modality type; Based on the segmentation task, the initial modality segmentation model is collaboratively trained using the modality clustering images to obtain the multimodal segmentation model.

6. The method for optimizing radiomics feature extraction based on deep learning according to claim 5, characterized in that: Based on the segmentation task, the initial modality segmentation model is collaboratively trained using the modality clustering image to obtain the multimodal segmentation model, including: Identifying image modality features of the modality clustering images; Based on the image modality features, analyzing the modality correlation coefficient of the modality clustering image; Based on the modal association coefficient, construct a multi-task loss function of the initial modal segmentation model; Co-training the initial modality segmentation model based on the multi-task loss function to obtain a training modality segmentation model; Calculating the Dice coefficient of the training modality segmentation model; When the Dice coefficient meets a preset Dice coefficient threshold, the training modality segmentation model is used as the multimodal segmentation model.

7. The method for optimizing radiomics feature extraction based on deep learning according to claim 6, characterized in that: The step of extracting the image feature map of the slice ROI image by using the convolutional layer corresponding to the integrated 3D-CNN model includes: Standardizing the slice ROI image to obtain a standard ROI image; Initializing the convolution kernel of the convolution layer to obtain an initialized convolution layer; Calculating the dot product of the initialized convolutional layer and the standard ROI image; Constructing an initial feature map of the standard ROI image through the dot product; A preset activation function is applied to the initial feature map to obtain the image feature map.

8. The method for optimizing radiomics feature extraction based on deep learning according to claim 7, characterized in that: The attention mechanism in the integrated 3D-CNN model calculates the attention weight of the image feature map, including: Input the image feature map into the image storage module of the attention mechanism, and calculate the channel attention and spatial attention of the image feature through the attention calculation module of the attention mechanism; Based on the channel attention and the spatial attention, the attention weight of the image feature map is calculated using the following formula: Among them, τ represents the attention weight of the image feature map, G r represents the channel attention matrix, G c represents the spatial attention matrix, Softmax(G r ) represents channel attention, Softmax(G c ) represents spatial attention, Softmax represents a function that normalizes the vector to a probability distribution, represents the attention weighted operation.

9. The method for optimizing radiomics feature extraction based on deep learning according to claim 8, characterized in that: The extracting the image features of the image feature map based on the attention weight using the fully connected layer corresponding to the integrated 3D-CNN model includes: Multiplying the attention weight and the image feature map element by element to obtain a weighted feature map; Flattening the weighted feature map into a one-dimensional vector using the fully connected layer; Multiplying the one-dimensional vector by a preset weight matrix to obtain a vector multiplication result; Based on the vector multiplication result and a preset bias vector, the image features of the image feature map are output.

10. A radiomics feature extraction optimization system based on deep learning, characterized in that: For executing the deep learning-based radiomics feature extraction optimization method according to any one of claims 1 to 9, the system comprises: A multimodal image matching module, used to acquire a multimodal image, construct a cumulative distribution function of the multimodal image, and match the multimodal image based on the cumulative distribution function to obtain a matched multimodal image; An image ROI extraction module is used to establish a multimodal segmentation model for the matching multimodal image, and perform ROI segmentation on the matching multimodal image through the multimodal segmentation model to obtain a target ROI image; A 3D-CNN model construction module, used to construct a 3D-CNN model of the target ROI image, determine an attention mechanism of the 3D-CNN model, and integrate the attention mechanism into the 3D-CNN model to obtain an integrated 3D-CNN model; An image feature extraction module is used to slice the target ROI image to obtain a sliced ​​ROI image, extract an image feature map of the sliced ​​ROI image using the convolutional layer corresponding to the integrated 3D-CNN model, calculate the attention weight of the image feature map using the attention mechanism in the integrated 3D-CNN model, and extract image features of the image feature map based on the attention weight using the fully connected layer corresponding to the integrated 3D-CNN model; The image feature fusion module is used to perform feature fusion on the image features to obtain target image features.

Images