A multi-modal feature fusion method and application

Abstract

The technical scheme of the present application is to provide a multi-modal feature fusion method, characterized in that the image deep learning features feature DL , a plurality of image manual feature records and patient clinical features feature clinic are fused to obtain final fusion features. Another technical scheme of the present application is to provide an application of the multi-modal feature fusion method, characterized in that the application is used in treatment effect prediction. The present application takes a convolutional neural network as a basis and uses the convolutional neural network as a backbone network for image feature extraction to extract image deep learning features. At the same time, the present application introduces manual image features of different ROI regions and patient clinical features. The present application fuses the image deep learning features, the manual image features and the patient clinical features in an end-to-end manner to form the same cross of different modal data. The classification and discrimination of the fusion features obtained by using the present application can obtain more accurate prediction results.

Description

Technical Field

[0001] This invention relates to a cross-modal information fusion method and its application in predicting treatment effects. Background Technology

[0002] Recently, machine learning algorithms have been increasingly widely applied in fields such as education, transportation, and finance. The medical field is also actively using machine learning techniques to explore applications in assisted diagnosis and examination. However, most machine learning techniques currently used in assisted diagnosis and examination are based on single-modality data or a single region of interest (ROI), and the few machine learning techniques based on fusion algorithms employ post-fusion techniques, which lack common intersections between different modalities. Summary of the Invention

[0003] The technical problem that this invention aims to solve is that most existing machine learning technologies applied in medical-related fields are based on single-modal data or a single region of interest. Even when fusion algorithms are used, these algorithms are post-fusion techniques.

[0004] To address the aforementioned technical problems, one technical solution of the present invention is to provide a multimodal feature fusion method, characterized by comprising the following steps:

[0005] Step 1: Obtain the patient's medical imaging data, and manually delineate the regions of interest (ROIs) in the medical imaging data to obtain N manual ROI data.

[0006] Step 2: Establish a deep learning network model. After cropping the largest ROI cross-section from the image data to obtain ROI images (roi_images), extract the features of the ROI images (roi_images) to obtain the deep learning features of the image. DL ;

[0007] Extract handcrafted features from N handcrafted ROI data points. The handcrafted image features extracted based on the i-th handcrafted ROI data point are denoted as...

[0008] Generate patient clinical features directly from patient clinical data. clinic ;

[0009] Step 3: Extract deep learning features from the image. DL Image manual features are recorded as and patient clinical features clinic Mapped to the same dimension;

[0010] Step 4: Manually record the features of N images Adding the corresponding positions together yields the fused handcrafted features of the ROI, resulting in:

[0011]

[0012] Step 5: Combine ROI handcrafted features with image deep learning features DL The images are stitched together to obtain fused image features.

[0013] Step 6: Combine the fused imaging features with the patient's clinical features. clinic The distance features are obtained by subtracting the corresponding positions.

[0014] Step 7: Combine the fused imaging features with the patient's clinical features. clinic Multiply at corresponding positions to obtain the inner product features;

[0015] Step 8: Concatenate the distance feature and the inner product feature to obtain the fused feature.

[0016] Another technical solution of the present invention is to provide an application of the above-mentioned multimodal feature fusion method, characterized in that, when applied to treatment effect prediction, the fused features obtained by the above-mentioned multimodal feature fusion method are input into the trained treatment effect classification prediction model to obtain the final classification result.

[0017] This invention uses a convolutional neural network as the backbone network for image feature extraction to obtain deep learning features. Simultaneously, it incorporates handcrafted image features from different Regions of Interest (ROIs) and patient clinical features. This invention fuses deep learning features, handcrafted image features, and patient clinical features end-to-end, forming identical cross-references between different modalities. Using the fused features obtained in this invention for classification and discrimination yields more accurate prediction results. Attached Figure Description

[0018] Figure 1 This is the overall flowchart of the present invention;

[0019] Figure 2 It illustrates typical data image formats, handcrafted features (shape, texture, etc.), and clinical features. Detailed Implementation

[0020] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0021] In clinical applications, patient information generally includes the following three different types of information:

[0022] a) Image data refers to medical images collected by users through machines such as CT and MRI.

[0023] b) Segmented data: For CT and clinical scenarios, the ROI data is obtained by delineating the region of interest in the image data.

[0024] c) Clinical data, which generally includes the patient's age, gender, etc., is inherently structured.

[0025] Combination Figure 1 Based on the above three types of information, the multimodal feature fusion method disclosed in this embodiment includes the following steps:

[0026] Step 1: Feature extraction, including deep learning feature extraction of images, manual feature extraction of images, and extraction of patient clinical features.

[0027] (1) Image Deep Learning Feature Extraction

[0028] A convolutional neural network-based model is established. The model crops the largest Region of Interest (ROI) sections from the image data to obtain ROI images (roi_images). Features of the ROI images (roi_images) are then extracted to obtain deep learning features of the image. DL In this embodiment, the model is trained based on transfer learning.

[0029] (2) Image manual feature extraction

[0030] We use pyradiomics (https: / / pyradiomics.readthedocs.io / en / latest / ) to extract handcrafted features from each segmentation data. The handcrafted image features extracted based on the i-th segmentation data are denoted as... N is the total number of data segments.

[0031] (3) Extraction of patient clinical features

[0032] Generate patient clinical features directly from clinical data. clinic .

[0033] Step 2, Feature Fusion, further includes the following steps:

[0034] Step 201: Use a deep neural network to learn the deep learning features of the image obtained in Step 1. DL Image manual characteristics Patient clinical characteristics clinicMapped to a semantic space of the same dimension.

[0035] Step 202: Manually record N types of image features Adding the corresponding positions together yields the fused handcrafted features of the ROI, resulting in:

[0036]

[0037] Step 203: Combine ROI handcrafted features with image deep learning features DL The images are stitched together to obtain fused image features.

[0038] Step 204: Combine the fused imaging features with the patient's clinical features. clinic The distance features are obtained by subtracting the corresponding positions.

[0039] Step 205: Combine the fused imaging features with the patient's clinical features. clinic Multiply at the corresponding positions to obtain the inner product features.

[0040] Step 206: Concatenate the distance feature and the inner product feature to obtain the fused feature.

[0041] When the fusion features obtained by the above method are applied to the prediction of treatment effect, the fusion features are input into the trained treatment effect classification prediction model to obtain the final classification result.

Claims

1. A multi-modal feature fusion method, characterized in that, The method comprises the following steps: Step 1, obtaining medical image data of a patient, and obtaining N manual ROI data after manual delineation of a region of interest on the medical image data; Step 2, a deep learning network model is established, after the model crops the maximum ROI section in the image data to obtain the ROI image roi_images, the features of the ROI image roi_images are extracted, thereby obtaining the image deep learning features feature DL ; extracting the manual features of the N manual ROI data, and the image manual features extracted based on the i-th manual ROI data are denoted as Generating patient clinical features directly based on patient clinical data clinic ; Step 3, map image deep learning features feature DL , image hand-crafted features feature and patient clinical features feature clinic to the same dimensionality; Step 4, N image manual feature records are manually recorded The corresponding positions are added to obtain the fused ROI manual feature, and there is: Step 5, fuse the ROI manual features with the image deep learning features feature DL The stitching is performed to obtain the fused image features; Step 6, fuse the image features with the patient clinical features feature clinic Subtract the corresponding positions to obtain the distance features; Step 7, fuse image features with patient clinical features feature clinic Multiply the corresponding positions to get the inner product features; Step 8, obtaining a fusion feature after splicing the distance feature and the inner product feature.

2. Use of the multi-modal feature fusion method of claim 1, characterized in that, In the application in the treatment effect prediction, the fusion feature obtained by the multi-modal feature fusion method of claim 1 is input into a trained treatment effect classification prediction model to obtain a final classification result.

Images