Breast cancer new auxiliary information analysis system based on multi-modal fusion

Abstract

The invention provides a breast cancer new auxiliary information analysis system based on multi-modal fusion, and aims to improve the accuracy and comprehensiveness of breast cancer analysis by integrating radiation image data, gene data and clinical data. In a feature extraction module, the system adopts a multi-scale variable convolution pyramid residual error method to extract image features; in the multi-modal fusion module, deep fusion is performed on different data sources in combination with a triple cross attention mechanism and a space-time consistency-negative logarithmic loss function, the effect of each data type is maximized, and redundant information is reduced, so that an optimized breast cancer analysis result is generated; and finally, the decision support module performs risk assessment and prediction according to an analysis result. The system can effectively integrate multi-modal data, improve the accuracy of breast cancer analysis, provide more comprehensive support for related research, and promote further development of the breast cancer analysis field.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and in particular to a new auxiliary information analysis system for breast cancer based on multimodal fusion. Background Art

[0002] With the continuous advancement of medical technology, information analysis of breast cancer has gradually shifted from traditional imaging and biomarker detection to comprehensive analysis of multimodal data. However, traditional breast cancer auxiliary information analysis systems still have the following shortcomings: First, due to the limitations of data sources, traditional systems cannot fully capture all tumor characteristics. The single data processing mode of these systems limits their performance in breast cancer analysis, making it difficult to provide more accurate analysis results. Second, traditional systems can only analyze from a single dimension and cannot fully explore the potential deep relationships between various data sources. This limitation makes it impossible for existing systems to fully utilize the synergy between data when faced with multidimensional data. Summary of the Invention

[0003] The present invention provides a new breast cancer auxiliary information analysis system based on multimodal fusion, aiming to improve the analysis accuracy of breast cancer-related data through the effective integration of multiple information sources. First, the system combines 3D variable-scale convolution, 2D convolution, and a pyramid residual module in the feature extraction module to perform multi-scale processing on the radiological imaging data of breast cancer patients, extracting more representative image features and providing high-quality feature maps for subsequent analysis. Second, the system introduces a triple cross-attention mechanism in the multimodal fusion module, combined with a spatiotemporal consistency-negative logarithmic loss function, to complete the fusion processing of multimodal data. Through sophisticated fusion methods, this module ensures the comprehensive utilization of various types of data, not only improving data relevance but also effectively avoiding information redundancy and loss, thereby improving the accuracy and reliability of analysis results. Through the above data processing and fusion methods, the breast cancer auxiliary information analysis system of the present invention can efficiently integrate multimodal data related to breast cancer, ensuring maximum synergy between different data sources, thereby providing more in-depth analysis tools and support for the field of breast cancer research.

[0004] The present invention provides a new auxiliary information analysis system for breast cancer based on multimodal fusion, which includes a data acquisition module, a data preprocessing module, a gene feature extraction module, an image feature extraction module, a multimodal fusion module, and a decision support module;

[0005] Data acquisition module, which collects MRI images, CT images and X-ray images as radiological imaging data, and collects genetic data and clinical data;

[0006] A data preprocessing module preprocesses radiographic image data, genetic data, and clinical data to generate preprocessed radiographic image data, preprocessed genetic data, and preprocessed clinical data;

[0007] Gene feature extraction module, extracts features of preprocessed gene data and generates gene feature data;

[0008] Image feature extraction module, constructs a multi-scale variable convolution pyramid residual model; pre-processes radiological image data through the multi-scale variable convolution pyramid residual model to generate a multi-scale pyramid residual feature map;

[0009] The multimodal fusion module builds a cross-attention fusion model, initializes the cross-attention fusion model parameters, inputs the multi-scale pyramid residual feature map, gene feature data and pre-processed clinical data into the cross-attention fusion model, and generates optimized breast cancer analysis results;

[0010] A decision support module, which performs disease risk assessment, prognosis prediction, personalized monitoring plan recommendations, disease progression trend analysis, and patient grouping and risk classification based on optimized breast cancer analysis results;

[0011] The multi-scale variable convolution pyramid residual model includes a 3D deep variable convolution layer, a 2D convolution layer and a pyramid residual module.

[0012] Furthermore, the image feature extraction module generates a multi-scale pyramid residual feature map, which specifically includes the following steps:

[0013] Step D1: Joint feature extraction is performed on the pre-processed radiological image data through a 3D deep variable convolutional layer to generate a multi-scale 3D convolutional feature map;

[0014] Step D2: further extract the detailed spatial features of the multi-scale 3D convolutional feature map through the 2D convolution layer to generate a 2D convolutional feature map;

[0015] Step D3: Increase the number of channels of the feature map layer by layer through the pyramid residual module, further extract the high-level features of the 2D convolution feature map, and generate a multi-scale pyramid residual feature map.

[0016] Furthermore, step D1 specifically includes the following steps:

[0017] Step D11: Select convolution kernel size: According to the complexity and data characteristics of the preprocessed radiological image data, adjust the convolution kernel size of the 3D deep variable convolution to obtain a variable-scale convolution kernel;

[0018] Step D12: Convolution operation: Perform 3D convolution on the preprocessed radiological image data using a variable-scale convolution kernel; generate a multi-scale 3D convolution feature map.

[0019] Furthermore, the multimodal fusion module generates a process for optimizing breast cancer analysis results, specifically including the following steps:

[0020] Step E1: Use the multi-scale pyramid residual feature map as the query and the gene feature data as the key and value to generate image-gene association data;

[0021] Step E2: Generate image-clinical correlation data using the multi-scale pyramid residual feature map as the query and the clinical data as the key and value;

[0022] Step E3: Generate gene-clinical association data using gene signature data as query and clinical data as key and value;

[0023] Step E4: splicing the image-gene association data, the image-clinical association data, and the gene-clinical association data to generate fused feature data;

[0024] Step E5: Process the fused feature data through three fully connected layers to generate breast cancer analysis results;

[0025] Step E6: Optimize the cross-attention fusion model parameters through the spatiotemporal consistency-negative logarithmic loss function to generate optimized breast cancer analysis results.

[0026] Furthermore, step E6 specifically includes the following steps:

[0027] Step E61: Introduce spatiotemporal consistency and negative log-likelihood loss function, set the adjustment coefficient, balance the weights between the spatiotemporal consistency and the loss term of the negative log-likelihood loss function, and construct the spatiotemporal consistency-negative log-likelihood loss function;

[0028] Step E62: Train the cross-attention fusion model through the spatiotemporal consistency-negative logarithmic loss function, optimize the cross-attention fusion model parameters through backpropagation, and use the spatiotemporal consistency constraint to adjust the prediction output of the cross-attention fusion model to generate optimized breast cancer analysis results.

[0029] By adopting the above scheme, the beneficial effects achieved by the present invention are as follows:

[0030] This invention provides a new breast cancer auxiliary information analysis system based on multimodal fusion. It achieves efficient fusion of multimodal data and solves the inefficiency and information loss problems of traditional breast cancer auxiliary information analysis systems during the data integration process. First, the feature extraction module combines 3D variable-scale convolution, 2D convolution, and a pyramid residual module to perform multi-scale processing on the radiological imaging data of breast cancer patients. The system can capture the spatial distribution and detailed characteristics of the tumor area, providing accurate information for subsequent analysis and improving the ability to express tumor characteristics. This processing can reveal the gene expression profile and gene mutation information of breast cancer patients, providing crucial basic data for subsequent multimodal fusion and analysis.

[0031] Furthermore, a multimodal fusion module based on a triple cross-attention mechanism, combined with a spatiotemporal consistency-negative logarithmic loss function, can deeply fuse multiple data sources, thereby enhancing the correlation between different data sources and maximizing the role of each data type in the analysis process. Through this sophisticated fusion approach, the system can eliminate redundant information between different data types, reduce information interference, and improve the accuracy of the fused data. This technology significantly enhances the predictive power of breast cancer analysis results, providing more efficient and reliable support for breast cancer research and risk assessment, and promoting further development in the field of breast cancer auxiliary information analysis.

[0032] The present invention significantly improves the performance and accuracy of breast cancer analysis systems by combining the aforementioned multimodal data fusion technology with advanced feature extraction methods and efficient data processing mechanisms. By effectively integrating radiological imaging data, genetic data, and clinical data, the present invention not only increases the comprehensive utilization rate of various data types but also ensures the synergy of information between different data sources. This provides more comprehensive and accurate support for breast cancer-related research and data analysis, and promotes the development of breast cancer analysis systems towards a more efficient and intelligent direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of a model of a new auxiliary information analysis system for breast cancer based on multimodal fusion proposed by the present invention;

[0034] Figure 2 This is a model structure diagram of the multi-scale variable convolutional pyramid residual model proposed in Example 2;

[0035] Figure 3 This is a model structure diagram of the cross-attention fusion model proposed in Example 5. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0037] Example 1, according to Figure 1 ,The present invention provides a new auxiliary information analysis system for breast cancer based on multimodal fusion, the system includes a data acquisition module, a data preprocessing module, a gene feature extraction module, an image feature extraction module, a multimodal fusion module, and a decision support module;

[0038] The data acquisition module collects MRI images, CT images, and X-ray images as radiological imaging data, as well as genetic data and clinical data. Genetic data includes RNA sequencing data, gene mutation information, and gene expression profiles. Clinical data includes basic patient information, clinical characteristics of the tumor, and the patient's treatment background.

[0039] The data preprocessing module performs slice annotation, region selection, image cropping, image segmentation and standardization on radiological image data to generate preprocessed radiological image data; standardizes and cleans genetic data and clinical data to generate preprocessed genetic data and preprocessed clinical data;

[0040] Gene feature extraction module, extracts features of preprocessed gene data and generates gene feature data;

[0041] The image feature extraction module combines 3D variable-scale convolution, 2D convolution, and pyramid residual modules to build a multi-scale variable convolution pyramid residual model. The multi-scale variable convolution pyramid residual model processes the pre-processed radiological image data to generate a multi-scale pyramid residual feature map. The features of the pre-processed genetic data are extracted to generate genetic feature data. The multi-scale variable convolution pyramid residual model includes a 3D deep variable convolution layer, a 2D convolution layer, and a pyramid residual module.

[0042] The multimodal fusion module combines the triple cross-attention mechanism and the spatiotemporal consistency-negative logarithmic loss function to build a cross-attention fusion model, initialize the parameters of the cross-attention fusion model, input the multi-scale pyramid residual feature map, gene feature data and pre-processed clinical data into the cross-attention fusion model, and generate optimized breast cancer analysis results;

[0043] The decision support module performs disease risk assessment, prognosis prediction, personalized monitoring plan recommendations, disease progression trend analysis, patient grouping and risk classification based on optimized breast cancer analysis results.

[0044] Example 2, according to Figure 2 This embodiment is based on the first embodiment. In this embodiment, the image feature extraction module processes the pre-processed radiographic image data through the multi-scale variable convolutional pyramid residual model to generate a multi-scale pyramid residual feature map, which specifically includes the following steps:

[0045] Step D1: Joint feature extraction of spatial and spectral information of preprocessed radiological image data is performed through a 3D deep variable convolutional layer to generate a multi-scale 3D convolutional feature map;

[0046] Step D2: Further extract the detailed spatial features of the multi-scale 3D convolutional feature map through the 2D convolution layer, and reduce the computational complexity through the pooling layer to generate a 2D convolutional feature map;

[0047] Step D3: The number of channels of the feature map is increased layer by layer through the pyramid residual module to further extract the high-level features of the 2D convolutional feature map. The residual block in each pyramid residual module optimizes feature transfer through cross-layer connection and addition operations, reduces information loss, and enhances the expressiveness of features to generate a multi-scale pyramid residual feature map.

[0048] Embodiment 3: This embodiment is based on embodiment 1. In this embodiment, the feature extraction module generates a multi-scale pyramid residual feature map, specifically including the following steps:

[0049] Step R1: perform joint feature extraction of spatial and spectral information on the preprocessed radiological image data through 3D convolution to generate a 3D convolution feature map;

[0050] Step R2: Further extract the detailed spatial features of the 3D convolutional feature map through the 2D convolution layer, and reduce the computational complexity through the pooling layer to generate a 2D convolutional feature map;

[0051] Step R3: In the pyramid residual module, the high-level features of the 2D convolutional feature map are further extracted by increasing the number of channels of the feature map layer by layer. The residual block in each pyramid residual module optimizes feature transfer through cross-layer connection and addition operations, reduces information loss, and enhances the expressiveness of features to generate a multi-scale pyramid residual feature map.

[0052] Embodiment 4: This embodiment is based on embodiment 2. In this embodiment, step D1 specifically includes the following steps:

[0053] Step D11: Select convolution kernel size: According to the complexity and data characteristics of the preprocessed radiological image data, adjust the convolution kernel size of the 3D deep variable convolution to obtain a variable-scale convolution kernel;

[0054] Step D12: Convolution operation: Use a variable-scale convolution kernel to perform 3D convolution on the preprocessed radiographic image data. The variable-scale convolution kernel slides across the width, height, and depth dimensions of the preprocessed radiographic image data to calculate the features of each local region and generate a multi-scale 3D convolution feature map. The formula used is as follows:

[0055] Variable scale 3D convolution formula:

[0056] ;

[0057] in, represents the position of the pixels of the preprocessed radiographic data in three-dimensional space, Represents the index of the variable-scale convolution kernel in three dimensions: width, length, and depth. represents the convolutional layer index, Represents the variable scale convolution kernel in The size of the layer, i.e. the number of elements covered by the variable-scale convolution kernel in width, length and depth; Represents the pixel value after the convolution operation, that is, the feature value at the position (x, y, z); represents the weight of the variable-scale convolution kernel, Represents the pixel value of the preprocessed radiographic image data at the current position.

[0058] Example 5, according to Figure 3 This embodiment is based on the fourth embodiment. In this embodiment, the multimodal fusion module generates an optimized breast cancer analysis result, specifically including the following steps:

[0059] Step E1: Use the multi-scale pyramid residual feature map as the query and the gene feature data as the key and value to generate image-gene association data through cross-attention calculation;

[0060] Step E2: Using the multi-scale pyramid residual feature map as the query and the clinical data as the key and value, the image-clinical correlation data is generated through cross-attention calculation;

[0061] Step E3: Generate gene-clinical association data by using gene feature data as query and clinical data as key and value through cross-attention calculation;

[0062] Step E4: splicing the image-gene association data, the image-clinical association data, and the gene-clinical association data to generate fused feature data;

[0063] Step E5: Process the fused feature data through three fully connected layers to generate breast cancer analysis results;

[0064] Step E6: The cross-attention fusion model is trained through the spatiotemporal consistency-negative logarithmic loss function to improve the accuracy of the prediction results of the cross-attention fusion model and generate optimized breast cancer analysis results.

[0065] Example 6: This example is based on Example 4. In this example, the multimodal fusion module generates an optimized breast cancer analysis result, specifically including the following steps:

[0066] Step Q1: Use the multi-scale pyramid residual feature map as the query and the gene feature data as the key and value to generate image-gene association data through cross-attention calculation;

[0067] Step Q2: Use the multi-scale pyramid residual feature map as the query and the clinical data as the key and value to generate image-clinical correlation data through cross-attention calculation;

[0068] Step Q3: splicing the image-gene association data and the image-clinical association data to generate fused feature data;

[0069] Step Q4: Process the fused feature data through three fully connected layers to generate breast cancer analysis results;

[0070] Step Q5: Train the cross-attention fusion model through the negative log-likelihood loss function to improve the accuracy of the prediction results of the cross-attention fusion model and generate optimized breast cancer analysis results.

[0071] Embodiment 7: This embodiment is based on embodiment 5. In this embodiment, step E6 specifically includes the following steps:

[0072] Step E61: Introduce spatiotemporal consistency and negative log-likelihood loss functions, set the adjustment coefficient, balance the weights between the spatiotemporal consistency and the loss terms of the negative log-likelihood loss function, and construct the spatiotemporal consistency-negative log-likelihood loss function. The formula used is as follows:

[0073] ;

[0074] in, represents the spatiotemporal consistency loss value, and represents the adjustment coefficient, Represents a position index in time or space, Indicates the current sample index, Represents the index on the time dimension, Represents the index on the spatial dimension; Indicates the Samples at time point The risk value, Indicates the Samples at time point The risk value, Indicates the Samples at spatial locations The risk value, Indicates the Samples at spatial locations Value at risk; represents the time consistency term, represents the spatial consistency term;

[0075] ;

[0076] in, Represents the loss value of the spatiotemporal consistency-negative logarithmic loss function, represents the index of censored samples, Indicates the The risk value of a sample, Indicates the The risk value of a sample, Indicates the a right-censored set of samples; Indicates the The natural logarithm of the risk value of the sample, Indicates the Risk value of a sample Perform exponential operations; Indicates the event indicator for each sample;

[0077] Step E62: Train the cross-attention fusion model through the spatiotemporal consistency-negative logarithmic loss function, optimize the cross-attention fusion model parameters through backpropagation, and use the spatiotemporal consistency constraint to adjust the prediction output of the cross-attention fusion model to generate optimized breast cancer analysis results.

[0078] The present invention and its embodiments are described above. Such description is not restrictive. What is shown in the accompanying drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in this field are inspired by it and do not depart from the purpose of the invention, they can creatively design structural methods and embodiments similar to the technical solution, which should all fall within the scope of protection of the present invention.

Claims

1. A new auxiliary information analysis system for breast cancer based on multimodal fusion, comprising a data preprocessing module and a gene feature extraction module. The data preprocessing module generates preprocessed radiographic image data and preprocessed clinical data; the gene feature extraction module generates gene feature data. The system is characterized by: The system also includes an image feature extraction module and a multimodal fusion module; The image feature extraction module constructs a multi-scale variable convolution pyramid residual model; pre-processes the radiographic image data through the multi-scale variable convolution pyramid residual model to generate a multi-scale pyramid residual feature map; The multimodal fusion module constructs a cross-attention fusion model, initializes the parameters of the cross-attention fusion model, inputs the multi-scale pyramid residual feature map, gene feature data and preprocessed clinical data into the cross-attention fusion model, and generates optimized breast cancer analysis results.

2. The breast cancer new auxiliary information analysis system based on multimodal fusion according to claim 1, characterized in that: The multi-scale variable convolution pyramid residual model includes a 3D deep variable convolution layer and a pyramid residual module.

3. The breast cancer new auxiliary information analysis system based on multimodal fusion according to claim 2, characterized in that: The feature extraction module generates a multi-scale pyramid residual feature map, which specifically includes the following steps: Step D1: Joint feature extraction is performed on the pre-processed radiological image data through a 3D deep variable convolutional layer to generate a multi-scale 3D convolutional feature map; Step D2: extract the features of the multi-scale 3D convolution feature map and generate a 2D convolution feature map; Step D3: Extract high-level features of the 2D convolutional feature map through the pyramid residual module to generate a multi-scale pyramid residual feature map.

4. The breast cancer new auxiliary information analysis system based on multimodal fusion according to claim 3, characterized in that: Step D1 specifically includes the following steps: Step D11: adjusting the convolution kernel size of the 3D deep variable convolution layer according to the preprocessed radiological image data to obtain a variable-scale convolution kernel; Step D12: Use a variable-scale convolution kernel to perform 3D convolution on the preprocessed radiological image data to generate a multi-scale 3D convolution feature map.

5. The breast cancer new auxiliary information analysis system based on multimodal fusion according to claim 1, characterized in that: The multimodal fusion module generates an optimized breast cancer analysis result, specifically comprising the following steps: Step E1: Use the multi-scale pyramid residual feature map as the query and the gene feature data as the key and value to generate image-gene association data; Step E2: Generate image-clinical correlation data using the multi-scale pyramid residual feature map as the query and the clinical data as the key and value; Step E3: Generate gene-clinical association data using gene signature data as query and clinical data as key and value; Step E4: splicing the image-gene association data, the image-clinical association data, and the gene-clinical association data to generate fused feature data; Step E5: Processing the fused feature data to generate breast cancer analysis results; Step E6: Optimize the cross-attention fusion model parameters through the spatiotemporal consistency-negative logarithmic loss function to generate optimized breast cancer analysis results.

Images